Kubernetes

The scriptworkers are hosted in Google Compute Cloud (GCP) using Kubernetes. The deployment process is managed by the CloudOps team.

If you want to debug the GCP workers in the console, make sure you have access to the GCP console for the production and nonprod environments. You can get access by filing a bug against CloudOps.

The CloudOps deployment process is managed by Jenkins and the corresponding files can be found in the cloudops-infra repo.

The environment variables are set per deployment and can be found in the k8s/values directory. The default values are set in a separate file. The variables use camelCase, but then converted to SHELL_STYLE in configmap.yml

When Kubernetes decides to stop a worker it gives it 1200s to do this gracefully, see the corresponding k8s config. As a part of shutdown process, Kubernetes runs docker.d/pre-stop.sh to us enough time to finish running tasks. docker.d/pre-stop.sh sends SIGUSR1 to scriptworker, what makes it exit as soon as it finishes the running task. 2 minutes before the deadline docker.d/pre-stop.sh exits and lets Kubernetes send SIGTERM what makes scriptworker cancel the task and upload the logs.

Kubernetes also runs a health check, by calling docker.d/healthcheck periodically. If the script exits non-zero or times out, the corresponding replica is marked as non healthy and will be removed from the pool.