GitLab Runners

[This is part 5/8 on GitLab Continuous Integration series]

An official definition:

The runners are the actual elements that do the work in a CI setup. They are assigned CI/CD jobs by the server according to a pipeline script that if all is set properly runs automatically when a commit is pushed to the server or manually if needed.

I found that very confusing so I like to explain those elements as follows:

• The GitLab Runner is an agent installed on a different machine from the GitLab server. It is the one performing the scripts.
• The ‘environment’ that a Runner uses to perform a task is the Runner Executor declared hwen we create the runner an written in its configuration file. It can do the tasks with [local or remote] shell commands, send ssh commands to servers, or in a virtual machine. The preferred one uses a docker container that you choose as it has the software needed (can be python or maven), or be a pod in a Kubernetes cluster that deploys and also monitors the jobs. You can mix Executors in your jobs on each stage.
• The Runner is Registered in the GitLab Server according the projects it can run. They can be Shared (for unassigned projects), Group or Project-specific (can be more secure as environment variables are only shared with our code).
• GitLab Job: Is the YAML script or pipeline, which contains one or more commands that need to be executed. You can use variables that you define or get from the server, pass objects from stage to stage, etc.

---

I. What we need

For automation using GitLab CI we need:

1. To install an executor or GitLab runner that will do the actual work aka jobs. To distribute load they should be installed on a machines that are separated from the Server
2. Then register that GitLab runner to the GitLab Server or coordinator. In the server the runners can be configured so they can be assigned all or specific pipelines.
3. Verify that running ‘CI pipelines’ are enabled in our server and project.
4. Add a pipeline script with stage our jobs. It is usually located in the project’s root directory. It is normally called .gitlab-ci.yml. Yes, a hidden file.

Executors

As the documentation states, there are several types of runners that can be used to run your builds for different scenarios.

Executor	SSH	Shell	Virtualbox	Docker	Kubernetes
Clean build environment for every build	✗	✗	✓	✓	✓
Reuse previous clone if it exists	✓	✓	✗	✓	✗
Runner file system access protected	✓	✗	✓	✓	✓
Complicated build environments	✗	✗ (1)	✓ (2)	✓	✓
Debugging build problems	easy	easy	hard	medium	medium
Note				runs each build in an isolated container	Creates a Pod for each Job

And each one supports different features:

Executor	SSH	Shell	VirtualBox	Docker	Kubernetes
Secure Variables	✓	✓	✓	✓	✓
GitLab Runner Exec command	✗	✓	✗	✓	✓
gitlab-ci.yml: image	✗	✗	✗	✓	✓
gitlab-ci.yml: services	✗	✗	✗	✓	✓
gitlab-ci.yml: cache	✓	✓	✓	✓	✓
gitlab-ci.yml: artifacts	✓	✓	✓	✓	✓
Passing artifacts between stages	✓	✓	✓	✓	✓
Use GitLab Container Registry private images	n/a	n/a	n/a	✓	✓
Interactive Web terminal	✗	✓ (UNIX)	✗	✓	✓ (Planned for Helm)

We choose to use the Docker Executor as it offers more features, specially that it allows access to the private Container Repository. And further ahead it will allow also the use of Docker in Docker (DinD) services.

---

II. Install a GitLab Runner

To use GitLab Runner we will install it as a Docker container. We can use Linux’s apt or download it. In my case, to follow the recommendation to install it in another machine than the server, I installed it in my laptop computer.

$ cd ~/Descargas<br>
$ sudo apt update

This following is the correct command but it didn’t work at the end on 2020 in my Ubuntu/Debian 20.10 Linux Laptop. It might work by the time you are reading it:

$ sudo curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash

Note: By January 2021 there is no code for installing a gitlab-runner in ‘Groovy-Gorilla’ (Ubuntu 20.10). The error shown is:
[Edit on dec 2021 the command runs fine]

Error as Groovy-Gorilla Ubuntu doesn't have a candidate.
Unable to download repo config from: https://packages.gitlab.com/install/repositories/runner/gitlab-runner/config_file.list?os=Ubuntu&dist=groovy&source=script
You can override the OS detection by setting 'os=' and 'dist=' prior to running this script.
You can find a list of supported OSes and distributions on our website: https://packages.gitlab.com/docs#os_distro_version

We can see in the list of official GitLab Runner repositories: Xenial (16.04), Bionic (18.04) and Focal (20.04). Considering tat the forums are full of reports for 19.03 we can assume that our 20.10 version will show them also. So we have Focal or Bionic as candidates.
Now we check the availability of our the Docker we will use in our CI Pipeline. For that DinD, the hub.docker list contains (20.10.3-dind, 20.10-dind, 20-dind, dind, 20.10.3-dind-rootless, 20.10-dind-rootless, 20-dind-rootless, dind-rootless, 19.03.15-dind, 19.03-dind, 19-dind, 19.03.15-dind-rootless, 19.03-dind-rootless, 19-dind-rootles).

In that Docker Hub page you can read two warnings:

• Although running Docker inside Docker is generally not recommended, there are some legitimate use cases, such as development of Docker itself. => We have to insist 🙂
• Starting in 18.09+ the dind image will automatically generate TLS certificates in the directory specified by the DOCKER_TLS_CERTDIR environment variable. It is recommended to enable TLS by setting the variable to an appropriate value (-e DOCKER_TLS_CERTDIR=/certs) => So we have to remember to disable that in our runner in the next section

After updating the command with an available distribution (focal=Ubuntu 20.04):

$ sudo curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo os=ubuntu dist=focal bash

# We are advised to use this for Debian Buster (Ubuntu 20.10) so it can create empty folders:
# (-E preserves environment variables) 
$ export GITLAB_RUNNER_DISABLE_SKEL=true; sudo -E apt install gitlab-runner

Check the Runner

To check the runner we can list them:

$ sudo gitlab-runner list
# Answer
Runtime platform             arch=amd64 os=linux pid=21747 revision=8fa89735 version=13.6.0
Listing configured runners   ConfigFile=/etc/gitlab-runner/config.toml
docker-runner                Executor=docker Token=o63mEQt4q-RJnj3yQNhW URL=http://gitlab.example.com/

get its version:

$ sudo gitlab-runner --version
# Answer
Version:      13.7.0
Git revision: 943fc252
Git branch:   13-7-stable
GO version:   go1.13.8
Built:        2020-12-21T13:47:06+0000
OS/Arch:      linux/amd64

and ask for a quick verification:

$ sudo gitlab-runner verify
# Answer
Runtime platform               arch=amd64 os=linux pid=70447 revision=775dd39d version=13.8.0
Running in system-mode.
Verifying runner... is alive   runner=xmWbMf4w

---

II. Register the Runner

Registering a Runner will bind the GitLab Runner with the GitLab Instance. There are three types of bindings:

• Shared that are available and runs jobs from all unassigned projects. In cloud services they are time limited. This is the one we will be using in the example.
• Group or
• Project-specific these can be more secure as environment variables are only shared with our code.

In the use the Docker executor Gitlab page we can find additional warnings:

• Each job is in a clean environment without the past history, there’s no caching of layers.
• A docker-privileged executor as --privileged is required for Docker-in-Docker. That is what we will use to package our app as a docker image and upload them to a repository. Bun if you enable --docker-privilegedyou are effectively disabling all of the security mechanisms of containers and exposing your host to privilege escalation.
• By default, Docker 17.09 and higher uses --storage-driver overlay2 which is the recommended storage driver.
• Since the docker:19.03.12-dind container and the runner container don’t share their root file system, the job’s working directory can be used as a mount point for child containers. For example, if you have files you want to share with a child container, you may create a subdirectory under /builds/$CI_PROJECT_PATH and use it as your mount point (check issue #41227):

variables:
  MOUNT_POINT: /builds/$CI_PROJECT_PATH/mnt

script:
  - mkdir -p "$MOUNT_POINT"
  - docker run -v "$MOUNT_POINT:/mnt" my-docker-image

Example project using this approach: https://gitlab.com/gitlab-examples/docker.

Back to the post. We are configuring an easy to use insecure CI Server. And we will be using the privileged attribute that can be exploited by running software to gain access to your machine. But we know what we are doing and love living dangerously. 🙂 (No, that is a joke, we have our Server behind our home firewall).

To register the runner we need data from the ‘shared runner’ page. Log in as Admin user into the server and go to: ‘Admin area’ > Overview > Runners Copy url & registration-token, and paste those values in the following command:

$ sudo gitlab-runner register \
  --non-interactive \
  --url "http://gitlab.example.com/" \
  --registration-token "QR6jYXGZdd7xHpsy1K5b" \
  --executor "docker" \
  --docker-image docker:stable \
  --description "docker-runner" \
  --docker-privileged \
  --tag-list "master" \
  --run-untagged="true" \
  --locked="false"
# access-level was added for version 12
# Answer
Runtime platform      arch=amd64 os=linux pid=21631 revision=8fa89735 version=13.6.0
Running in system-mode.
Registering runner... succeeded      runner=7f4nVsxX
Runner registered successfully. Feel free to start it, but if its running already the config should be automatically reloaded! 

Note: You can use an interactive version, that is started with:

$ sudo gitlab-runner register
# Answers you will need to provide:
http://gitlab.example.com/
QR6jYXGZdd7xHpsy1K5b
gitlab-runner
<Enter>
docker
docker:stable

Check if the Runner is registered

To list runners registered and available in the GitLab Server:

• As a user, go to Settings > CI/CD and expand the Runners section.
• As an administrator (root), for a Shared Runner go to ‘Admin Area’ (click wrench icon on toolbar) > Overview > Runners

---

Fine Tuning the System

As it is, the runner can run common shell commands like maven clean or maven build, but it will break when using the Docker file system to send files (artifacts) between stages or posting files to the GitLab Docker repository. We need to make some configuration modifications so edit the config file to add a few lines and modify a Docker tag line:

• add ‘clone_url’ that holds the IP of the GitLab server as a backup if it can’t find our ‘gitlab.example.com’ url:

clone_url = "http://192.168.1.221/"

• add also extra_hosts with the GitLab Server and repository names and IPs so the runner can find them:

extra_hosts = ["gitlab.example.com:192.168.1.221","registry.example.com:192.168.1.221"]

• Lets share the docker insecure registry configuration with our Docker that runs our jobs in the CI:

volumes = ["/etc/docker/daemon.json:/etc/docker/daemon.json","/cache"]

• In the ‘image’ change “docker:latest” to:

image = "docker:stable"

• Lets turn off TLS for our insecure setup:

  environment = ["GIT_SSL_NO_VERIFY=true", "DOCKER_TLS_CERTDIR="]

The resulting file will be:

$ sudo cat /etc/gitlab-runner/config.toml
# Content

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "docker-runner"
  url = "http://gitlab.example.com/"
  clone_url = "http://192.168.1.221/"
  token = "xmWbMf4wYFsPPGq7gJFR"
  executor = "docker"
  environment = ["DOCKER_TLS_CERTDIR=","GIT_SSL_NO_VERIFY=true"]
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
    Insecure = true
  [runners.docker]
    tls_verify = false
    image = "docker:stable"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/etc/docker/daemon.json:/etc/docker/daemon.json","/cache"]
    shm_size = 0
    extra_hosts = ["gitlab.example.com:192.168.1.221","registry.example.com:192.168.1.221"]

Don’t forget to restart the runner service to update it:

$ sudo service gitlab-runner restart

That’s it. You’ve got a CI server ready to work.

---

Remove a GitLab Runner

If later on you need to remove a runner you can click on the Runner page of the server or:

1. First unregister it in the GitLab Server or in the CLI:

$ sudo gitlab-runner unregister --token ugNjkgTFHcndF4_btFuQ --url http://gitlab.example.com/
# Answer
Runtime platform                                    arch=arm os=linux pid=19719 revision=8fa89735 version=13.6.0
Running in system-mode.                                           
Unregistering runner from GitLab succeeded          runner=ugNjkgTF
Updated /etc/gitlab-runner/config.toml

1. Then delete it so it won’t appear in the list:

$ sudo gitlab-runner list
# Answer
Runtime platform   arch=arm os=linux pid=19991 revision=8fa89735 version=13.6.0
Listing configured runners ConfigFile=/etc/gitlab-runner/config.toml

# To verify delete
$ sudo gitlab-runner verify --delete
$ sudo service gitlab-runner restart

# To uninstall the runner executable
# sudo apt remove gitlab-runner

Advertisement