[This is part 5/8 on GitLab Continuous Integration series]
An official definition:
runners are the actual elements that do the work in a CI setup. They are assigned CI/CD jobs by the server according to a pipeline script that if all is set properly runs automatically when a
pushed to the server or manually if needed.
I found that very confusing so I like to explain those elements as follows:
- The GitLab Runner is an agent installed on a different machine from the GitLab server. It is the one performing the scripts.
- The ‘environment’ that a Runner uses to perform a task is the Runner Executor declared hwen we create the runner an written in its configuration file. It can do the tasks with [local or remote] shell commands, send ssh commands to servers, or in a virtual machine. The preferred one uses a docker container that you choose as it has the software needed (can be python or maven), or be a pod in a Kubernetes cluster that deploys and also monitors the jobs. You can mix Executors in your jobs on each stage.
- The Runner is Registered in the GitLab Server according the projects it can run. They can be Shared (for unassigned projects), Group or Project-specific (can be more secure as environment variables are only shared with our code).
- GitLab Job: Is the YAML script or pipeline, which contains one or more commands that need to be executed. You can use variables that you define or get from the server, pass objects from stage to stage, etc.
I. What we need
For automation using GitLab CI we need:
- To install an
GitLab runnerthat will do the actual work aka
jobs. To distribute load they should be installed on a machines that are separated from the Server
- Then register that
GitLab runnerto the GitLab Server or
coordinator. In the server the runners can be configured so they can be assigned all or specific pipelines.
- Verify that running ‘CI pipelines’ are enabled in our server and project.
- Add a
pipeline scriptwith stage our jobs. It is usually located in the project’s root directory. It is normally called
.gitlab-ci.yml. Yes, a hidden file.
As the documentation states, there are several types of runners that can be used to run your builds for different scenarios.
|Clean build environment for every build||✗||✗||✓||✓||✓|
|Reuse previous clone if it exists||✓||✓||✗||✓||✗|
|Runner file system access protected||✓||✗||✓||✓||✓|
|Complicated build environments||✗||✗ (1)||✓ (2)||✓||✓|
|Debugging build problems||easy||easy||hard||medium||medium|
|Note||runs each build in an isolated container||Creates a Pod for each Job|
And each one supports different features:
|GitLab Runner Exec command||✗||✓||✗||✓||✓|
|Passing artifacts between stages||✓||✓||✓||✓||✓|
|Use GitLab Container Registry private images||n/a||n/a||n/a||✓||✓|
|Interactive Web terminal||✗||✓ (UNIX)||✗||✓||✓ (Planned for Helm)|
We choose to use the Docker Executor as it offers more features, specially that it allows access to the private Container Repository. And further ahead it will allow also the use of
Docker in Docker (DinD) services.
II. Install a GitLab Runner
GitLab Runner we will install it as a Docker container. We can use Linux’s apt or download it. In my case, to follow the recommendation to install it in another machine than the server, I installed it in my laptop computer.
$ cd ~/Descargas<br> $ sudo apt update
This following is the correct command but it didn’t work at the end on 2020 in my Ubuntu/Debian 20.10 Linux Laptop. It might work by the time you are reading it:
$ sudo curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
Note: By January 2021 there is no code for installing a
gitlab-runner in ‘Groovy-Gorilla’ (Ubuntu 20.10). The error shown is:
[Edit on dec 2021 the command runs fine]
Error as Groovy-Gorilla Ubuntu doesn't have a candidate. Unable to download repo config from: https://packages.gitlab.com/install/repositories/runner/gitlab-runner/config_file.list?os=Ubuntu&dist=groovy&source=script You can override the OS detection by setting 'os=' and 'dist=' prior to running this script. You can find a list of supported OSes and distributions on our website: https://packages.gitlab.com/docs#os_distro_version
We can see in the list of official GitLab Runner repositories: Xenial (16.04), Bionic (18.04) and Focal (20.04). Considering tat the forums are full of reports for 19.03 we can assume that our 20.10 version will show them also. So we have Focal or Bionic as candidates.
Now we check the availability of our the Docker we will use in our CI Pipeline. For that DinD, the hub.docker list contains (20.10.3-dind, 20.10-dind, 20-dind, dind, 20.10.3-dind-rootless, 20.10-dind-rootless, 20-dind-rootless, dind-rootless, 19.03.15-dind, 19.03-dind, 19-dind, 19.03.15-dind-rootless, 19.03-dind-rootless, 19-dind-rootles).
In that Docker Hub page you can read two warnings:
- Although running Docker inside Docker is generally not recommended, there are some legitimate use cases, such as development of Docker itself. => We have to insist 🙂
- Starting in 18.09+ the dind image will automatically generate TLS certificates in the directory specified by the
DOCKER_TLS_CERTDIRenvironment variable. It is recommended to enable TLS by setting the variable to an appropriate value (
-e DOCKER_TLS_CERTDIR=/certs) => So we have to remember to disable that in our runner in the next section
After updating the command with an available distribution (focal=Ubuntu 20.04):
$ sudo curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo os=ubuntu dist=focal bash # We are advised to use this for Debian Buster (Ubuntu 20.10) so it can create empty folders: # (-E preserves environment variables) $ export GITLAB_RUNNER_DISABLE_SKEL=true; sudo -E apt install gitlab-runner
Check the Runner
To check the runner we can list them:
$ sudo gitlab-runner list # Answer Runtime platform arch=amd64 os=linux pid=21747 revision=8fa89735 version=13.6.0 Listing configured runners ConfigFile=/etc/gitlab-runner/config.toml docker-runner Executor=docker Token=o63mEQt4q-RJnj3yQNhW URL=http://gitlab.example.com/
get its version:
$ sudo gitlab-runner --version # Answer Version: 13.7.0 Git revision: 943fc252 Git branch: 13-7-stable GO version: go1.13.8 Built: 2020-12-21T13:47:06+0000 OS/Arch: linux/amd64
and ask for a quick verification:
$ sudo gitlab-runner verify # Answer Runtime platform arch=amd64 os=linux pid=70447 revision=775dd39d version=13.8.0 Running in system-mode. Verifying runner... is alive runner=xmWbMf4w
II. Register the Runner
Registering a Runner will bind the
GitLab Runner with the
GitLab Instance. There are three types of bindings:
- Shared that are available and runs jobs from all unassigned projects. In cloud services they are time limited. This is the one we will be using in the example.
- Group or
- Project-specific these can be more secure as environment variables are only shared with our code.
In the use the Docker executor Gitlab page we can find additional warnings:
- Each job is in a clean environment without the past history, there’s no caching of layers.
- A docker-privileged executor as
--privilegedis required for Docker-in-Docker. That is what we will use to package our app as a docker image and upload them to a repository. Bun if you enable
--docker-privilegedyou are effectively disabling all of the security mechanisms of containers and exposing your host to privilege escalation.
- By default, Docker 17.09 and higher uses
--storage-driver overlay2which is the recommended storage driver.
- Since the
docker:19.03.12-dindcontainer and the runner container don’t share their root file system, the job’s working directory can be used as a mount point for child containers. For example, if you have files you want to share with a child container, you may create a subdirectory under
/builds/$CI_PROJECT_PATHand use it as your mount point (check issue #41227):
variables: MOUNT_POINT: /builds/$CI_PROJECT_PATH/mnt script: - mkdir -p "$MOUNT_POINT" - docker run -v "$MOUNT_POINT:/mnt" my-docker-image
Example project using this approach: https://gitlab.com/gitlab-examples/docker.
Back to the post. We are configuring an easy to use insecure CI Server. And we will be using the privileged attribute that can be exploited by running software to gain access to your machine. But we know what we are doing and love living dangerously. 🙂 (No, that is a joke, we have our Server behind our home firewall).
To register the runner we need data from the ‘shared runner’ page. Log in as Admin user into the server and go to: ‘Admin area’ > Overview > Runners Copy
registration-token, and paste those values in the following command:
$ sudo gitlab-runner register \ --non-interactive \ --url "http://gitlab.example.com/" \ --registration-token "QR6jYXGZdd7xHpsy1K5b" \ --executor "docker" \ --docker-image docker:stable \ --description "docker-runner" \ --docker-privileged \ --tag-list "master" \ --run-untagged="true" \ --locked="false" # access-level was added for version 12 # Answer Runtime platform arch=amd64 os=linux pid=21631 revision=8fa89735 version=13.6.0 Running in system-mode. Registering runner... succeeded runner=7f4nVsxX Runner registered successfully. Feel free to start it, but if its running already the config should be automatically reloaded!
Note: You can use an interactive version, that is started with:
$ sudo gitlab-runner register # Answers you will need to provide: http://gitlab.example.com/ QR6jYXGZdd7xHpsy1K5b gitlab-runner <Enter> docker docker:stable
Check if the Runner is registered
To list runners registered and available in the GitLab Server:
- As a user, go to Settings > CI/CD and expand the Runners section.
- As an administrator (root), for a
Shared Runnergo to ‘Admin Area’ (click
wrenchicon on toolbar) > Overview > Runners
Fine Tuning the System
As it is, the runner can run common shell commands like
maven clean or
maven build, but it will break when using the Docker file system to send files (
artifacts) between stages or posting files to the GitLab
Docker repository. We need to make some configuration modifications so edit the config file to add a few lines and modify a Docker tag line:
- add ‘clone_url’ that holds the IP of the GitLab server as a backup if it can’t find our ‘gitlab.example.com’ url:
clone_url = "http://192.168.1.221/"
- add also extra_hosts with the GitLab Server and repository names and IPs so the runner can find them:
extra_hosts = ["gitlab.example.com:192.168.1.221","registry.example.com:192.168.1.221"]
- Lets share the docker insecure registry configuration with our Docker that runs our jobs in the CI:
volumes = ["/etc/docker/daemon.json:/etc/docker/daemon.json","/cache"]
- In the ‘image’ change “docker:latest” to:
image = "docker:stable"
- Lets turn off TLS for our insecure setup:
environment = ["GIT_SSL_NO_VERIFY=true", "DOCKER_TLS_CERTDIR="]
The resulting file will be:
$ sudo cat /etc/gitlab-runner/config.toml # Content concurrent = 1 check_interval = 0 [session_server] session_timeout = 1800 [[runners]] name = "docker-runner" url = "http://gitlab.example.com/" clone_url = "http://192.168.1.221/" token = "xmWbMf4wYFsPPGq7gJFR" executor = "docker" environment = ["DOCKER_TLS_CERTDIR=","GIT_SSL_NO_VERIFY=true"] [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure] Insecure = true [runners.docker] tls_verify = false image = "docker:stable" privileged = true disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = false volumes = ["/etc/docker/daemon.json:/etc/docker/daemon.json","/cache"] shm_size = 0 extra_hosts = ["gitlab.example.com:192.168.1.221","registry.example.com:192.168.1.221"]
Don’t forget to restart the runner service to update it:
$ sudo service gitlab-runner restart
That’s it. You’ve got a CI server ready to work.
Remove a GitLab Runner
If later on you need to remove a runner you can click on the Runner page of the server or:
- First unregister it in the GitLab Server or in the CLI:
$ sudo gitlab-runner unregister --token ugNjkgTFHcndF4_btFuQ --url http://gitlab.example.com/ # Answer Runtime platform arch=arm os=linux pid=19719 revision=8fa89735 version=13.6.0 Running in system-mode. Unregistering runner from GitLab succeeded runner=ugNjkgTF Updated /etc/gitlab-runner/config.toml
- Then delete it so it won’t appear in the list:
$ sudo gitlab-runner list # Answer Runtime platform arch=arm os=linux pid=19991 revision=8fa89735 version=13.6.0 Listing configured runners ConfigFile=/etc/gitlab-runner/config.toml # To verify delete $ sudo gitlab-runner verify --delete $ sudo service gitlab-runner restart # To uninstall the runner executable # sudo apt remove gitlab-runner