October | 2021 | Interesting IT Tip's

This post is an add-on to the series as I tried to do a “LAMP automated deployment from a GitLab repository” using Ansible but the information I found online was too difficult to understand or was outdated.

Lets start with a LAMP Ansible playbook example from the official example repository and let’s modify it so it can upload it to a GitLab Repository and execute the deployment to set up a Web server in one machine and a MariaDB Server in other (two Raspberry Pi’s).

There are two things that we have to consider:

ONE
GitLab currently doesn’t have built-in support for managing SSH keys in a build environment (where the GitLab Runner runs). The most widely supported method is to inject an SSH key into our .gitlab-ci.yml.

A common way to make secrets available to containers is using a volume mounted into the container at runtime. Volumes can be added to the executor through the GitLab Runner configuration.

Another approach would be using GitLab’s Secure Variables to store the private key and configure the GitLab Runner with a pre_build_script that adds the SSH key to the ssh-agent.

But the most widely supported method so GitLab can provide them is to inject the SSH key into your build environment (in the.gitlab-ci.yml). This solution works with any type of executor.
We will supply the information from Secure Variables.

TWO
The docker image we configured in the runner it is a basic Docker image. The appropriate way is to use an image that has the necessary software, that is Ansible, Python and the MySql utilities to configure the database.

I browsed what was available in hub.docker.com. This one from mullnerz has Ansible 2.9 and Python 3.8. That is good enough for our project. As shown in:

$ docker run --rm --name ansible mullnerz/ansible-playbook ansible --version
ansible 2.9.6
python version = 3.8.2 (default, Feb 29 2020, 17:03:31) [GCC 9.2.0]

I. Create a GitLab Project

We need a repository for our code the GitLab Server. Lets create one:

Browse to our local
http://gitlab.example.com
or your GitLab, GitHub code service.
We use devguy/Clave123 to log in (Credentials shown as reference as it is used in the url)
Click on right blue button: New project’ > Create blank project
Project name: infra
Visibility level: Public
Create readme file: no
Click on ‘Create project’ button
We can get access to it opening the url:
http://gitlab.example.com/devguy/infra

II. Check The Runner

We have set up previously a Docker executor to run our CI/CD jobs. To verify that the Docker Runner is still running as we configured it:

[On the PC]

$ sudo gitlab-runner verify
Runtime platform              arch=amd64 os=linux pid=50507 revision=7a6612da version=13.12.0
Running in system-mode.
Verifying runner... is alive  runner=ejJb1t1N

It seems ok. Let’s continue

III. Set The Log In Keys

Gitlab writes that: “When your CI/CD jobs run inside Docker containers and you want to deploy your code in a private server, you need a way to access it. In this case, you can use an SSH key pair.”

Create two new CI/CD variables.

Access our GitLab Server in
http://example.gitlab.com
log in as devguy/Clave123
(This are the url and credentials used in previous posts)
Go to Project > infra
Then Settings > CI/CD
Expand Variables, and click ‘Add Variable’ button
As Key enter the name SSH_PRIVATE_KEY
and in the Value field paste the content of your private key
To get the private key you can use the following command (copy the whole answer):
$ cat ~/.ssh/id_rsa
—–BEGIN OPENSSH PRIVATE KEY—–
…
—–END OPENSSH PRIVATE KEY—–
In all variables set as Type: Variable
Environment scope: all
In Flags uncheck Protect and Mask
Click Create variable button
The second will hosts a list of all your hosts:
key: SSH_KNOWN_HOSTS
value: output of the command ssh-keyscan <IPS-list>
(It will be a big string. Copy the answer):
$ ssh-keyscan 192.168.1.223 192.168.1.224
<<output>>

III. Modifying the code

We need to create one file and modify an existing one.

ONE
Create the .gitlab-ci.yml pipeline file (the official usage example is here).

$ cd <your-root-dir>/Ansible/105_IaC_with_GitLab/Infra

$ vi .gitlab-ci.yml

Add as content:

image:
    name: mullnerz/ansible-playbook

before_script:
  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client git -y )'
  - eval $(ssh-agent -s)
  - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh
  - echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
  - chmod 644 ~/.ssh/known_hosts

deploy_app:
  stage: deploy
  script:
    - 'ansible-playbook -i hosts site.yml'

Notes to the before_script section:

Install ssh-agent if not already installed
Run ssh-agent (inside the build environment)
Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store Use tr to fix line endings which makes ed25519 keys work without extra base64 encoding the ssh-add - command does not display the value https://gitlab.com/gitlab-examples/ssh-private-key/issues/1#note_48526556
Create the SSH directory and give it the right permissions

TWO
The executor runs Python 3.8 but the MySql utility used in the Ansible example uses Python 2.7 version (mysql-connector-python). So we need to replace it with python3-mysqldb
Edi the file: ./Ansible/IaC_with_GitLab/infra/roles/db/tasks/main.yml

- name: Install MariaBD package
  apt:
    name:
    - 'python3-mysqldb'
    - 'mariadb-server'
    state: present

IV. Upload the Code Sources

[In the PC]

$ mkdir ~/Ansible/IaC_with_GitLab/infra

$ cd /Ansible/IaC_with_GitLab/infra

$ git config --global user.name "devguy"
$ git config --global user.email "devguy@gitlab.example.com"

# Start a politically correct init repository (no master)
$ git init
$ git remote add main http://devguy:Clave123@gitlab.example.com/devguy/infra.git

# Add the files
$ git add .
$ git commit -m "Initial commit"
$ git ls-tree -r main

Push the local repo branch under <local_branch_name> to the remote repo at <remote_name>.

$ git push main main
Enumerating objects: 38, done.
Counting objects: 100% (38/38), done.
Delta compression using up to 4 threads
Compressing objects: 100% (26/26), done.
Writing objects: 100% (38/38), 5.52 KiB | 807.00 KiB/s, done.
Total 38 (delta 0), reused 0 (delta 0), pack-reused 0
remote: 
remote: To create a merge request for master, visit:
remote:   http://gitlab.example.com/devguy/infra/-/merge_requests/new?merge_request%5Bsource_branch%5D=master
remote: 
To http://gitlab.example.com/devguy/infra.git
 * [new branch]      master -> master

$ git status
On branch main
nothing to commit, working tree clean

If a typo was made and a file needs updating the following sequence can be used:

# Make the editing
$ vi .gitlab-ci.yml
# Update the remote repository
$ git add .gitlab-ci.yml
$ git commit -m "Adding IaC CI pipeline"
$ git push main main

V. Automatic Pipeline Execution

The output in the GitLab Server is

Running with gitlab-runner 13.12.0 (7a6612da)
on docker-runner ejJb1t1N
Preparing the "docker" executor 00:02
Using Docker executor with image mullnerz/ansible-playbook …
...
Getting source from Git repository 00:02
...
$ which ssh-agent || ( apt-get update -y && apt-get install openssh-client git -y )
/usr/bin/ssh-agent
$ eval $(ssh-agent -s)
Agent pid 11
$ echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
Identity added: (stdin) (xxxxx@pcxxx)
$ mkdir -p ~/.ssh
$ chmod 700 ~/.ssh
$ echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
$ chmod 644 ~/.ssh/known_hosts
$ ansible-playbook -i hosts site.yml
PLAY [apply common configuration to all nodes] *********************************
TASK [Gathering Facts] *********************************************************
ok: [192.168.1.223]
ok: [192.168.1.224]
...
PLAY RECAP *********************************************************************
192.168.1.223 : ok=14 changed=4 unreachable=0 failed=0 skipped=0    rescued=0 ignored=0   
192.168.1.224 : ok=16 changed=2 unreachable=0 failed=0 skipped=0    rescued=0 ignored=0   
Job succeeded

IV. Test

Test in a browser

http://192.168.1.223/index.html

Hello World! My App deployed via Ansible V6.

and

http://192.168.1.223/index.php

Homepage
Hello, World! I am a web server configured using Ansible and I am : kmaster
List of Databases:
foodb information_schema mysql performance_schema

Cleanup(Manual)

There is an Ansible playbook to clean up our servers:

$ cd ~/Ansible/IaC_with_GitLab/infra

$ ansible-playbook -i hosts reset.yml

Notes to myself but hope they can be of use to other curious people.
The code pruned for the Raspberry Pi’s is here.

Interesting IT Tip's

IT & IS articles that might work for you

Monthly Archives: October 2021

Run a CD pipeline for MariaDB and Web App.