Monthly Archives: September 2021

k3s Raspberry Pi Kubernetes Cluster

Posted on by
1

[This post has been edited several times, sorry]

The objective of this post is to automate the creation of a Kubernetes cluster in a couple of Raspberry Pi using Ansible.

In the next two posts we will deploy two Apps using our GitLab Server:

  1. The first will run a NodeJS App in a kind Kubernetes cluster hosted in a laptop machine. There are alternatives like Minikube, k3d/k3s or MicroK8s.
  2. The second one will run a PHP-Angular-Redis App into the Raspberry’s cluster created in this post.

The Raspberry Pis 4 will use Ubuntu Server (arm64) and K3s as they are a light platform to build Kubernetes on.

An excellent guide to build a cluster is at blog.zachinachshon.com. He explains in step by step how to set up a Kubernetes cluster in a bare metal installation. You can follow the series or just the first steps until you have the Linux machines with password-less SSH log-ins.

You’ll also need to install Kubectl, Python and Ansible in the controlling machine (where you’ll run the scripts).

Note [December 2021]: If you use Ubuntu 20.10 you’ll also need to run the following command to install libraries that were removed – The answer was provided by ‘siretart’ in stackoverflow

$ sudo apt install linux-modules-extra-raspi

Then download and modify this official examples of Github scripts that will install k3s in AMD, Arm64 or Arm32 processors. Or use my scripts that are edited with just the Arm64 commands:

  • This files will execute k3s a shell install: here.
  • This files will firs download the k3s binary, set up the services configuration files and start the k3s or k3s-agent services in what is known as a binary k3s install: here.

I. Inventory & Variables

The inventory file is key to define the ‘groups’ of machines that are ‘k3s_cluster’ (all machines), or ‘master’ and ‘node’.

Modify the ./inventory/hosts.ini file with your IP’s

[master]
192.168.1.223

[node]
192.168.1.224

[k3s_cluster:children]
master
node

At the same folder level of the inventory file, we can read the ‘group_vars’ folder where we define the variables to be used on the playbook, those are mainly the version of k3s to install, and the user to log into the machines. That are: k3s_version (v1.17.5+k3s1) and ansible_user (that was changed it to ubuntu as it was the user that the installation created).

File: ./inventory/group_vars/all.yml

k3s_version: v1.21.4+k3s1
ansible_user: ubuntu
systemd_dir: /etc/systemd/system
master_ip: "{{ hostvars[groups['master'][0]]['ansible_host'] | default(groups['master'][0]) }}"
extra_server_args: ""
extra_agent_args: ""

II. Playbook

This file ‘site.yml’ doesn’t need editing it is shown to see the hole process. The Ansible script prepare the master & nodes (k3s_cluster in prereq). Configure the master and nodes including the repository. And finally copy the final kubeconfig from the master to the PC.

---

- hosts: k3s_cluster
  gather_facts: yes
  become: yes
  roles:
    - role: prereq

- hosts: master
  become: yes 
  roles:
    - role: install/master
    - role: k3s/master
    - role: post/repositories

- hosts: node
  become: yes
  roles:
    - role: install/node
    - role: k3s/node
    - role: post/repositories

- hosts: master
  become: yes
  roles:
    - role: post/master

III. The Ansible playbook

To run the ansible-playbook, In a terminal use the following command

$ ansible-playbook site.yml -i ./inventory/hosts.ini

IV. Check

Confirm that the ‘post’ step did copy the .kube/config file to the control PC so we can use kubectl commands to interact with the cluster.

$ kubectl get nodes

You have a working Raspberry K8s cluster.

Notes

Brief summary of tasks that are executed as listed in the ‘site.yml‘ file:

  • ./prereq/tasks/main.yml
    Will enable IPv4 and IPv6ip. For systems with CentOS or RedHat SELinux will be disabled, the netfilter and iptables will be reloaded and path will now contain /usr/local/bin.
  • ./download/tasks/main.yml
    k3s will be downloaded for the appropriate processor: AMD, arm64 o arm32 to /usr/local/bin/k3s
  • ./raspberri/tasks/main.yml
    These are server config related actions for the Raspberry: The CPU-Model version is ‘registered’ and accordingly actions are taken from a subdirectory (prereq) for: Default (no action), Ubuntu & CentOS. Configure cgroup if it is not already in cmdline.txt, Raspbian (iptables) and if needed, a reboot.
  • ./k3s/tasks/main.yml
    Some config ‘jinja’ templates are copied to the servers, including the ones to configure the gitlab repository where we store our docker images.
    Then it uses the downloaded k3s binaries as the ‘systemd’ k3s services are started in the master and nodes. A modified version is used where the environment variables are set into and additional file as the documentation allows.
    The previuos code uses the shell scripts to install k3s as it executes additional config steps for newer k3s than the template-binary option.
    In both notice how the nodes use the token stored as a variable in the master play.
  • ./post/tasks/main.yml
    Lastly the config file is configured (.kube/config) and copied to the machine running Ansible.

Note: There is also a reset.yml playbook file to erase the k3s installation.

Advertisement