I’ve been working a lot with different Kubernetes clusters in the last couple of months. More and more applications these days are required to run in containers, and Kubernetes is the de-facto orchestration tool for these kinds of things. There are various cloud offerings you can use for relatively low cost, however you’ll never be in full control of your cluster; intermittent issues do happen (network / DNS), and reasons for that are always different depending on which cloud service you use. Naturally, I wanted to build my own lab cluster to learn more about Kubernetes and its inner workings. And if things go south I can jump in and find out why and fix it.

Setup⌗

We have two major goals in mind for this project:

  • Our cluster should be resilient to failure
  • Network traffic between our nodes should be encrypted

The first bullet point speaks for itself; we’re in the ‘cloud’ after all. Our cluster should handle failure and minimize downtime for consumers of your services. And since we’re installing our cluster on bare metal (or VM’s) and (at least in my case) that is not owned by me directly, I want to encrypt traffic between all potential nodes.

There are a couple of providers to choose from if you want to rent a cheap VPS or two - I can highly recommend Hetzner or Contabo for their reliable services. Of course you can build this cluster in your home lab as well if you so choose.

Prerequisites⌗

To run a highly available Kubernetes cluster we need the following:

  • An odd number of server nodes (we use three here)
  • A domain under which we can make our cluster available from
  • Each server requires a unique hostname

We going to use the embedded etcd database. This database uses the Raft Consensus algorithm to manage quorum between the master nodes. You can find out more about the topic on why an odd number of cluster members is necessary here.

Our architecture diagaram is going to look like this once we’re done:

m a k s 3 t s e r - 1 m u a s k s e 3 t r s e r h - t 2 t p s : / / e x a m m a p k s l 3 t e s e . r c - o 3 m

All our master nodes are interconnected and our users will connect to https://example.com and will be able to consume our services from there. Neat!

We’re not going to talk much about load balancing in front of our cluster here. I recommend some form of load balancing in front of our cluster, but having a CNAME record for all A records containing the external server IPs is totally fine for now.

Once your machines are ready to go we have everything ready to get started!

I recommend reading through resources like this to properly secure your machines if you plan on using the cluster for live services.

Wireguard⌗

After all VM’s (or whatever you ended up provisioning) are installed and ready we’re going to setup our Wireguard network. Wireguard gives us the ability to create a VPN that will take care of the encrypted communication between our master nodes. There are options to automate setting up this network (Netmaker for example); hoewever we’re going to do it from scratch.

Log into your first master server - we’re going to start with the installation of wireguard there:

I’m using Debian 11 as the operating system and systemd as the system manager in this example; note that some commands may differ if you’re using a different setup.

sudo apt update && sudo apt install wireguard -y

In this example we’re going to use 10.222.0.0/30 as our address range here for our master nodes.

Wireguard requires by default that port 51871/udp is allowed for incoming and outgoing traffic. We also want to allow all incoming traffic from our Wireguard address range.

ufw allow 51871/udp
ufw allow in from 10.222.0.0/30

Next we need to create a private key with a matching public key to configure our network interface.

wg genkey | tee private.key | wg pubkey > public.key

And finally we’re going to create the configuration file for our interface (/etc/wireguard/wg0.conf):

[Interface]
Address = 10.222.0.1
ListenPort = 51871
PrivateKey = <private-key-master-1>

Let’s flip the switch and start the interface!

sudo systemctl enable wg-quick@wg0
sudo systemctl start wg-quick@wg0

Your server will now start start the Wireguard (wg0) interface after every reboot. The interface is now ready to go:

wg show wg0
ping 10.222.0.1

The first command will show you the registered configuration of the interface, and if all went well the packets will come back from the ping command.

Connecting our peers to the network⌗

The next step would be to essentially do the same we just did on our first master node and replicate that setup on the other nodes. Once you have Wireguard installed and the firewall configured on your second and third master node we’re going to configure their wg0 interface as follows:

Here is our configuration for our second master node:

[Interface]
Address = 10.222.0.2
ListenPort = 51871
PrivateKey = <private-key-master-2>

[Peer]
Endpoint = <external-ip-master-1>
PublicKey = <public-key-master-1>
AllowedIPs = 10.222.0.1

[Peer]
Endpoint = <external-ip-master-3>
PublicKey = <public-key-master-3>
AllowedIPs = 10.222.0.3

… and here is our configuration for the third master node:

[Interface]
Address = 10.222.0.3
ListenPort = 51871
PrivateKey = <private-key-master-3>

[Peer]
Endpoint = <external-ip-master-1>
PublicKey = <public-key-master-1>
AllowedIPs = 10.222.0.1

[Peer]
Endpoint = <external-ip-master-2>
PublicKey = <public-key-master-2>
AllowedIPs = 10.222.0.2

You also need to update the configuration on your first master node accordingly - this is left as an exercise for the reader.

Once you’re done make sure that you restart your Wireguard interfaces on all nodes:

sudo systemctl restart wg-quick@wg0

The last thing we need to verify is that you can ping all your nodes on the internal network. For example (if you’re logged into your first master node):

ping 10.222.0.2
ping 10.222.0.3

… should successfully ping all your nodes on your Wireguard network! Congratulations!

Summary⌗

We’ve accomplished quite a bit - we know what our infrastructure is going to look like, how we’re going to build it and and we have our own VPN setup and ready to. In the next part we’re going to actuall install the k3s cluster and deploy some applications.