New Node

While the new Raspberry Pi 5 is running just fine and bringing much-needed power to the cluster, there’s still a problem—the local storage on the Pi is 97% full. When I deploy new services, Kubernetes first clears old images and then downloads new ones. So, it’s time to swap the 32GB microSD card for a beefier 128GB card.

Idea One – Cloning

The first idea was simple enough: clone the 32GB microSD card to the larger one and swap them out. From a storage-only perspective, this makes sense and is easy. My NAS regularly clones files to another storage location for safekeeping. But I’m not just cloning storage—I’m also cloning a running operating system.

That’s where things get interesting. I need to transfer data that is currently in use by running processes. Luckily, there are tools for this, such as Clonezilla (which I tested).

The basic process these tools follow sounds simple:

1. Copy all files from the original storage to the new storage. The challenge here is ensuring files in use are copied without breaking anything. However, this also copies unnecessary files like logs and temporary data.
2. Expand the partition on the new storage device to utilize the full available space. The partition also needs to be made bootable.

Here’s what I noticed: you end up with the old system running on new hardware. That has some unexpected implications. For example, many services remain configured for the smaller storage capacity.

MicroK8s (my Kubernetes stack) has a “prune image” setting that determines when old images should be removed based on available storage at the time of installation. So even though the Pi now had more space, it still behaved as if it didn’t.

In the end, I decided to wipe the new microSD and do a fresh install of Ubuntu Server. I’m sure there’s a way to clone the OS without issues, but I wasn’t comfortable with the process.

Idea Two – Fresh Installation

A clean install was easy. The longest step was waiting for the Ubuntu image to flash onto the microSD card. After that, I installed MicroK8s and set up a swap file. This time, I could allocate more swap space than before—but more on that later.

Swapping out a Kubernetes node turned out to be easier than I expected:

1. Stop MicroK8s on the “old” node. Since the Raspberry Pi 5 had become the master node (remember Plug Day?), I had to wait a few seconds for the cluster to adjust.
2. Once MicroK8s was stopped, the original Raspberry Pi automatically reassigned itself as the master node (since it was the only one left). I then removed the old node from the cluster, temporarily leaving me with a single-node setup.
3. Swap out the microSD card, reinstall everything, and re-add the Pi 5 to the cluster. Since its hostname and IP address remained the same, the cluster instantly recognized it.

Interestingly, I didn’t need to redeploy any services. ArgoCD treated the new node as if the old one had simply come back online. This means that even though I manually removed the node, ArgoCD still remembered it. I’m not sure if ArgoCD would eventually forget an old node, but in my case, it restarted all the affected pods automatically.

Now, the cluster is back to two nodes, and Grafana reports everything as healthy. After stabilizing, the new node showed 27% storage usage (compared to 97% before). With 128GB of total storage, it now uses 31.5GB—almost identical to the old 32GB card’s usage, minus system bloat from long-term operation.

Idea Three – Automating the Process

While swapping out a node was straightforward, I had to look up commands and manually check the versions of critical components. MicroK8s should have the same version on all nodes, and configuring hostnames, static IPs, and firewall rules needs to be done consistently. Could I over-engineer a solution to automate this? Obviously, yes.

I documented all the steps I performed and turned them into an Ansible script. Now, adding a new node to the cluster is much easier.

Ansible works by running commands from a control host (any device in my network with Ansible installed) over SSH. The cool part? The script can also execute commands on other machines, like adding firewall exceptions for the Pi’s IP.

But this script has massive privileges in my network—it can modify firewall rules and run commands on Kubernetes nodes. I don’t want that kind of power just lying around. Thankfully, I have a solution: a Raspberry Pi Nano.

Years ago, a friend gifted me a Raspberry Pi Nano. It’s a tiny version of a normal Raspberry Pi—not very powerful, but perfect for running Ansible. Now, only the Nano holds the necessary SSH keys, and it stays safely tucked away and offline most of the time.

On Swap Files

Some technical readers might wonder why swap is so important to me.

Swap extends memory by allowing the system to use disk space as slow RAM. While Kubernetes doesn’t use swap (for good reasons), I still find it valuable.

When developing applications, they aren’t always memory-optimized. If a process isn’t actively using some memory, it gets moved to swap. This helps prevent crashes from memory exhaustion.

The Raspberry Pis barely use their swap, but it’s nice to have. With 128GB of storage, dedicating 16GB for swap is no big deal. One quick note: Grafana already deducts the swap space from total storage, so my nodes now show 117GB of available storage instead of 128GB.

Conclusion

Maybe using the wrong microSD card wasn’t such a bad mistake after all. At least now I have an Ansible script to set up new nodes if I ever add more.

And best of all? I finally have enough storage for fully green Grafana monitoring.

homelab svartalf