Taking Back Control of Your IT: Exploring Open Source Alternatives

In the current geopolitical context, governance and sovereignty over your data and infrastructure have become increasingly important. Companies and organizations are looking for ways to maintain control over their IT environments, and on-premises infrastructure is becoming a trend again.

In this landscape, Proxmox VE is a powerful Open Source virtualization platform, based on battle-tested technologies like QEMU/KVM and LXC. It provides a web-based management interface, high availability clustering with live migration, hyper-converged storage with Ceph, amongst other features. It has become a popular production-ready alternative to proprietary virtualization solutions like VMware ESXi, especially after the licensing changes following Broadcom's acquisition of VMware.

On the other hand, Kubernetes is already more than 10 years old by now and, through its many flavours, is arguably the most popular container orchestration platform. However, managing and maintaining a Kubernetes cluster can be complex and time-consuming. This is where Talos Linux comes in.

Talos Linux is a modern, minimal, and secure operating system designed specifically for running Kubernetes clusters. It is built from the ground up to be immutable, meaning that the entire OS is read-only and cannot be modified. It is also configurable entirely through a declarative API, which makes it easy to manage and automate. This makes Talos Linux an excellent choice for organizations looking for security and efficiency.

At Camptocamp, we are always looking for ways to improve our infrastructures and for innovations that might benefit our clients. Open Source solutions are part of our DNA: we believe they provide transparency, flexibility, and community support that proprietary solutions often lack. So we tested Talos Linux, deploying it in a lab environment using Proxmox VE as a hypervisor, and running it on a Scaleway bare-metal server.

Let’s go over the architecture of our lab setup in broad strokes.

I was a doctor before starting my career in IT, so indulge me in my heart/arteries analogy below.

The Heart

We wanted a clean environment to test Talos Linux as if it was running on bare metal, trying to mirror the necessary network setup for an on-premises deployment as much as possible. In the end, we decided to install Talos Linux on virtual machines running on Proxmox VE, which is another technology we wanted to test and get familiar with.

There was just the question of where to run Proxmox VE. We could have used a local server, but we ended up on a Scaleway Elastic Metal instance. You can have instances with Proxmox VE pre-installed on them, and they can also be billed by the hour, which is great for lab environments.

This is the power behind our deployment, hence the heart.

The Arteries

For networking, along with the Talos Linux nodes, we have a virtual machine running pfSense installed on the same Proxmox VE host. This VM acts as a router and firewall for the Talos Linux nodes. It provides DHCP and DNS services, and manages the NAT for the nodes to access the internet. Internet connectivity is provided through a public Elastic IP address assigned to the pfSense VM.

Having a Kubernetes cluster by itself, means nothing if you cannot access its services. To expose services running on the Talos Linux cluster, we chose to install Cilium as a CNI plugin. Cilium allows us to configure a LoadBalancer service type and then announce the necessary routes using BGP to the pfSense router.

These are the arteries, transporting traffic from one point to another.

Finally, after much tinkering, we ended with something resembling the following architecture. In this scheme we are demonstrating how the six Talos nodes are deployed in a single Proxmox VE node and how they are interconnected and accessible to/from the internet through a pfSense firewall, also residing on the same Proxmox VE node.

Lessons from the Lab

Take a moment to appreciate what we've assembled. Every single piece of this setup, from the Proxmox VE hypervisor to the Kubernetes orchestrator, from the Talos Linux operating system to the Cilium networking layer, is a testament to the power of Open Source.

In our opinion, Proxmox VE is mature enough for production environments and for organizations searching for VMware ESXi alternatives. Additionally, Talos Linux will remove the headaches of managing the base operating system just to have a properly secured Kubernetes node. Finally, Cilium CNI will enable us to easily expose and announce our services in an on-premises environment.

A cloud-native stack with best-in-class tools for each job. This allows for incredible flexibility and innovation, freeing you from vendor lock-in.

Go deeper

If you are interested in a more detailed explanation of the setup along with some configuration examples and the reasoning behind some technical choices, please refer to this GitHub repository.

Finally, if you have any questions or want to discuss further, we would be happy to hear from you! Feel free to reach out to us through our contact page.