I went RKE2, k3s is nice, but a little too minimal for my tastes. With a few hundred MB ram used, I've got an internal container registry, openbao for secrets, caddy for edge TLS, rabbitmq, and powerdns for exposing k8s ingress. Plus all the standard network policies, which while verbose, gets me nearly all the way there of traditional firewalls and networking.
You can pretty install it without ssh under 60s. The fun starts after it has been installed.
We have been running into lot of issues at production with k3s. There I embarked on journey to writing a kubernetes compliant and equivalent platform in rust with the help of claude [1]. It is a fun little project for now, still figuring out stuff, idea is to keep it minimal and single binary every embedded including CNI, and support various runtimes like docker, containerd etc but also wasm, vms and also jvm.
Do you have a writeup what problems you ran into?
We do, let me check with my team and post it here.
There were many issues. On top of my mind was, after a DR drill where in a VM was booted, node did not join the cluster. Apart from that bunch of issues due to etcd, longhorn.
Another major one was the CNI stopped work for a particular node. Garbage collection for images was another, we labelled the images, it would still remove then from the node.
Bunch of these kind of issues when our requirement is fairly straightforward. Therefore we are working towards a strip down version.
There is lot of operation complexity in general and most of us can do without.
the best part of k8s is network, most of agentic systems presume no network , since it's a security concern, what are scenarios when you'd like to spin up k3sup?
I use official ‘ansible-playbook k3s.orchestration.site -i inventory.yml’ and it installs k3s over SSH and adds it into my kubectl context, all under 60s too.
I have just been `ssh ... -- k3s.sh ...`, been meaning to ansible my homelab