Skip to main content

Setting Up a K3S Cluster

· 4 min read
tanzhuo

I've been learning how to deploy K8S clusters recently and tried quite a few approaches along the way: the official kubeadm tool, QingCloud's KK one-click installer, and various open-source (free and paid) one-click deployment tools on GitHub. After trying them all, I settled on K3S-up for deploying my cluster.

Why K3S​

Compared to full K8S, K3S runs far fewer system containers and is much more lightweight. It uses containerd as the container runtime by default, ships traefik (written in Go) as its built-in ingress, and bundles SQLite as a replacement for etcd. That said, with multiple master nodes you should use etcd to guarantee data consistency and achieve HA.

Environment Planning​

For this setup I used six shared-type servers on Alibaba Cloud to build an HA k3s cluster backed by embedded etcd:

  • 3 masters (2 cores, 4 GB RAM)
  • 3 workers (2 cores, 4 GB RAM)

All running Debian 10.10.

Prerequisites​

  1. Disable the firewall and update the package index:
apt-get update
  1. Install cgroupfs-mount, then reboot the machine:
apt-get install cgroupfs-mount

  1. Download the K3s binaries. Since k3s is hosted on GitHub, downloads from servers in mainland China can be painfully slow. It's better to grab the k3s binary from GitHub first and copy it onto the servers yourself, saving the download time during installation.

  2. Verify the environment:

k3s help          # show basic commands
k3s check-config  # check whether the environment meets the requirements

Repeat steps 1-4 on every server. If that sounds tedious, set up one server first, snapshot its system disk as an image, and select that image when creating the remaining cloud servers.

Deploying the Cluster​

  1. Once the environment checks pass, use k3sup to initialize the first control-plane master node:
# --cluster starts the server in cluster mode with embedded etcd
k3sup install --ip #{your_service_ip} --user root --cluster --k3s-version v1.23.4+k3s1 --print-command
  1. Join the other master nodes to the cluster:
# --server means join as a master; --server-ip points to the first master
k3sup join --ip #{next_service_ip} --user root --server-user root --server-ip #{first_service_ip} --server --k3s-version v1.23.4+k3s1 --print-command
  1. Join the worker nodes:
k3sup join --user root --server-ip $SERVER1 --ip $AGENT1 --print-command
  1. Check node status:
kubectl get nodes

Common k3sup Flags​

  • --cluster — start this server in cluster mode using embedded etcd (embedded HA)
  • --skip-install — if k3s is already installed, run this to just fetch the kubeconfig
  • --ssh-key — specify the SSH key path for remote login
  • --local-path — set where the kubeconfig is saved; defaults to ./kubeconfig, which gets overwritten by default
  • --merge — merge the config into an existing file instead of overwriting it, e.g. add it to the default kubectl config: --local-path ~/.kube/config --merge
  • --context — set the kubeconfig context name; defaults to default
  • --ssh-port — specify the SSH port; defaults to 22
  • --k3s-extra-args — optional extra arguments passed to the k3s installer, wrapped in quotes, e.g. --k3s-extra-args '--no-deploy traefik' or --k3s-extra-args '--docker'; combine multiple arguments inside a single pair of quotes: --k3s-extra-args '--no-deploy traefik --docker'
  • --k3s-version — pin a specific k3s version, e.g. v1.21.1
  • --k3s-channel — set the k3s version by channel, e.g. stable
  • --ipsec — force k3s to use --flannel-backend ipsec
  • --print-command — print the command being sent to the remote machine over SSH
  • --datastore — pass a SQL connection string to k3s's --datastore-endpoint; the format must follow what k3s requires per the Rancher docs

Cluster Dashboard​

k3s pairs well with kuboard. Kuboard dashboard docs: Kuboard Overview | Kuboard

Wrapping Up​

With that, we have a working k3s cluster that can scale out as the workload grows. The whole process runs remotely over SSH via k3sup, which is a lot less hassle than running kubeadm by hand, and it makes repeated rebuilds practical even under mainland China's network conditions. K3S itself is lightweight, and with embedded etcd it delivers master-node HA, making it a solid choice for small clusters. Next up: wiring in monitoring and a service mesh stack.

COMMENTS