How to switch container runtime in a Kubernetes cluster

#kubernetes#docker#devops#linux

As you might know,Kuberneteshas deprecatedDockeras container runtime, and Docker support will be removed in next versions (currently planned for the 1.22 release in late 2021).

If you are using a managed Kubernetes cluster (like GKE, EKS, AKS) you shouldn't have a lot to handle and it should be pretty straight forward for you. But if you are managing a cluster by yourself (withkubeadm

for example) and use Docker as container runtime, you will have to handle that runtime switch soon or later to keep enjoying Kubernetes updates.

The aim of this post is not to deep dive into the reasons of that change introduced by Kubernetes, or deep dive into container runtime behaviour in a Kubernetes cluster, but tostep by step

describe how to switch your container runtime from Docker to any runtime that implementsContainer Runtime Interface(CRI). If you need more details on the reasons which lead to Docker deprecation, you can read Kubernetes Blog postDon't Panic: Kubernetes and Docker

What to check in the first place

Appart from the changes linked to Kubernetes installation itself, the impacts on the workloads running in your cluster should be limited, if not non-existent. One of the only thing you have to care about is if you are usingDocker-in-Docker

in any of your container workload by mounting theDocker socket/var/run/docker.sock. In that case you will have to find an alternative (Kanikofor example) before switching from Docker to your new container runtime.

It's also warmly advised tobackupyour data before proceeding with the container runtime switch!

Let's proceed with the changes !

Ok now that you are ready to apply the container runtime switch, let's proceed with the changes. I will usecontainerdas container runtime in this post but the steps below can be adapted to any container runtime (likeCRI-O)

We will first start by impacting allworker nodes, and then finish by thecontrol plane.

Worker nodes

The steps below have to be applied oneachworker node.

1.First we willcordonanddrainthe node so that no more workload will be scheduled and executed on the node during the procedure.

kubectl cordon <node_name>
kubectl drain <node_name>

Remark: if you haveDaemonSetsrunning on the node, you can use the flag--ignore-daemonsetsto proceed with the drain without evicting the pods linked to your DaemonSet (which is by the way impossible with thedraincommand). Don't worry, these pods will be automatically restarted bykubeletat the end of the procedure with thenewcontainer runtime. If you have critical workload linked to the DaemonSets and don't want to let them run during the process, you can either specify anodeSelectoron your DaemonSet or completely uninstall and reinstall them at the end of the process.

2.Once the node is drained,stopthekubeletservice:

sudo systemctl stop kubelet
sudo systemctl status kubelet

3.Uninstall Docker.
I will not detail the commands here as it depends on your Linux distribution and the way you have installed Docker. Just be carefull if you want completely clean Docker artifacts, you might have to manually remove some files (for example/var/lib/docker)

You can checkDocker documentationto help you uninstalling the engine.

4.Installcontainerd(same here, I let you choose your favorite way to install it followingcontainerd documentation)

5.EnableandStartcontainerd service

sudo systemctl enable containerd
sudo systemctl start containerd
sudo systemctl status containerd

6.Kubernetes communicates with the container runtime through theCRI plugin. Be sure this plugin is not disabled in your containerd installation by editing the config file/etc/containerd/config.tomland check thedisabled_pluginslist:

disabled_plugins = [""]

Thenrestartcontainerd service if needed

sudo systemctl restart containerd

7.Editkubeletconfiguration file/var/lib/kubelet/kubeadm-flags.envto add the following flags toKUBELET_KUBEADM_ARGSvariable (adapt container-runtime-endpoint path if needed):

--container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock

8.Startkubelet

sudo systemctl start kubelet

9.Check if the new runtime has been correctly taken into account on the node:

kubectl describe node <node_name>

You should see the container runtime version and name:

System Info:
  Machine ID:                 21a5dd31f86c4
  System UUID:                4227EF55-BA3BCCB57BCE
  Boot ID:                    77229747-9ea581ec6773
  Kernel Version:             3.10.0-1127.10.1.el7.x86_64
  OS Image:                   Red Hat Enterprise Linux Server 7.8 (Maipo)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.4.3
  Kubelet Version:            v1.20.2
  Kube-Proxy Version:         v1.20.2

10.Uncordonthe node to mark it as schedulable and check your pods running status

kubectl uncordon <node_name>

That's it, once all your pods have been restarted you can proceed with thenextworker node !

Control Plane

The procedure to upgrade the container runtime onmasternodes is exactly the same than on the worker node. However you have to be careful if you are on asinglemaster node configuration. Indeed, while the new container runtime will pull kube-apiserver, etcd and coredns images and then create corresponding containers, the cluster will beunavailable. You shouldn't also be able to runkubectlcommand.

Here are some tips to help you follow the new container runtime start and troubleshoot potential problems:

1.Usejournalctlto follow kubelet logs:

journalctl -u kubelet

2.As well watchcontainerdlogs:

journalctl -u containerd

3.Usecrictlcommand to follow container deployments

crictl --runtime-endpoint /run/containerd/containerd.sock ps

4.Check at the end of the upgrade that you are well using the newcontainer runtimeby executing adescribecommand on your master nodes:

kubectl describe node <master_node_name>

Congratulations!You are now running a Kubernetes cluster without Docker and are now ready to receive future releases!