The idea: the kubelet doesn't run containers

The kubelet is the Kubernetes agent on every node. It decides which containers should run there, but it never starts a process itself. It asks a container runtime to do it, through one API: the Container Runtime Interface (CRI). Today that runtime is usually containerd. Until Kubernetes 1.24 it was often Docker, through a translator called dockershim.

Three stacks side by side, each with the kubelet on top and runc at the bottom. containerd: kubelet, CRI gRPC, containerd with its CRI plugin, shim, runc. Docker with dockershim: kubelet with dockershim inside, REST to dockerd, gRPC to containerd, shim, runc. Docker with cri-dockerd: kubelet, CRI gRPC to cri-dockerd, REST to dockerd, gRPC to containerd, shim, runc
All three stacks end in containerd and runc; Docker only adds layers above containerd.

On the canvas: the API server and the registry on top. Below them, the node: the kubelet, the Docker layers (only in the Docker modes), containerd and its parts, the shims and runc, and the pod. Under that, what the kernel holds for the pod and what each command-line tool shows. Pick a runtime, then Create pod, or run a demo.

The CRI: two gRPC services

The CRI is a gRPC API, defined in a single api.proto. The kubelet connects to it on a unix socket, set with --container-runtime-endpoint (for containerd, unix:///run/containerd/containerd.sock). It has two services:

servicecalls the page useswhat they do
RuntimeServiceRunPodSandbox, StopPodSandbox, RemovePodSandboxcreate and remove the pod's sandbox
CreateContainer, StartContainer, StopContainerrun one container inside a sandbox
ListPodSandbox, ContainerStatuslet the kubelet compare what runs with what should
ImageServicePullImage, ImageStatusget images onto the node

Anything that implements these calls can be the runtime: containerd, CRI-O, or an adapter in front of something else. That is why Kubernetes could drop Docker without changing how pods work.

Pod sandbox and the pause container

A pod is created in two parts. First RunPodSandbox starts the pause container. It does nothing but sleep. It holds the pod's network, IPC and UTS namespaces, and the CNI plugin gives that network namespace the pod IP. Then each app container is created with CreateContainer and joins those namespaces. So all containers in a pod share one IP and can talk over localhost, and an app container can crash and restart without the pod losing its IP (Demo 6: nginx crashes).

Inside containerd

containerd is a daemon made of plugins. The ones on the canvas:

  • CRI plugin: the CRI gRPC server. It turns pods and containers into containerd objects.
  • Image service and content store: pulls images and keeps every blob by its sha256 digest.
  • Snapshotter (overlayfs): unpacks layers into snapshots and prepares each container's root file system.
  • Task service (runtime v2): starts a shim, containerd-shim-runc-v2, which runs runc.

runc is the OCI runtime. It sets up namespaces and cgroups, execs the process and exits. The shim stays as the container's parent: it keeps stdio and the exit status, so containerd and the kubelet can restart without killing anything (Demo 5: restart containerd). For Kubernetes, the CRI plugin uses one shim per pod; Docker uses one shim per container.

Time runs down. Lanes: kubelet, containerd, shim, runc, pod. kubelet sends RunPodSandbox; containerd starts the shim; the shim runs runc; runc creates pause and exits; CNI ADD gives the IP; reply to the kubelet. Then PullImage, CreateContainer and StartContainer; the shim runs runc again; runc starts nginx in the pause namespaces and exits
A pod starts in two halves: the sandbox (pause plus IP), then each app container joining it.

Each set of containers lives in a containerd namespace: the CRI plugin uses k8s.io, Docker uses moby. They share the daemon but don't see each other.

Where Docker fits, and why dockershim was removed

Docker Engine (dockerd) is built on top of containerd. It adds the docker CLI, image builds, Docker networks and volumes. When Kubernetes started, Docker was the only runtime, and Docker doesn't speak the CRI. So the kubelet carried a translator, dockershim, that turned CRI calls into Docker REST calls (Demo 2). Every call then went kubelet → dockershim → dockerd → containerd → shim → runc.

containerdDocker + dockershimDocker + cri-dockerd
Kubernetes versionsall current onesup to 1.231.24 and later
daemons on the nodecontainerddockerd + containerdcri-dockerd + dockerd + containerd
who maintains the translatorno translatorKubernetes (in the kubelet)Mirantis and Docker
shims per pod (pause + nginx)122
containerd namespacek8s.iomobymoby

Keeping dockershim inside the kubelet was extra code for one runtime, and it added a daemon and two hops on every node. Kubernetes 1.24 (May 2022) removed it. A node still configured with --container-runtime=docker no longer starts its kubelet after the upgrade (Demo 4). Its containers keep running, but nothing watches them, and the node goes NotReady. The usual fix is to point the kubelet at the containerd that Docker already installed. If you need Docker Engine itself, install cri-dockerd, the same translator as a separate process (Demo 3).

Images are not affected. docker build makes OCI images, and containerd runs them unchanged.

docker, crictl and ctr: which tool sees what

containerd holds two namespaces: k8s.io with the pod's pause and nginx containers, and moby with Docker's containers. crictl talks CRI to the CRI plugin and sees k8s.io; ctr -n k8s.io reads k8s.io directly; docker ps asks dockerd, which only looks at moby, so it shows nothing for Kubernetes pods on a containerd node
On a containerd node, docker ps is empty because Docker only looks in the moby namespace.
tooltalks touse it for
kubectlthe API serverthe cluster's view: pod status, restarts, events
crictlthe CRI socketdebugging a node: crictl pods, crictl ps, crictl logs, works with any CRI runtime
ctrcontainerd's own APIlow-level: ctr -n k8s.io containers ls; not CRI-aware
dockerdockerdonly containers dockerd created (namespace moby)

Crashes, restarts and image pull errors

When a container exits, the shim reports TaskExit to containerd. The kubelet learns about it from a CRI container event and from its PLEG (pod lifecycle event generator), which relists the runtime every second. With restartPolicy: Always it creates a new container in the same sandbox after a back-off: 10 s, 20 s, 40 s… up to 300 s. While it waits, the status is CrashLoopBackOff (Demo 6).

A failed PullImage works the same way. The first failure shows ErrImagePull; between retries the status is ImagePullBackOff, with the same doubling delay (Demo 7: pulls at 0, 10, 30 and 70 s). For the rest of a pod's life, from scheduling to deletion, see the life of a Kubernetes pod. For what runc does in the kernel, see what a container really is. For the control plane, see Kubernetes cluster architecture.

See also: What PullImage actually downloads (only the layers whose sha256 digests the node doesn't have yet) and how those layers are built and stacked is shown in How a Dockerfile Becomes Image Layers.

What the page leaves out

CRI-O and other CRI runtimes; RuntimeClass with gVisor or Kata Containers; image pull secrets and registry mirrors; streaming calls (exec, logs, port-forward); the cgroup driver (systemd vs cgroupfs) that must match between kubelet and runtime; NRI plugins; Windows nodes; Docker's own networking (the CNI plugin is used in every mode here).