How the Kubernetes Scheduler Places a Pod

The idea: the scheduler only picks a node

When you create a pod, it has no node. Its spec.nodeName is empty. kube-scheduler watches the API server for such pods. For each one it runs a scheduling cycle: it rules out the nodes that cannot run the pod (Filter), ranks the rest (Score), and writes the winner into spec.nodeName with a Binding. That is all it does. It never talks to a node. The kubelet on the chosen node sees the pod in its watch and starts it, as in The Life of a Kubernetes Pod.

The page has four nodes. node-2 has a GPU and a taint that keeps ordinary pods away. node-1 and node-3 have SSDs. node-3 is small. Seven pods already run. You add pods from the pod: menu and watch where they go.

The scheduling queue

New pods wait in the scheduling queue. It has three parts:

• activeQ: pods ready to be tried. It is a heap sorted by priority, then by creation time, so a high-priority pod jumps ahead (the PrioritySort plugin).
• backoffQ: pods that failed before and must wait. The wait starts at podInitialBackoffSeconds (1 s) and doubles up to podMaxBackoffSeconds (10 s).
• unschedulablePods: pods that fit nowhere. They stay until a cluster event (a pod deleted, a node added, a label changed) might help them, or for at most 5 minutes.

The scheduler pops one pod at a time. Each cycle sees the cluster as the previous cycles left it.

Filter plugins

Filter asks each node a yes/no question, plugin by plugin. The first plugin that says no rules the node out. The page shows three of the default plugins, in their real order:

Plugin	Checks	Failure message
TaintToleration	Every NoSchedule taint on the node has a matching toleration in the pod.	node(s) had untolerated taint {gpu: true}
NodeAffinity	The node's labels match the pod's nodeSelector and required node affinity.	node(s) didn't match Pod's node affinity/selector
NodeResourcesFit	The node's allocatable cpu, memory and pod count minus the requests of the pods on it leave room for the new pod's requests.	Insufficient cpu, Insufficient memory, Too many pods

NodeResourcesFit adds up requests, not real usage. A node whose pods are idle can still be "full". A taint pushes pods away; a toleration only lets a pod in, it does not pull it there. That is why the GPU pod also has a nodeSelector (Demo 4: taint + toleration). Required affinity keeps the db pod on SSD nodes (Demo 3).

Score

Each feasible node gets points from the score plugins, each 0 to 100, times the plugin's weight. The page uses two:

• NodeResourcesFit with the LeastAllocated strategy, weight 1: for cpu and memory, (allocatable − requested) × 100 / allocatable after adding the pod, then the mean. More room left, more points. Pods spread out (Demo 2: three web pods land on node-3, node-1, node-4).
• NodeAffinity, weight 2: the weights of the matching preferred terms, scaled to 0–100. In Demo 3 node-1 gets 24 + 2 × 100 = 224 and node-3 gets 12.

The highest total wins. When only one node passes Filter, the scheduler skips Score. Then comes Reserve: the pod is "assumed" on the node in the scheduler's cache, so the next cycle counts its requests before the bind is done. Bind then runs apart from the main loop.

Pending: when no node fits

If every node fails, the pod stays Pending with the condition PodScheduled=False, reason Unschedulable. The scheduler writes a FailedScheduling event that counts the reasons, which is the first thing to read in kubectl describe pod:

0/4 nodes are available: 1 node(s) had untolerated taint {gpu: true}, 3 Insufficient cpu.
preemption: 0/4 nodes are available: 1 Preemption is not helpful for scheduling,
3 No preemption victims found for incoming pod.

The pod moves to unschedulablePods. Nothing is retried in a loop. When a pod is deleted, the queue moves it to backoffQ and then activeQ, and it gets a new cycle (Demo 5: report-1 is Pending until batch-1 finishes, then it lands on node-1).

Preemption by priority

A pod's priorityClassName gives it a number. Here batch is 0, app is 100, critical is 1000. When a pod fits nowhere, the PostFilter plugin DefaultPreemption asks: on which node would evicting lower-priority pods make room? For each node it removes every lower-priority pod, checks the fit, then puts pods back (highest priority first) while the pod still fits. Those left out are the victims.

Among the candidate nodes it picks the one with the least damage: the lowest highest-victim priority, then the smallest sum of priorities, then the fewest victims. It sets the pod's nominatedNodeName and deletes the victims. They get SIGTERM and their grace period. When they are gone, the pod is scheduled on the next cycle (Demo 6). A taint or a missing label cannot be fixed by evicting pods, so such nodes are "not helpful".

What the page leaves out

The real scheduler runs more plugins: PodTopologySpread, InterPodAffinity, NodePorts, volume plugins, ImageLocality and BalancedAllocation. It breaks score ties at random, and on big clusters scores only a sample of nodes (percentageOfNodesToScore). Preemption also respects PodDisruptionBudgets. Queueing hints move only the pods an event can help. There can be several scheduler profiles, plus Permit and PreBind for volumes. The page has no Cluster Autoscaler, which adds a node when pods stay Pending.

References

Kubernetes docs: kube-scheduler

Kubernetes docs: Scheduling Framework

Kubernetes docs: Taints and Tolerations

Kubernetes docs: Pod Priority and Preemption