Raj's Notes

Containers and the Kubernetes object model

Connect processes and images to clusters, Pods, Deployments, Services, configuration, storage, and replaceable workload instances.

Working model#

An image is a packaged filesystem and launch contract. A container is a running process under isolation controls. A Pod is Kubernetes' smallest scheduling envelope.

Start with the application concepts Kubernetes assumes#

Kubernetes does not replace the application process, network protocol, or storage engine. It coordinates them. Before reading its object names, keep these ordinary runtime facts visible:

Application fact What Kubernetes needs from it
Process An entrypoint that starts, remains in the foreground, reports failure through exit status, and stops on signal
Listener An IP address and port where the process accepts traffic; declaring a port does not make a process listen
Health behavior A startup, readiness, or liveness signal whose failure has a stated consequence
Packaged files An immutable image reference plus runtime configuration and mounted data
Durable state A database, object store, or volume contract that survives replacement when the workload requires it
Resource demand CPU, memory, storage, and optional devices that the scheduler and runtime can account for
Network dependency A resolvable name, route, policy, credential, deadline, and retry behavior
Declaration Structured YAML or JSON that identifies an API type and supplies a desired-state object

A process that only listens on 127.0.0.1 will not accept ordinary traffic sent to the Pod IP. A process that writes its only copy of an order to the container writable layer loses that state when the Pod is replaced. A health endpoint that reports a shared dependency outage as a liveness failure can make every replica restart together. Kubernetes executes the contract you declare; it cannot infer the intended behavior from the application.

YAML is a common serialization format for Kubernetes objects, not the control system itself. Indentation forms nested maps and lists, --- separates documents, and the API schema decides which fields are valid. The parsed object sent to the API server is the contract. A rendered manifest should therefore be reviewed and validated like any other API request.

If one of these foundations is unfamiliar, LL1: The kernel boundary explains processes, signals, files, sockets, and system calls; LL6: Containers and cgroups explains namespaces and resource controls; and CI1: AWS foundations explains addresses, ports, DNS, routes, identity, and storage. You can begin here and follow those links only when the lower layer blocks the current explanation.

Kubernetes coordinates processes across machines#

One machine can run a container directly. The harder production problem is keeping several copies alive across many machines, replacing failures, rolling out a new image, giving changing copies a stable network name, and attaching configuration or storage. Kubernetes addresses that problem through an API and controllers.

A cluster is one Kubernetes control plane plus its worker nodes. A node is a machine, physical or virtual, that runs workload processes. The control plane stores desired objects and makes cluster-wide decisions. A node agent called the kubelet receives assigned Pods and asks a container runtime to start them. CI3: Control planes, etcd, and reconciliation follows that full path.

flowchart TD
  accTitle: Kubernetes cluster from API request to running containers
  accDescr: A user or delivery controller sends object definitions to the Kubernetes API server. The control plane stores desired state and controllers create dependent objects. The scheduler assigns pending Pods to worker Nodes. Each Node's kubelet asks a container runtime to start the Pod's containers. The application process runs inside those containers rather than inside the control plane.

  USER["User, CI system,<br/>or GitOps controller"] --> API["Kubernetes API server"]
  API --> STORE["Persisted cluster objects"]
  STORE --> CONTROLLERS["Controllers"]
  STORE --> SCHED["Scheduler"]
  CONTROLLERS --> PODS["Pending Pod objects"]
  SCHED --> BIND["Pod-to-Node binding"]

  subgraph NODEA["Worker Node A"]
    KUBELETA["kubelet"] --> RUNTIMEA["container runtime"]
    RUNTIMEA --> PODA["Pod<br/>one or more containers"]
  end

  subgraph NODEB["Worker Node B"]
    KUBELETB["kubelet"] --> RUNTIMEB["container runtime"]
    RUNTIMEB --> PODB["Pod<br/>one or more containers"]
  end

  BIND --> KUBELETA
  BIND --> KUBELETB

kubectl apply sends desired object data to the API server. It does not SSH to a node or start a container directly. Controllers and node agents perform the later work, which is why an accepted manifest can still lead to a Pending Pod, failed image pull, unready process, or empty Service.

Kubernetes objects use a common shape:

Labels are small identifying key-value pairs. Selectors use those labels to join objects, such as a Deployment to its Pods or a Service to its backends. A typo can produce valid YAML and valid objects that no longer select one another.

Most application objects are namespaced. A namespace scopes names and can participate in quota, policy, and access control, but it is still inside one cluster and shares that cluster's control plane. Nodes, PersistentVolumes, and some other resources are cluster-scoped. A namespace does not create another VPC, node fleet, kernel, or Kubernetes cluster.

The fields below spec depend on kind. A Deployment contains a Pod template, which is the recipe its controller uses to create Pod objects; the template is not itself a running Pod. A Service uses a label selector rather than embedding or owning those Pods.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders
  namespace: bookshop
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders
  template:
    metadata:
      labels:
        app: orders
    spec:
      containers:
        - name: api
          image: registry.example/orders@sha256:<digest>
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /ready
              port: http
---
apiVersion: v1
kind: Service
metadata:
  name: orders
  namespace: bookshop
spec:
  selector:
    app: orders
  ports:
    - name: http
      port: 80
      targetPort: http

The Deployment asks for three interchangeable Pods. The Service selects ready network endpoints carrying app: orders, gives clients a stable Service name and virtual address, accepts traffic on port 80, and forwards it to the named http port on each selected Pod. The application still has to bind a listener on port 8080; containerPort records intent and supplies a named port, but it does not open a socket or firewall rule.

A Deployment creates Pods; a Service routes to them#

These objects cooperate without doing the same job:

Object Job What it does not do
Deployment Declares interchangeable replicas and rollout policy Provide a stable client address or route external HTTP
ReplicaSet Maintains one revision's desired Pod count Choose rollout strategy across revisions
Pod Describes colocated containers, volumes, identity, and runtime needs Survive replacement as one stable application instance
Service Provides stable discovery and transport to selected endpoints Create, restart, or roll out Pods
EndpointSlice Records batches of current backend addresses, ports, and readiness Decide the desired replica count
Gateway or Ingress path Routes external or cross-service protocol traffic to a Service Replace Service backend selection or Deployment rollout 
flowchart TB
  accTitle: Deployment ownership and Service traffic are separate Kubernetes graphs
  accDescr: A Deployment owns a ReplicaSet, which owns three replaceable Pods. A Service does not own those Pods; its selector matches their labels, and the control plane records ready Pod addresses in EndpointSlices. Internal clients call the Service DNS name. An optional Gateway or Ingress route sends external HTTP traffic to that same Service.

  subgraph OWNERSHIP["Workload ownership and replacement"]
    DEPLOY["Deployment<br/>replicas: 3"] --> RS["ReplicaSet<br/>one template revision"]
    RS --> PODS["Three replaceable Pods<br/>label: app=orders"]
  end

  subgraph TRAFFIC["Discovery and traffic"]
    INTERNAL["Internal client"] --> SVC["Service<br/>selector: app=orders"]
    EXTERNAL["External client"] --> ROUTE["Gateway or Ingress route"]
    ROUTE --> SVC
    SVC --> EPS["EndpointSlice<br/>ready Pod IPs and target port"]
  end

  SVC -. "label selector" .-> PODS
  EPS -. "ready addresses" .-> PODS

Ownership and selection have different lifecycle effects. Deleting a Deployment normally garbage-collects the ReplicaSets and Pods it owns. Deleting a Service does not delete selected Pods; it removes that stable traffic path and its managed EndpointSlices. Changing a Service selector can redirect traffic without creating a Deployment rollout. Changing the Deployment's Pod template creates a new ReplicaSet but leaves the Service identity in place as long as the new Pods carry matching labels.

A Service with type ClusterIP is ordinarily reachable through cluster networking. NodePort adds a port on cluster nodes, and LoadBalancer asks an installed implementation to provision or connect an external load balancer. Gateway API or Ingress adds protocol-aware routes such as host and path matching in front of Services. CI4: Kubernetes networking, storage, and security follows those packet and controller paths.

Learn object families before memorizing fields#

Family Common objects Question answered
Workload Pod, Deployment, StatefulSet, DaemonSet, Job, CronJob What should run, how many copies, with which identity and completion rule?
Discovery and traffic Service, EndpointSlice, Gateway, HTTPRoute, Ingress Which current endpoints receive which traffic?
Configuration and credentials ConfigMap, Secret Which runtime values are delivered to the Pod?
Storage PersistentVolumeClaim, PersistentVolume, StorageClass, CSI objects Which storage interface, lifecycle, topology, and provider satisfy the mount?
Scaling and disruption HorizontalPodAutoscaler, PodDisruptionBudget When may replica intent change, and how much voluntary disruption is allowed?
Placement and isolation Node, Namespace, ResourceQuota, NetworkPolicy Where can work run, which names and budgets are scoped, and which traffic is admitted?
API extension CustomResourceDefinition and custom resources Which domain object and reconciliation behavior does an operator add?

An object name does not guarantee an implementation. A Service needs a functioning network data plane; a NetworkPolicy needs a network plugin that enforces it; a PersistentVolumeClaim needs matching storage or a provisioner; a custom resource needs a controller if creating it should cause domain work.

An image isn't a tiny virtual machine#

An Open Container Initiative (OCI) image contains ordered filesystem layers plus metadata such as the entrypoint, arguments, environment defaults, and platform. A runtime unpacks that image and starts a host-kernel process with namespaces, cgroups, capabilities, mounts, and a restricted root filesystem around it.

Those Linux terms describe separate controls. A namespace changes what the process can see, such as process IDs, mounts, or network devices. A cgroup accounts for processes and can apply CPU or memory policy. Capabilities split privileged kernel operations into narrower checks. None creates a separate kernel; LL6: Containers and cgroups assembles the complete host-side path.

The process still makes system calls into the node's kernel. That shared kernel is why a container starts quickly and why its security boundary differs from a hardware virtual machine. Immutability applies to the image artifact; a running container can still write to its writable layer or mounted volumes.

A registry stores image manifests and layers. A tag such as stable is a movable name; a digest identifies exact content. Production promotion should preserve the tested digest, while signatures, provenance, vulnerability scanning, and a software bill of materials (SBOM) answer separate supply-chain questions. A multi-platform image index can point to different manifests for architectures such as linux/amd64 and linux/arm64, so confirm the resolved platform when a Pod pulls on one node type but fails on another.

A Pod is a colocated process group#

Every container in a Pod lands on the same node and shares the Pod network namespace. They can use localhost, see the same Pod IP, and mount shared volumes when configured. They don't merge filesystems or process namespaces by default. A sidecar should exist only when tight lifecycle and locality are part of the design.

Regular init containers run to completion in order before application containers start. Kubernetes also supports restartable sidecars as init containers with restartPolicy: Always; they start in init order, keep running with the Pod, and no longer prevent a Job from completing after its main container finishes. Check the target cluster version before adopting that form because older clusters used only ordinary application containers as sidecars.

Pods are replaceable. Their names, UIDs, and IPs change as controllers create new instances. Put stable access behind a Service and durable state behind an external store or persistent volume; don't teach clients to remember a Pod.

Fictional case. Each bookshop storefront-api Pod runs an API container and an OpenTelemetry Collector sidecar. The sidecar receives telemetry over localhost, but clients reach the replaceable Pods through a Service. Nothing about the Pod name or IP becomes durable application identity.

Controllers own replacement and rollout#

A Deployment manages interchangeable replicas through ReplicaSets. StatefulSets add stable ordinal identity and storage claims; DaemonSets target nodes; Jobs seek completion, while CronJobs create Jobs on a schedule. Pick the controller whose failure and completion semantics match the process.

ConfigMaps carry non-secret configuration. Secret objects improve API separation and role-based access control (RBAC), but base64 is only an encoding, and Kubernetes stores Secret data unencrypted in etcd unless the cluster enables encryption at rest. Keep credentials out of images and Git, restrict API and node access, encrypt the store, and arrange rotation before the first incident. Mounting a Secret or injecting it as an environment variable changes delivery, not its lifetime or authority.

A DaemonSet derives its count from eligible nodes#

A Deployment starts from a requested replica count. A DaemonSet starts from the current node set. Its controller creates one Pod on every eligible node, creates another when a matching node joins, and removes the corresponding Pod when that node leaves. “One per node” means one per node that passes selectors, affinity, taints, and tolerations, not one per cluster and not one sidecar inside every application Pod.

This fits node-local infrastructure such as CNI and CSI agents, log collectors, security sensors, and storage daemons. Each DaemonSet Pod still requests CPU and memory, so its overhead reduces the allocatable capacity left for application Pods. A node autoscaler must account for that overhead when deciding whether a new machine can fit pending work.

Controller Desired cardinality Identity and completion contract
Deployment Explicit interchangeable replica count Replaceable Pods; rolling rollout through ReplicaSets
StatefulSet Explicit replica count with stable ordinals Ordered identity and usually one persistent claim per ordinal
DaemonSet One Pod on each eligible node Node-local instance follows node eligibility and lifetime
Job Enough Pods to reach the declared completions Finite work; retries until success or its failure policy stops it
CronJob Jobs created from a schedule Scheduling policy plus each child Job's finite-work contract
Custom controller Whatever relationship its API defines Must be learned from its custom resource, status, and reconciliation

Read a fictional base as one workload contract#

Suppose the bookshop keeps the following toy repository. These paths and values exist only in this note:

deploy/storefront/base/kustomization.yaml
  ├─ deployment.yaml → Pod template, probes, resources, lifecycle
  └─ service.yaml    → selector and target port

deploy/storefront/overlays/dev/kustomization.yaml → two replicas and a dev image digest
deploy/storefront/overlays/prod/kustomization.yaml → four replicas and a prod image digest

The base includes a Deployment and Service. Both use the label app: storefront-api; the Service exposes port 80 and forwards it to the named container port http on port 8080. The Pod template declares readiness at /ready, requests CPU and memory, and gives the process time to drain on termination. An overlay supplies a replica count and immutable image digest.

Render each overlay with kubectl kustomize and inspect the result before applying it. A changed image digest alters the Pod template and starts a rollout. A changed Service selector alters traffic membership without replacing Pods. A selector typo can therefore produce healthy Pods and an empty EndpointSlice—the Kubernetes object that lists a Service's current network backends—at the same time. CI4 follows that traffic path in full.

The example is self-contained; build the field relationships rather than copying a deployment from another service.

The word base is overloaded. A Kustomize base is reusable YAML that renders Kubernetes objects. An OCI image root filesystem is the packaged directory tree from which a container starts. An immutable filesystem template can be a separate ext4 block image selected by an identifier in that YAML. Patching a template UUID into an overlay changes a reference; it does not insert the referenced repository files into the Kubernetes manifest.

A Deployment changes ReplicaSets when its Pod template changes#

Take a four-replica Deployment with maxSurge: 1 and maxUnavailable: 1. During a rolling update, the controller may run at most five old-plus-new Pods and should keep at least three available under that strategy. It creates a new ReplicaSet for the changed Pod template, increases the new ReplicaSet, and decreases the old one while readiness determines which Pods count as available. Scheduling, image download, startup, and readiness can pause the sequence independently.

Only changes to the Deployment's Pod template start this rollout. Editing a ConfigMap that a Pod reads does not by itself change the template or replace existing Pods. Common patterns put a content hash or versioned ConfigMap name in the template so a reviewed configuration change also changes the template. Keep the old ReplicaSet long enough for rollback, but remember that rolling back a Deployment cannot reverse a database migration or restore an overwritten external object.

Availability is based on Pod readiness and any minimum-ready time, not merely a Running phase.

desired replicas = 4
maxSurge = 1       → at most 5 Pods during rollout
maxUnavailable = 1 → at least 3 available Pods
Pod-template change → new ReplicaSet → ready new Pods → old scale-down

Read the controller chain before opening a shell#

Start with kubectl get deployment,replicaset,pod in the intended namespace and compare desired, current, updated, available, and ready counts. kubectl rollout status deployment/<name> reports rollout progress, while kubectl describe shows conditions and recent events. Match the Pod template labels to the Service selector before assuming traffic reaches the new Pods. Sort namespace events by time when a scheduling or image problem is suspected.

A Pending Pod points first to scheduling events, storage claims, namespace quotas, or image credentials; CI4 covers claims and CI5 covers quotas and placement. ImagePullBackOff points to image reference, registry reachability, or authentication. A Running but unready Pod points to the readiness path, listener, configuration, or dependency. CrashLoopBackOff calls for current and previous container logs, exit reason, and restart count. An OOM-killed process calls for memory limit and working-set evidence. Use kubectl debug or an ephemeral container—a temporary diagnostic container added to an existing Pod—only when the image lacks tools and policy allows it; record any change made during diagnosis.

Summary#

Kubernetes schedules process groups, not miniature machines. The image defines what to start, the Pod defines what must be colocated, and a controller defines how instances are replaced, completed, or kept present.

References#