Raj's Notes

AI inference infrastructure

Build an inference service from the first token and tensor through GPU execution, distributed serving, measurement, and safe rollout.

Scope#

These notes assume you understand an HTTP request and a container, but they assume no machine-learning, CUDA, or distributed-inference knowledge. Start with AI1 and read in number order. AI2 follows one complete request before AI3 opens the GPU, so the hardware vocabulary always has a service-stage to attach to. Each later note uses that request path.

The sequence traces one request through a model server, estimates its memory and token demand, compares single- and multi-GPU layouts, places the service on Kubernetes, measures latency and energy, and explains rollout and failure behavior. Production decisions still require workload-specific profiling, incident evidence, and access to the target hardware.

AI1 compares generation, embeddings and reranking, streaming speech, vision, diffusion, and asynchronous batch inference by request state, batching, latency, throughput unit, and failure mode. The remaining notes take autoregressive language-model serving as the deepest end-to-end case. The same measurement method transfers to other model families, but their production paths need separate hardware- and workload-specific study.

Reading path#

Note Why it comes here
AI1: Model basics Defines models, tokens, tensors, attention, retained state, and the workload families before any serving mechanism depends on them.
AI2: Request lifecycle Turns those model units into admission, prefill, decode, streaming, cache growth, and cleanup for one request.
AI3: GPU systems Maps the request's tensor work and memory movement onto CPU, GPU, CUDA, drivers, and device topology.
AI4: Serving engines Places vLLM, SGLang, TensorRT-LLM, and Triton between the model bundle and the surrounding service contract.
AI5: Node optimization Tunes one measured node before distributed communication and fleet orchestration add more variables.
AI6: Distributed serving Splits a replica or replicates traffic only after the single-node fit and bottleneck are known.
AI7: GPU orchestration Places complete serving groups on Kubernetes and connects device readiness to warm fleet capacity.
AI8: Performance and SRE Measures quality, phase latency, goodput, energy, saturation, and failure headroom across that deployed path.
AI9: Safety and rollout Joins artifact trust, privacy, evaluation, capacity, canarying, rollback, and the complete design case.

Engine features, Kubernetes maturity labels, accelerator support, and command-line flags change by release. Check the linked project documentation before choosing a production configuration.

Useful background#

The Kubernetes chapter includes the small amount of cluster vocabulary it needs and links back to the cloud notes for the full control-plane path. Arithmetic uses bytes, seconds, requests, tokens, and rates, with the units shown in each worked case.

Notes

  1. Inference fundamentals Trace tensors through a model, then compare how generation, embedding, speech, vision, diffusion, and batch workloads change serving state and latency.
  2. The inference request lifecycle Break one streamed completion into admission, tokenization, prefill, decode, KV-cache growth, and delivery to the client.
  3. GPU execution and memory Trace the deployable GPU stack, then read performance and failure as interactions among software compatibility, arithmetic, memory movement, launch overhead, and topology.
  4. Serving engines as moving choices Compare vLLM, TensorRT-LLM, Triton, and SGLang by workload contract, hardware, integration cost, and measured behavior.
  5. Single-node inference optimization Tune request scheduling, KV allocation, attention kernels, graph replay, precision, and speculative work against a stated latency target.
  6. Distributed inference and topology Place model state and request work across GPUs without treating parallelism as free capacity.
  7. GPU orchestration on Kubernetes Advertise, allocate, place, share, and autoscale accelerators while respecting hardware topology and startup time.
  8. Inference performance and SRE Define request-phase SLOs, replay a real workload, and turn latency, useful throughput, power, and energy measurements into capacity decisions.
  9. Production safety and rollout Ship a versioned model behind explicit privacy, tenant, evaluation, canary, observability, and rollback controls.