Phase 17
Infrastructure and Production
Ship AI to the real world. Scale, monitor, optimize.
Lessons (28)
- 01Managed LLM Platforms — Bedrock, Vertex AI, Azure OpenAI
- 02Inference Platform Economics — Fireworks, Together, Baseten, Modal, Replicate, Anyscale
- 03GPU Autoscaling on Kubernetes — Karpenter, KAI Scheduler, Gang Scheduling
- 04Serving Engine Internals — PagedAttention, Continuous Batching, Chunked Prefill
- 05EAGLE-3 Speculative Decoding in Production
- 06Prefix-Cache Serving — RadixAttention and KV Reuse
- 07Hardware-Specialized Inference Compilation — FP8 and NVFP4 on Blackwell
- 08Inference Metrics — TTFT, TPOT, ITL, Goodput, P99
- 09Production Quantization — AWQ, GPTQ, GGUF K-quants, FP8, MXFP4/NVFP4
- 10Cold Start Mitigation for Serverless LLMs
- 11Multi-Region LLM Serving and KV Cache Locality
- 12Edge Inference — Apple Neural Engine, Qualcomm Hexagon, WebGPU/WebLLM, Jetson
- 13LLM Observability Stack Selection
- 14Prompt Caching and Semantic Caching Economics
- 15Batch APIs — the 50% Discount as Industry Standard
- 16Model Routing as a Cost-Reduction Primitive
- 17Disaggregated Prefill/Decode — NVIDIA Dynamo and llm-d
- 18Production Serving Stack — KV Offloading and Cache-Aware Routing
- 19AI Gateways — LiteLLM, Portkey, Kong AI Gateway, Bifrost
- 20Shadow Traffic, Canary Rollout, and Progressive Deployment for LLMs
- 21A/B Testing LLM Features — GrowthBook, Statsig, and the Vibes Problem
- 22Load Testing LLM APIs — Why k6 and Locust Lie
- 23SRE for AI — Multi-Agent Incident Response, Runbooks, Predictive Detection
- 24Chaos Engineering for LLM Production
- 25Security — Secrets, API Key Rotation, Audit Logs, Guardrails
- 26Compliance — SOC 2, HIPAA, GDPR, PCI-DSS, EU AI Act, ISO 42001
- 27FinOps for LLMs — Unit Economics and Multi-Tenant Attribution
- 28Self-Hosted Serving Selection — Matching Engine to Hardware and Scale