The New Engineer
Sign inStart free

Phase 10

LLMs from Scratch

Build, train, and understand large language models.

Lessons (24)

  1. 01Tokenizers: BPE, WordPiece, SentencePiece
  2. 02Building a Tokenizer from Scratch
  3. 03Data Pipelines for Pre-Training
  4. 04Pre-Training a Mini GPT (124M Parameters)
  5. 05Scaling: Distributed Training, FSDP, DeepSpeed
  6. 06Instruction Tuning (SFT)
  7. 07RLHF: Reward Model + PPO
  8. 08DPO: Direct Preference Optimization
  9. 09Constitutional AI and Self-Improvement
  10. 10Evaluation: Benchmarks, Evals, LM Harness
  11. 11Quantization: Making Models Fit
  12. 12Inference Optimization
  13. 13Building a Complete LLM Pipeline
  14. 14Open Models: Architecture Walkthroughs
  15. 15Speculative Decoding and EAGLE-3
  16. 16Differential Attention (V2)
  17. 17Native Sparse Attention (DeepSeek NSA)
  18. 18Multi-Token Prediction (MTP)
  19. 19DualPipe Parallelism
  20. 20DeepSeek-V3 Architecture Walkthrough
  21. 21Jamba — Hybrid SSM-Transformer
  22. 22Async and Hogwild! Inference
  23. 25Speculative Decoding and EAGLE
  24. 34Gradient Checkpointing and Activation Recomputation

Learning paths covering this phase