Glossary · Math & training
Warmup
An initial training phase in which the learning rate rises from a smaller value toward the main schedule's target value.
Why it matters
Early gradients and optimizer statistics can be unstable, especially in large-batch or transformer training, so abrupt full-size updates may damage optimization.
In practice
Define warmup in steps or processed tokens, log the realized curve, and tune it with the batch, optimizer, and total training budget held visible.
Common confusion
Warmup is not required for every model and does not make an otherwise unsuitable learning rate safe.
Related terms
Sources
Browse the learning paths to see this term in context — every lesson is free to read.