Glossary · Math & training
Cross-Entropy
A loss based on the negative log probability assigned to the target outcome. In next-token training, it penalizes the model when it assigns low probability to the observed next token.
Common confusion
Perplexity is the exponentiated average cross-entropy only when the averaging and logarithm base are defined consistently.
Related terms
Browse the learning paths to see this term in context — every lesson is free to read.