What is Learning-rate warmup?

Learning-rate warmup

Learning-rate warmup: A phase at the start of training where the learning rate rises gradually from near zero, avoiding unstable updates on fresh weights.

Related terms

Learning rate
Learning-rate schedule
Automated machine learning (AutoML)
Cache hit rate (prompt)
Character error rate (CER)
Continual learning
Contrastive learning
Curriculum learning

Browse the full glossary →