Load-balancing loss
AILoad-balancing loss: An auxiliary training objective encouraging a mixture-of-experts router to spread tokens evenly across experts and avoid overloading a few.
Load-balancing loss: An auxiliary training objective encouraging a mixture-of-experts router to spread tokens evenly across experts and avoid overloading a few.