SwiGLU

SwiGLU: A gated activation used in transformer feedforward layers, combining a Swish activation with a gated linear unit to improve quality.

Browse the full glossary →