What is Multi-query attention (MQA)?

Multi-query attention (MQA)

Multi-query attention (MQA): An attention variant where all query heads share a single set of keys and values, reducing memory and accelerating generation.

Related terms

Grouped-query attention (GQA)
Multi-head attention (MHA)
Attention head
Attention Is All You Need
Attention mask
Attention mechanism
Attention sink
Attention with linear biases (ALiBi)

Browse the full glossary →