Attention with linear biases (ALiBi)
AIAttention with linear biases (ALiBi): A scheme that adds a distance-based penalty to attention scores instead of positional embeddings, favouring nearer tokens and extending to longer sequences.
Attention with linear biases (ALiBi): A scheme that adds a distance-based penalty to attention scores instead of positional embeddings, favouring nearer tokens and extending to longer sequences.