PagedAttention
AIPagedAttention: A memory management technique that stores the KV cache in small non-contiguous blocks, like operating-system paging, reducing waste and letting more requests run at once.
PagedAttention: A memory management technique that stores the KV cache in small non-contiguous blocks, like operating-system paging, reducing waste and letting more requests run at once.