Nvidia says it can shrink LLM memory 20x without changing model weights

Nvidia's KV Cache Transform Coding (KVTC) compresses LLM key-value cache by 20x without model changes, cutting GPU memory ...
EE World Online

How to approach AI hardware design to address the memory wall?

This article outlines the design strategies currently used to address these bottlenecks, ranging from data center systolic ...
Semiconductor Engineering

Will In-Memory Processing Work?

The cost associated with moving data in and out of memory is becoming prohibitive, both in terms of performance and power, and it is being made worse by the data locality in algorithms, which limits ...

New KV cache compaction technique cuts LLM memory 50x without accuracy loss

MIT researchers developed Attention Matching, a KV cache compaction technique that compresses LLM memory by 50x in seconds — ...