GitHub

LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models

transformers had made a major change on kv cache implementation since version 4.36.0. Please use ppl_legacy if you are using transformers < 4.36.0 ...