Like a lot of people here, I run local models on a single 24 GB GPU, and the thing that caps me sometimes isn't the model weights, it's the KV cache. My daily driver (Qwen3.6-27B) fits fine, but pushing context toward its native 262k blows past 24 GB. So I had an idea that's probably occurred to half this sub: keep the "hot" parts of the KV cache in VRAM, park the "cold" parts in system RAM, and shuttle the relevant pieces back in as the conversation needs them. I used claude to do some research and organize some it feels should take a look at. It included systems like Quest, ShadowKV and InfiniGen do versions of this, so the idea has at least has some merit to it.
This is a companion discussion topic for the original entry at https://www.reddit.com/r/LocalLLM/comments/1v74tpe/tried_to_build_a_queryaware_kv_cache_offloading/