AI Models & Platforms
Valkey 9.2 Launches with Distributed KV Caching for AI Inference

Key PointsDistributed KV caching lets inference engines reuse prompt tensors instead of…
View more
- Distributed KV caching lets inference engines reuse prompt tensors instead of recomputing them during prefill.
- Multiverse Computing served 50% more requests on the same GPUs while meeting its first-token latency target.
- The shared cache offers little benefit when the working set already fits in GPU and host memory.
Valkey released version 9.2 on November 19, 2026, accompanied by distributed key-value caching for AI inference. The project’s accompanying announcement says the shared cache can cut inference spending by up to 40% and the wait for a response’s first token by up to 80% on existing GPU infrastructure.
A Valkey technical post published October 20, 2026, describes a shared cache beyond GPU and host memory that stores the key-value tensors generated when a model processes prompt instructions, tool definitions and conversation history. Inference engines can reuse those tensors for repeated context instead of recomputing them during prefill, the processing stage before new output is generated. Sharing the tier across GPU servers lets any node load a conversation that local memory has evicted.
Self-managed deployments running open-weight models can connect the cache through the LMCache connector to engines including vLLM and SGLang with a configuration change and no application-code updates. Path Hash, a new radix-tree-backed data type in 9.2, supports exact lookup, longest-prefix matching and prefix traversal. Valkey positions it as a lightweight index for finding the longest matching cached prefix and its location while larger AI data remains in GPU or host memory.
Measured Gains and Workload Limits
Multiverse Computing’s director of AI platform, Ennio Pastore, said distributed KV caching on Valkey let the company serve 50% more requests on the same GPUs while meeting its one-second target for median time to first token. Median first-token latency fell 23% at normal traffic and 95% at peak traffic, and Multiverse also contributed LMCache connector improvements intended to make distributed caching more resilient.
The synthetic benchmark described in Valkey’s technical post used 100 concurrent conversations with 12,000 to 28,000 tokens of context, arriving at 8 requests per second across two GPU nodes with eight H100 accelerators each. The nodes served Llama-3.1-70B through vLLM and LMCache, with a three-shard Valkey 9 cluster in the same availability zone as the shared cache. The project measured about 25% more requests and about 28% more output tokens, while the cache added about 15% to the GPU bill.
Valkey’s testing and deployment guidance identifies clear boundaries: the shared cache offers little benefit when the working set already fits in GPU and host memory or requests lack repeated context, and it cannot create GPU compute when new-token prefill becomes the bottleneck. The recommended placement is the same availability zone as the GPU fleet, because the cache sits on the first-token path and cross-zone traffic can add latency and transfer charges.
Lower Memory and Network Overhead
Beyond AI caching, version 9.2 targets the datastore’s own infrastructure costs. Opt-in forkless snapshots and full replica synchronizations use a background thread instead of creating a separate server process, avoiding copy-on-write memory overhead during write-heavy work. The infrastructure adds four bytes per key, and the prefer-forkless policy falls back to fork when necessary.
Compression for ongoing replication is negotiated separately with each replica, so older or opted-out replicas continue receiving plaintext. Valkey’s release post reports that in a benchmark of 3 million keys of about 315 bytes each, LZ4 reduced replication bandwidth by 52% with less than 1% throughput overhead. Whole-stream snapshot compression supports LZ4 and Zstandard; Zstandard requires a build-time option and is aimed particularly at cross-region replication, where network costs are high relative to compute.
Large sorted sets now use a B+ tree instead of a skiplist. Valkey reports per-member memory overhead falling 36% to 43% in a test with 5 million members containing 20-byte elements. Other operational additions include server-side hot-key detection, reusable access-control roles, priority handling for cluster heartbeats and replication, reserved administrative connections and optional write throttling when a replica falls behind.
The 9.2.0 release record marks this as the first stable 9.2 release, and the download page lists Docker tags and platform-specific binary artifacts. Valkey describes 9.2 as a minor release without breaking changes, but warns that append-only file bases written with LZ4 or Zstandard compression require a compatible Valkey version if an operator later downgrades.
Valkey 9.2 connects two sides of inference economics: recovering GPU capacity lost to repeated prefill and reducing the memory and network overhead of the data layer. For operators assessing the new tier, the documented results tie its value to workload reuse and eviction pressure, with an explicit ceiling when new-token prefill already consumes the available GPU compute.












