Pinecone Open-Sources VQ-Bench Vector Quantization Framework

Pinecone has released VQ-bench, an open-source framework for building and benchmarking vector quantization methods, in a September 17, 2026 announcement by Ashwin Padaki, Amir Ingber, and Edo Liberty. The release pairs a public website hosting a running benchmark of popular quantizers with an MIT-licensed GitHub repository and a paper presented at VecDB@VLDB 2026.

Why Pinecone Built a Quantization Benchmark

Vector quantization reduces the number of bits needed to store a vector, which the authors describe as a critical part of maintaining a vector database and as important to both vector databases and large language models. Pinecone has used quantization since its first prototypes, and the authors wrote that when they set out to survey and benchmark newer research, they found that each paper evaluated performance differently: different metrics, different datasets, and different target hardware. They said they were unable to find any systematic attempt to evaluate the leading methods against one another.

The project’s premise, as the authors describe it, is that most published quantizers are assembled from a relatively small set of primitive operations, so building a quantizer can be reduced to a recipe of which primitives to use and in what order. The companion paper, submitted to arXiv on July 31, 2026, states that the framework describes seven conceptual quantization primitives, shows how to compose them arbitrarily, and re-expresses 25 common quantizers as pipelines of those primitives.

How VQ-Bench Composes Quantizers

In VQ-bench, a quantizer is anything that can take a set of vectors, compress them, and recover desired information later. A quantizer must implement four methods: fit, which learns a model from a sample of vectors and optionally queries; encode, which returns per-vector codes given the model; reconstruct, which rebuilds a vector from the model and its code; and score, which estimates the dot product between a query vector and a coded vector.

A primitive implements the same four methods plus two more, apply and apply_queries, which specify how the stage hands data to the next one. Those two methods form the chaining contract that lets primitives be composed. VQ-bench implements three groups of primitives: conditioners such as Center, Normalize, PCA, and RandomRotate; rounders such as CastUint, CastAngular, CastNormal, and KMeans; and splitters such as Segment.

A pipeline chains two or more primitives. The fit and encode methods walk vectors forward through the chain, performing each stage’s job and concatenating each stage’s learned model and output codes; reconstruct starts at the last stage and walks backward, with each stage folding its own contribution back in; and score first pushes the query forward through apply_queries before running the same backward pass. A quantizer does not have to be a pipeline, since anything implementing the four methods qualifies, but the authors report that most published quantizers can be expressed as pipelines of primitives.

The announcement works through E-RaBitQ as an example pipeline of four primitives: Center, which subtracts the average dataset vector from each vector; Normalize, which scales each vector to unit norm; a random rotation applying a random orthogonal or Hadamard transform; and an angular cast that snaps each vector to a b-bit integer grid by rounding to the nearest grid point in angle.

Benchmark Setup and Reported Findings

The authors evaluated a suite of 14 quantizers on five datasets from VIBE, presenting detailed results for two of them: ArXiv, with 1,344,643 vectors in 768 dimensions, and Yahoo, with 677,305 vectors in 384 dimensions. Full results are published on the project’s benchmark website.

Reconstruction mean squared error, which the authors call the traditional metric for vector quantization and important for applications such as LLM weight compression, is measured on 1,000 randomly sampled dataset vectors as the average squared distance between each vector and its reconstruction. For vector databases, the authors describe recall as the more relevant metric, specifically for reranking; recall@10 is measured by computing each query’s 1,000 maximum dot-product dataset vectors and checking what fraction of the quantizer’s estimated top-10 falls within the true top-10, averaged over all queries. The encode-time figures in the announcement were obtained on an Apple M2 Pro with 16GB of RAM using six threads, with encoding done in chunks and accelerated through multithreading.

The authors report that PQ and OPQ consistently produced the lowest reconstruction MSE, that EDEN and E-RaBitQ were comparable on recall, especially at higher bit budgets, and that EDEN encoded much faster than PQ, OPQ, and E-RaBitQ, which they describe as making it a good candidate for most quantization applications.

Repository, Tooling, and Contributions

The GitHub repository is MIT-licensed and lists maintainers Amir Ingber and Edo Liberty of Pinecone and Ashwin Padaki of the University of Pennsylvania. It ships a Rust-based command-line tool, vqb, whose JSON run configurations select datasets, methods and their parameters, quality metrics, k values for recall, softmax temperatures, a master seed, subsampling counts, and a thread count; a dry-run flag validates a configuration before anything is computed. A streaming mode processes datasets larger than memory by reading base vectors from disk in blocks, 256MB by default.

The repository accepts two kinds of contributions, according to the announcement: new quantizers, which are often a few lines of code reordering primitives the library already ships, and new primitives implementing the six interface methods, which then compose with every other primitive in the catalog. The evaluation harness measures recall@k, reconstruction and score error, bias, softmax KL and total variation, size in bits per dimension, and encode, score, and reconstruction cost. The authors describe this release as the first iteration of VQ-bench and said they will add more quantizers over time; corrections can be filed through the repository’s issue tracker, and the published benchmark is refreshed on a regular cadence with new methods folded in.