H Company Releases NeoMME, an Open-Source Multimodal Encoder Family

H Company researchers released NeoMME on September 3, 2026, a family of 260M- and 800M-parameter multimodal and multilingual encoders trained from scratch and published under the Apache 2.0 license. Fine-tuned retrieval variants sit on the model-size Pareto frontier of the ViDoRe v3 visual document retrieval benchmark, the team reported.

According to the release post, many recent visual document retrievers are adapted from pretrained generative vision-language models, in which a separately pretrained vision encoder produces visual features that a projector maps into a causal language model’s input space. Retrieval, classification, and token labeling do not generate text autoregressively, the authors write, so those tasks do not require a causal decoder or its parameter and compute overhead. NeoMME instead runs text tokens and raw image patches through one shared bidirectional Transformer and is not based on any existing pretrained vision tower, text encoder, or text decoder.

The post positions the design against two earlier encoder efforts: ModernBERT, which brought efficiency improvements to bidirectional text encoders, and ModernVBERT, which applied a ModernBERT-style text encoder to visual document retrieval while retaining a separate pretrained SigLIP2 vision tower. The authors write that they wanted to remove the overhead of carrying a vision-language model’s components into an encoder.

Architecture and From-Scratch Pretraining

Both NeoMME sizes share the same architecture. Text inputs use factorized token embeddings, while images are divided into a grid of non-overlapping 32×32 patches and projected with a small multilayer perceptron; both then enter the same Transformer encoder. Images keep their aspect ratio and size, so the model can spend more tokens on a high-resolution, information-dense document page than on a smaller image. The context length is 16,384 tokens, enough for up to two standard 3840×2160 4K UHD images. Most layers use symmetric sliding-window attention, while every sixth layer and the final layer use global attention. The stack also includes grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs. For text, the team trained a BPE tokenizer with a 131,072-token vocabulary from scratch on multilingual text, code, mathematics, and machine-produced image transcripts.

NeoMME is pretrained from scratch as a discrete masked-diffusion text denoiser. For text-only examples, a corruption rate is sampled uniformly between 0 and 1, and each eligible text token is independently masked at that rate. Multimodal examples use corruption rates between 0.3 and 1, with the image patches left visible while the model reconstructs masked text; the authors write that heavy masking forces the model to learn image-grounded descriptions rather than relying on language-only shortcuts. Pretraining mixes multilingual text, code, mathematics, natural images, and document images, and each model processes about 524 billion packed input tokens, including 290 billion from text-only examples. Noting that this text budget is small relative to ModernBERT’s 2 trillion training tokens, the authors write that they chose the NorMuon optimizer to improve data efficiency.

Retrieval Fine-Tuning and Benchmark Results

To evaluate the backbone on a downstream task, the team fine-tuned it for visual document retrieval using the page-image methodology introduced by ColPali. Rather than retrieving extracted text chunks, NeoMME-Retriever ranks screenshots of document pages, bypassing OCR preprocessing and preserving layout, charts, tables, and typography. The retriever adds two jointly trained heads to the backbone: a dense head that mean-pools hidden states into a normalized vector, and a late-interaction head that projects each text token or image patch to a 128-dimensional normalized vector, preserving fine-grained matches between query tokens and image regions. One forward pass returns both representations. The authors recommend late-interaction embeddings in general, and a pipeline of dense retrieval followed by late-interaction reranking for very large corpora.

On the ViDoRe v3 benchmark, the post reports an nDCG@10 of 0.523 for NeoMME-Retriever-260M, the highest score among evaluated models strictly below 800M parameters and within 0.002 of ColQwen2.5 while using about 14 times fewer parameters. NeoMME-Retriever-800M reaches 0.556, within 0.009 of the similarly sized Vultron Retriever Flash, and both models lie on the benchmark’s model-size Pareto frontier. The post’s comparison table marks competitor scores as sourced from MTEB and the NeoMME scores as the team’s own evaluations. On the older ViDoRe v1 and v2 benchmarks, which use nDCG@5, the post reports that the 260M model outperforms ColModernVBERT and the twice-larger ColSmol-500M, while the 800M model outperforms ColPali v1.3 with 3.6 times fewer parameters.

The model card for NeoMME-260M-Retriever lists 263M parameters, a hidden size of 1,024, BF16 weights, and 1,024-dimensional dense embeddings with Matryoshka truncation points at 128, 256, 512, and 1,024 dimensions, alongside the 128-dimensional multi-vector output. The card also reports text-retrieval results on BEIR-15, where the 260M retriever scores 0.4881 nDCG@10 with late interaction and the 800M model scores 0.5126.

Compression, Indexing Throughput, and Availability

Because late-interaction storage scales linearly with the number of embedding vectors, high-resolution pages are expensive to index: a 2048×2048 page produces 4,200 vectors with NeoMME-Retriever, about 2.1 MB in float32, and the post reports a measured average of about 1.5 MB per document across ViDoRe v3. The team combined hierarchical token pooling, which clusters similar document vectors and stores each cluster’s mean, with asymmetric quantization, which stores document embeddings at int8 or binary precision while keeping on-the-fly query embeddings at higher precision. On ViDoRe v3, the post reports that a pooling factor of 10 with int8 queries and documents cut storage to 39 kB per page, a 39× reduction, while retaining more than 99% of baseline nDCG@10. A more aggressive setting, pooling factor 8 with int8 queries and binary documents, uses 6 kB per page, 255 times smaller, and retains more than 95% of retrieval quality.

The team also measured encoding throughput using preprocessed image tensors, with batch sizes calibrated separately for each model and image size. At a matched 2048×2048 input on one NVIDIA L40S GPU, NeoMME-Retriever-260M encodes about 51 pages per second, nearly twice ColModernVBERT’s 26 pages per second, and the post reports that both NeoMME-Retriever sizes are faster than the other compared models at smaller input resolutions.

All NeoMME checkpoints are released under Apache 2.0 with a day-zero implementation in Hugging Face Transformers, and a visual retrieval-augmented generation demo is available as a Hugging Face Space. For fine-tuning, the team provides separate dense and late-interaction checkpoints compatible with Sentence Transformers v6, which currently supports one retrieval head per model; training both heads together requires the NeoMMEForRetrieval class with a custom Trainer.

The accompanying technical report, by Aurélien Lac and Tony Wu, was submitted to arXiv on August 31, 2026, in the information retrieval category. In the post’s acknowledgements, the authors describe NeoMME as a side project built with limited time and compute, and thank H Company for supporting the work and providing the compute used to train it.