H Company has released NeoMME, a new family of 260 million and 800 million parameter encoders that remove the separate vision tower and causal decoder found in most visual document retrievers. These single-tower models process multilingual text and raw image patches through the same layers, trained from random initialization.
The retrieval fine-tune, NeoMME-Retriever, achieves a 0.523 nDCG@10 score on ViDoRe v3 with the 260 million parameter version. Every checkpoint ships under Apache 2.0 and includes day-zero support in Hugging Face Transformers. The 260M model indexes 51.3 pages per second on a single NVIDIA L40S and encodes a query in 78.3 ms on a CPU-only host.
One tower, two modalities
Text enters through an ALBERT-style factorized embedding: a 256-dimensional lookup projected to model width. Images are split into non-overlapping 32×32 patches and projected by a 2-layer MLP trained from scratch. There is no patch-merging module and no SigLIP2 tower.
Both models support a 16,384-token context, enough for two standard 3,840×2,160 4K UHD images after patching. Most layers use symmetric sliding-window attention; every sixth layer and the final layer attend globally. The stack uses grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs. Exact parameter counts are 262,937,906 and 793,715,032.
The tokenizer is a whitespace-unconstrained BPE with a 131,072-entry vocabulary, trained from scratch. Across 14 target languages in FLORES-200 devtest, it emits 44.4% fewer tokens than ModernBERT.
Trained as a masked diffusion denoiser
Pretraining is discrete masked diffusion over text, optionally conditioned on visible image patches. Text-only segments draw a corruption rate uniformly from 0 to 1. Multimodal segments draw from 0.30 to 1, which removes the language-only shortcut and forces the model to read the page.
A cross-modal ablation probe confirms this works. At 90% masking, visible page patches raise masked-token accuracy by 38.4 points for the 260M model and 40.5 points for the 800M model. Each run processes about 524 billion packed input tokens, roughly 290 billion of them text-only, on 16 and 32 H100 accelerators respectively.
Retrieval results
NeoMME-Retriever adds two jointly trained heads on the shared backbone: a mean-pooled dense head with Matryoshka widths, and a late-interaction head projecting every token and patch to 128 dimensions. One forward pass returns both.
On ViDoRe v3, the 260M model scores 0.523 nDCG@10 and the 800M model 0.556. The 260M result sits within 0.002 of ColQwen2.5-v0.2 at 3.75B parameters, and 26.1 points above the best other sub-300M model. The 800M model lands 0.9 points behind the similarly sized Vultron Retriever Flash. On ViDoRe v1 and v2 the models reach 0.860/0.522 and 0.874/0.559 nDCG@5.
Text retrieval is weaker. On BEIR-15, late interaction reaches 0.4881 and 0.5126, against 0.5722 for LateOn at 149M parameters. The authors attribute this partly to supervision scale: NeoMME saw roughly 430K text query examples, against roughly 660M contrastive examples for mLateOn.
Storage and throughput
Late-interaction indexes are expensive. A 2048×2048 page yields 4,162 vectors, about 1.5 MB per ViDoRe v3 document in float32. Two methods bring that down. Hierarchical token pooling at factor 10 with int8 queries and documents gives 39.0 kB per page, a 39.4× reduction retaining 99.16% of baseline nDCG@10. Pool factor 8 with int8 queries and binary documents gives 6.0 kB, a 255.5× reduction retaining 95.19%.
Indexing is fast for the vector count. At a matched 2048×2048 input on one L40S, NeoMME-260M encodes 51.3 pages per second against ColModernVBERT’s 26.0, a 1.97× gap.
What it means
For engineers building document search systems, this removes the need to maintain two distinct model architectures. The single stack reduces memory footprint and simplifies deployment pipelines. The quantization techniques allow dense late-interaction vectors to fit into standard memory constraints without losing precision.
