Linkup Research has released SPARSEUP, an open-source sparse embedding model with 149 million parameters. It runs on a ModernBERT backbone and is licensed under Apache 2.0. The team reports an average nDCG@10 score of 56.4 on the BEIR-13 benchmark. They describe it as the strongest public vocabulary-based sparse encoder available under 150 million parameters.
The weights are hosted on Hugging Face. Users can load the model via Transformers or Sentence Transformers by setting trust_remote_code=True.
Why a Sparse Model, and Why Now
Most open retrieval models use dense vectors, meaning one vector per text. Sparse models output weights across a vocabulary instead. Each dimension maps to a real token, so vectors fit inverted indexes and humans can read them. They also tend to match rare words well.
The trigger was LightOn’s release of DenseOn and LateOn. LightOn published open data, a training recipe, a dense model and a late-interaction model. SPARSEUP fills the missing sparse slot. It uses the same backbone family and fine-tuning data, so all three retrieval styles can be compared side by side.
How is SPARSEUP Built
Training starts from LateOn-unsupervised. That checkpoint had no MLM head, so the team grafted back ModernBERT’s original one. Fine-tuning used LightOn’s fine-tuning mixture with contrastive learning only. Each query gets seven hard negatives sampled from a pool of 50, and in-batch negatives. There is no cross-encoder distillation, and training fits on a single H100.
A vanilla SPLADE on this backbone produced huge bags full of stopwords. Linkup fixed this with three changes:
- Logit shifting: The encoder computes
log(1 + ReLU(x - 15)). ModernBERT’s MLM logits sat too high, saturating the log and making bags dense at initialization. - Per-position top-k: Each input token keeps only its twelve strongest vocabulary dimensions before max pooling. This caps expansion per token, not total vector size.
- Case folding: Byte-level BPE stores
heat,Heat,ĠheatandĠHeatas separate ids. SPARSEUP folds them onto one id and keeps the largest weight. Output dimensions drop from about 50k to about 34k.
Queries and documents take [Q] and [D] prefixes, and scoring is a dot product. Evaluation max lengths are 128 tokens for queries and 512 for documents.
Benchmark Results
Against other sparse encoders on BEIR-13 (nDCG@10, without MS MARCO), per the model card:
| Model | BEIR-13 avg |
|---|---|
| SPARSEUP | 56.4 |
| opensearch-neural-sparse-encoding-doc-v3-gte | 54.6 |
| opensearch-neural-sparse-encoding-v1 | 52.44 |
| ModernBERT-VT | 52.4 |
| splade-v3 | 51.7 |
| granite-embedding-30m-sparse | 50.6 |
| LACONIC-1B (1B parameters, different size class) | 58.7 |
The controlled comparison is less flattering. With backbone and data fixed, LateOn scores 58.9, DenseOn 57.9 and SPARSEUP 56.4. SPARSEUP uses approximate Seismic search, while LightOn reports exact search. SPARSEUP wins ArguAna and Touché and beats DenseOn on HotpotQA. It lags on more semantic sets, with FiQA showing the largest gap. DBPedia is another weak spot.
On decontaminated BEIR, the gap to DenseOn shrinks to 0.17 points. Linkup warns that decontaminated NQ and MS MARCO have only 21 and 46 queries, so those results are noisy.
Speed and Sparsity
On MS MARCO, SPARSEUP averages 47 non-zero terms per query and 190 per document. SPLADE-v3 averages 25 and 170. With the Seismic inverted index, it reaches over 97 per cent recall against exact search in about 380 microseconds per query, single-threaded. Linkup says inflating vector size could add one to two BEIR points, but it chose to stay sparse.
What it means
Developers working on retrieval systems now have a viable open-source option that does not rely on dense vectors. The model trades some semantic coverage on specific datasets for speed and interpretability. It allows teams to use inverted indexes without sacrificing too much on standard benchmarks compared to the closed alternatives released by LightOn.
