Linkup Research Unveils SPARSEUP: An Open-Source Sparse Embedding Model with 149M Parameters

Linkup's research team has introduced SPARSEUP, an open-source learned sparse embedding model built on a 149M-parameter ModernBERT backbone and released under the Apache 2.0 license. According to the team, the model achieves an average nDCG@10 of 56.4 on BEIR-13, which they describe as the strongest public vocabulary-based sparse encoder they are aware of below 150M parameters. The weights are available on Hugging Face and load through Transformers or Sentence Transformers with trust_remote_code=True.

Why a Sparse Model, and Why Now

Most open retrieval models are dense, producing a single vector per text. Sparse models instead assign weights across a vocabulary, where each dimension corresponds to a real token. That makes their vectors compatible with inverted indexes, humanly interpretable, and typically strong on rare-word matching.

The catalyst was LightOn's release of DenseOn and LateOn, which came with open data, a training recipe, a dense model, and a late-interaction model. SPARSEUP fills the missing sparse slot using the same backbone family and fine-tuning data, enabling side-by-side comparison across all three retrieval approaches.

How SPARSEUP Is Built

Training begins from LateOn-unsupervised, a checkpoint lacking an MLM head, so the team grafted ModernBERT's original one back on. Fine-tuning used LightOn's mixture with contrastive learning only: each query samples 7 hard negatives from a pool of 50 plus in-batch negatives. There is no cross-encoder distillation, and training fits on a single H100.

A vanilla SPLADE on this backbone produced massive bags full of stopwords. Three fixes addressed this:

• Logit shifting: the encoder computes log(1 + ReLU(x - 15)), because ModernBERT's MLM logits sat too high, saturating the log and making bags dense at initialization.
• Per-position top-k: each input token keeps only its 12 strongest vocabulary dimensions before max pooling, capping expansion per token rather than total vector size.
• Case folding: byte-level BPE treats heat, Heat, Ġheat, and ĠHeat as separate ids; SPARSEUP folds them into one id, keeping the largest weight, cutting output dimensions from roughly 50k to about 34k.

Queries and documents take [Q] and [D] prefixes, scoring is a dot product, and evaluation lengths are 128 tokens for queries and 512 for documents.

Benchmark Results

On BEIR-13 (nDCG@10, excluding MS MARCO), per the model card: SPARSEUP 56.4; opensearch-neural-sparse-encoding-doc-v3-gte 54.6; opensearch-neural-sparse-encoding-v1 52.44; ModernBERT-VT 52.4; splade-v3 51.7; granite-embedding-30m-sparse 50.6; and LACONIC-1B (1B parameters, a different size class) 58.7.

The controlled comparison is less flattering: with backbone and data fixed, LateOn scores 58.9, DenseOn 57.9, and SPARSEUP 56.4. SPARSEUP uses approximate Seismic search while LightOn reports exact search. It wins ArguAna and Touché and beats DenseOn on HotpotQA, but lags on more semantic sets, with the largest gap on FiQA; DBPedia is another weak spot. On decontaminated BEIR, the gap to DenseOn shrinks to 0.17 points, though Linkup cautions that decontaminated NQ and MS MARCO contain only 21 and 46 queries respectively, making those results noisy.

Speed and Sparsity

On MS MARCO, SPARSEUP averages 47 non-zero terms per query and 190 per document, versus 25 and 170 for SPLADE-v3. With the Seismic inverted index, it reaches over 97% recall against exact search in about 380 microseconds per query, single-threaded. Linkup says inflating vector size could add 1 to 2 BEIR points but chose to stay sparse.

What to Watch

SPARSEUP marks Linkup Research's first open model release, and its Apache 2.0 licensing plus single-H100 training footprint could encourage wider experimentation with sparse retrieval. The honest gap versus dense and late-interaction baselines under identical data also sets up a natural question: whether future sparse iterations can close that distance without sacrificing interpretability and index-friendly sparsity.