Publikation
Soft-Prompt Tuning for Fair and Efficient LLM Benchmark Evaluation
Selen Erkan; Bastian Boll; Kristian Kersting; Björn Deiseroth; Letitia Parcalabescu
In: Computing Research Repository eprint Journal (CoRR), Vol. abs/2606.12117, Pages 1-34, arXiv, 2026.
Zusammenfassung
Benchmark scores often misrepresent a large language model’s (LLM’s) knowl-
edge, because they rely, e.g., on the model’s ability to follow specific formatting
requirements. This especially penalizes base models that may know the correct
answers but lack the ability – typically introduced in post-training – to structure
them as instructed. To overcome this, we propose soft-prompt tuning, an effi-
cient, fair, and architecture-agnostic model evaluation. By optimizing only 10
soft-prompt vectors (roughly 0.0006% parameters for a 7B model) over a short
tuning period, we adapt models to specific benchmark formats, closing gaps in
format-following and ensuring that underlying knowledge is accurately reflected
in benchmark scores. This allows one to fairly compare different base models –
trained with various pre-training recipes – on benchmarks without the need for full
post-training. We evaluated soft-prompt tuning across 7 models and 7 datasets. The
results show that (a) soft-prompt tuning saturates format-following within 80 steps
(∼640 samples) making it highly efficient, (b) soft-prompt tuning significantly
outperforms zero- and few-shot prompting, surfacing base model knowledge that
standard prompting misses, that (c) even post-trained models can benefit from
soft-prompts to maximize format compliance, and that (d) soft-prompted base
model performance predicts post-trained model rankings more reliably than zero-
and few-shot baselines, offering a low-cost proxy for downstream model quality.
Our contributions include (1) metrics which disentangle format-following and
knowledge accuracy, (2) a fairer benchmarking protocol of LLM knowledge, and
(3) a cost- and memory-effective recipe to identify optimal pre-training strategies
early in LLM development.