Strong yet Controllable LLM Abstention
without Learning to Abstain

Junwei Zhang^{1,2 *}, Yifei Wang^{3 *}, Wanli Yang^1,2, Hexiang Tan^1,2, Rui Tang²,
Xin Chen, Jingang Wang, Du Su¹, Fei Sun^{1 †}

¹ State Key Laboratory of AI Safety, Institute of Computing Technology, CAS
² University of Chinese Academy of Sciences ³ Beijing Institute of Technology

* Equal contribution † Corresponding author

Paper Poster BibTeX

Just select on a good answerer.

Abstention mitigates hallucination by declining to answer. An LLM can abstain in two ways. Selective answering checks each answer after it is written and withholds the doubtful ones, using a confidence threshold you can turn. More recent Abstention training teaches the model itself to say “I don’t know,” but fixes how conservatively it answers. This sacrifice of controllability can yield better performance. However, we find that, against strong selective answering, its remaining edge comes mainly from answering better, not from rejecting better. So we propose Train for Accuracy, then Post-hoc Select (TAPS). At matched coverage, TAPS reaches lower mean risk than three representative abstention-training methods, and coverage stays adjustable. These results establish TAPS as a strong baseline and raise the bar for abstention training: internalizing the decision should yield more than post-hoc selection already delivers.

Follow the story

Qwen3-8B · Natural Questions

Selective answering (Probing)TAPS (Probing)Abstention training

Selective answering41.3%

Abstention training37.3%

TAPS33.3%

01 / TWO ROUTES

Two distinct abstention paradigms.

Declining to answer is a simple way to avoid a likely hallucination. The design question is who makes the call. Abstention training teaches the model itself to output either an answer or a refusal such as “I don’t know.” Selective answering lets the model answer first; an external confidence estimator then keeps or withholds that answer.

Abstention training jointly learns answering and rejection inside the model. — **Two paradigms to abstention.** Abstention training learns answering and rejection together inside the model. Selective answering leaves the decision to an external selector.

Selective answering generates an answer, then uses an external confidence estimator and adjustable threshold to accept or withhold it. — **Two paradigms to abstention.** Abstention training learns answering and rejection together inside the model. Selective answering leaves the decision to an external selector.

02 / THE TRADE-OFF

Abstention training sacrifices controllability for performance.

Abstention trades quantity for quality. The standard risk–coverage formulation tracks quantity with coverage, the share of questions answered, and quality with risk, the error rate among those answers. A better method achieves lower risk at matched coverage (Risk@Cov). We compare TruthRL, a state-of-the-art abstention-training method, with selective answering that uses a strong confidence estimator (S.E.).

Qwen3-8B on NQ. Selective answering uses semantic entropy and traces a risk coverage curve. Three TruthRL checkpoints with abstention rewards minus 0.5, 0 and plus 0.5 cluster between 46.37 and 50.45 percent coverage. — Qwen3-8B / NQ. Selective answering uses semantic entropy (S.E.) confidence estimation. TruthRL achieves lower risk at matched coverage, but three separately trained reward variants span only 4.08 percentage points of coverage.

Better performance. TruthRL achieves lower risk at matched coverage.

Less control. Selective answering changes coverage by turning a threshold. In contrast, a trained policy can only move by retraining, and even separate runs with different abstention rewards tend to cluster in a narrow coverage range.

Can abstention be both strong and controllable?

03 / THE KEY FINDING

The performance advantage comes from better answers, not better rejection.

We analyze where the advantage of abstention training comes from. At a fixed coverage c, risk r(c) can fall in only two ways: the model answers better (fixing more answers than training breaks) or rejects better (withholding answers that would have been wrong). It is like grading a student who may skip questions on an exam: a higher score means they either learned more or skipped more wisely. We split risk exactly into these two parts: the net accuracy gain, G_ans, and rejection precision, G_dis, the share of withheld answers that would have been wrong.

Paper Figure 1. Left: exact risk decomposition into normalized net accuracy gain and rejection precision. Right: on Qwen2.5-7B-Instruct and TriviaQA at 72.7 percent coverage, semantic entropy has 86.8 percent rejection precision and TruthRL 86.2 percent, while TruthRL gains 6.1 percent from answering and has 16.2 percent risk versus 22.1 percent. — **Answering and rejecting are different skills.** On Qwen2.5-7B-Instruct / TriviaQA at the same 72.7% coverage, semantic entropy (S.E.) and TruthRL reject with nearly the same precision (86.8% vs. 86.2%). TruthRL’s lower risk comes from its answering gain.

A stronger selector closes the rejection gap fast. Across nine model–dataset settings, mean rejection precision of selective answering rises from 59.47% with Vanilla confidence to 74.84% with P(True) and 84.64% with semantic entropy, essentially level with TruthRL’s 84.29%. TruthRL’s answering gain, meanwhile, is positive in every single setting. Despite its name, then, what sets abstention training apart from strong selective answering is mostly how well it answers, rather than how well it abstains.

What if selective answering had an equally strong answerer?

04 / THE RECIPE

So: Train for Accuracy, then Post-hoc Select (TAPS).

TAPS (Train for Accuracy, then Post-hoc Select) enhances selective answering by strengthening the answerer, then leaves the abstention decision to the external estimator.

Selective answering: a starting LLM answers, then an external confidence estimator and threshold decide whether to abstain. — The same post-hoc decision, with a stronger answerer. TAPS adds accuracy training and pairs the trained model with a strong selector; the threshold remains adjustable.

TAPS keeps the same external selection mechanism, but trains the LLM for answer accuracy first and adopts a strong confidence estimator. — The same post-hoc decision, with a stronger answerer. TAPS adds accuracy training and pairs the trained model with a strong selector; the threshold remains adjustable.

STAGE 1 · TRAIN FOR ACCURACY

Improve answers with ordinary post-training. We use GRPO with a common outcome reward: +1 for a correct answer, −1 otherwise. There is no abstention objective anywhere.

STAGE 2 · POST-HOC SELECT

Score each answer with an off-the-shelf confidence estimator, such as semantic entropy or a probe on the model’s hidden states. Keep the answer if its score clears τ; otherwise, abstain.

The answerer stays fixed while τ sets coverage. TAPS simply combines two well-developed research fields through a minimal interface, so it is easy to put into real-world use and can benefit from advances in either field, making it a simple, general, and evolving approach.

05 / PERFORMANCE AND CONTROL

TAPS is competitive with abstention training while keeping coverage adjustable.

We test three LLMs on NQ, PopQA, and TriviaQA in a closed-book setting, with chain-of-thought reasoning. In each setting, TAPS and the abstention-training baselines start from the same model and train on the same data. Each block below sets TAPS to the coverage of one baseline and compares risk.

Complete paper Table 1, retaining all values, coverage bands, best-value bolding and second-best underlining. Five groups: R-Tuning, TruthRL with three abstention rewards, and UCPO. Each includes selective answering and TAPS with semantic entropy and probing across nine settings and their average. — **Risk at matched coverage (%, ↓)**. Shaded rows give each baseline’s coverage; Avg. is the mean over the nine model–dataset pairs. For TruthRL, parentheses show the abstention reward. Bold is best; underlined is second best.

Lower risk. With a probing estimator, TAPS cuts mean risk by 8.12 pp versus R-Tuning and 9.41 pp versus UCPO. Against the stronger TruthRL, it wins 17 of 27 comparisons, with a mean gain of 1.49 pp across three reward settings. The gain is uneven: largest on the reasoning model Qwen3-8B (5.05 pp below TruthRL), roughly on par on the two instruction-tuned models (0.30 pp higher risk), which highlights its promise for native reasoning models.

Risk coverage curves for all nine model and dataset settings: Qwen2.5-7B-Instruct, Qwen3-8B and Llama-3.1-8B-Instruct on NQ, PopQA and TriviaQA. TAPS with semantic entropy or probing traces full curves; abstention-trained checkpoints give individual operating points. — All three models across all three datasets. Each TAPS curve comes from one fixed answerer and one estimator; only the threshold changes. Each abstention-training marker is a separately trained model.

Control, kept. One TAPS answerer plus one estimator covers the whole risk–coverage curve. Three separately trained TruthRL reward variants, by contrast, span less than 5 pp of coverage in eight of nine settings.

Paper heatmap of changes after accuracy training. Answer accuracy rises by 2.4 to 5.9 percentage points in all nine settings, while confidence AUROC can fall, particularly for Llama. Probing is comparatively stable after refitting. — **Effects of accuracy training on answering and rejection.** ΔAcc measures the change in answer accuracy; ΔAUROC measures the change in the estimator’s ability to distinguish correct from wrong answers.

Stage 1 improves answering, but can degrade confidence estimation. Accuracy training lowers risk in 81 of 90 matched-coverage comparisons (3.55 pp on average). It does so by improving the answers: accuracy rises in all nine model–dataset settings, by 2.4–5.9 pp. However, post-training can degrade confidence estimation, potentially lowering rejection precision.

Probing in Stage 2 largely recovers the degradation. Accuracy training damages confidence estimation particularly for the three unsupervised estimators, mainly on Llama. A correctness probe, in contrast, is refit on each checkpoint’s hidden states, allowing it to adapt to the changed representations and largely recover the degradation. Therefore, probing is comparatively stable in the heatmap and gives TAPS its strongest overall results, although the best estimator still depends on the setting.

06 / THE TAKEAWAY

Insight

Abstention training wins by answering better.

Risk depends on both how well a model answers and how precisely it rejects wrong answers. A strong post-hoc selector can already match much of abstention training’s rejection precision. Against that selector, abstention training’s remaining advantage comes mainly from better answering, rather than more precise rejection. Improved abstention performance alone does not establish that learning to abstain is what helped.

Implication

Internalizing abstention should achieve more than TAPS.

TAPS combines an accuracy-trained answerer with a strong post-hoc selector and retains direct control over coverage. This makes it a more informative baseline for evaluating abstention training. A method that gives up that control should deliver further gains over TAPS, beyond improvements over the model it started from.


  @misc{zhang2026strongcontrollablellmabstention,
  title={Strong yet Controllable LLM Abstention without Learning to Abstain},
  author={Junwei Zhang and Yifei Wang and Wanli Yang and Hexiang Tan and Rui Tang and Xin Chen and Jingang Wang and Du Su and Fei Sun},
  year={2026},
  eprint={2610.16318},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2610.16318},
  }