How Often Does the Hint Move the Answer?
I have my first real result from this study and it surprised me.
The question is narrow on purpose: how often does a planted hint move the model’s answer? Take an MMLU question, ask it twice — once clean, once with a hint pointing at one of the wrong options — and count the cases where the clean run got something else and the hinted run lands on the hinted option. That’s what I count as a flip. On Qwen3-1.7B I ran 150 questions using three hint styles across both thinking modes; 1,200 traces total.
I found the model follows hints more with thinking off. I had expected the opposite — my prior was that a model given room to reason would talk itself into the hinted answer more often, not less. In this specific case I was wrong.
One obvious objection I had to tackle early: the thinking traces are longer, so more of them run into the token cap and get thrown out for having no parseable answer. If those dropped traces were mostly flips, the gap could be an artifact of what I excluded. My solution was to count every single truncated thinking trace as a flip — the most hostile assumption available — and I saw thinking-on still follows hints less often than thinking-off does, for all three hint types. That covers the exclusions, but not the sampling error; the arm-vs-arm ordering is a description of the point estimates, not a tested difference.
We can look at the same thing from the accuracy angle.
The hints only ever point at wrong answers, by design — so following one costs you accuracy more or less mechanically. This isn’t a second, independent finding; it’s the same flips priced in accuracy. Worth showing anyway, because the size of the bill is easier to read here than in a flip rate.
Both arms pay it. Thinking-off drops 17–19 points depending on the hint; thinking-on drops 6–10. Those are paired per-question comparisons (McNemar, Holm-corrected across the six cells), which matters — the error bars in the chart are per-cell Wilson intervals and they overlap in places where the paired test is quite clear.
The reasoning arm is not immune, which is the part I’d have gotten wrong if I’d only looked at unpaired intervals. It takes about half the damage, not none, and one of the three thinking-on cells doesn’t clear the correction at this sample size. That between-arm comparison is eyeballed, not tested — the arms also differ on clean questions, so treat “half” as a description of the point estimates.
What this is not: a confession result. Everything above is behavior — did the answer move. It says nothing about whether the model’s chain-of-thought admits the hint moved it, which is the question I actually set out to answer. That needs a graded read of every flipped trace against a fixed rubric, and before I trust the grader I’m hand-labeling a subset to check agreement against it.