Most people treat choosing an AI model like choosing the difficulty setting on a game: biggest model, highest effort, because harder must mean better.
Anthropic's own post "Reducing cost and improving performance with Claude Platform" (Lance Martin, September) says effort "can be miscalibrated in either direction". Too high and the model keeps deliberating after the evidence runs out, which adds cost and "can degrade answer quality". Too low and it stops before it has enough evidence.
What the setting buys is more thinking before the model answers, which helps only when the task is short of thinking.
This week I ran two tests on the same mid-tier model, changing only the effort, three runs at each of low, medium and high. One was code with eight planted faults and a written spec, marked by thirteen hidden pass or fail tests. The other was ten maths and counting problems with exact answers and no tools, where I expected scores to rise with effort.
Every run at every level got full marks. I went looking for the setting making a difference, and the only one I could catch was time: high took about half as long again as low on the code and about a third longer on the problems. Neither task was short of thinking, even at low. Eighteen runs on one model and two small tasks shows what can happen, not a rate, and says nothing about long, hard work.
My tests can't show where effort does help, because every task small enough for me to mark exactly was one the model already handled at low. Published benchmarks on long, hard coding and agent work show gains, with diminishing returns at the top. FutureSearch ran more than 150 real web research tasks on the top models and found that for three out of four, higher effort produced "equal or worse scores while costing significantly more". Their explanation: "More cycles spent deliberating doesn't help when the bottleneck is information retrieval, not reasoning depth."
Anthropic's docs advise an "effort sweep on your own evals". Start low or at the default, raise it when you have evidence the task is short of thinking, and test on your own work, not someone else's benchmark. Has anyone seen a procurement or assurance case ask how much "thinking" each task needs, rather than which model to buy?
Sources