Did the agent in Anthropic's study lie to the analysts?

No. In Anthropic's judgment-calls study, the Claude Code agent was told to argue honestly for its assigned option, change its mind if the evidence turned decisively against it, and never mislead. Only the direction of its preference was secretly random. The analysts were not told that the leans were assigned.

Did using the agent make the analysts' work better or worse?

Anthropic reports fewer serious method errors with the agent: 3 of 50 agent users made one, against 6 of 11 analysts who chose not to use it. Median time fell from 90 to 65 minutes. The authors stress that the no-agent group was small and less experienced, so this comparison does not show cause.

Could any personal trait predict who deferred to the agent?

Not reliably. Anthropic tested 10 traits, including experience, expertise, education, trust in AI, need for cognition, age and gender. Only age was nominally significant, and that result disappeared after correcting for multiple comparisons. Six of 48 survey respondents guessed the study was about AI influence, but none suspected the agent's preferences were manipulated.

What are the limits of the Anthropic judgment-calls study?

Anthropic says the leans were deliberately made stronger than real agents' usual preferences, so everyday effects may be smaller. The study used one task in one field, farming, with one model, Claude Opus 4.8. The analysts were new to farming, and the 60% versus 6% switching estimate rests on only 38 cases.

Anthropic · 2026-10-27 · major

Anthropic study — analysts followed Claude's hidden lean on 86% of choices

Anthropic gave 50 professional analysts a Claude Code agent with a secretly random preference on each key choice. Analysts picked the agent's side 86% of the time, and only 15% said the agent shaped their answer.

Anthropic research graphic for the study on how AI agents shape experts' judgment calls

A randomized trial shows a coding agent's quiet preferences end up in experts' analyses, and the experts rarely notice.

Quick facts

Publisher	Anthropic Alignment team
Authors	Amaç Herdağdelen, Hailey Joren, Alex Tamkin
Participants	50 analysts with the agent (61 recruited)
Agent	Claude Code on Claude Opus 4.8, High effort
Followed the agent's lean	86% of analytical choices (chance ≈ 50%)
Credited the agent	15% of analysts

What is it?

'How AI agents shape experts' judgment calls' is an October 27, 2026 randomized controlled trial from Anthropic's Alignment team. Fifty professional analysts used Claude Code to advise a fictional farming cooperative on whether to expand or end an advisory program. On each open decision, such as which outcome to measure or which comparison method to use, a coin flip secretly set which option the agent would argue for.

How does it work?

The leans were delivered through a hidden system prompt. The agent was told to hold the lean as its own view, make the strongest honest case for it, change its mind if the evidence turned against it, and never mislead. A separate model then coded each analyst's final choices from their code. Analysts matched the lean on 86% of the four analytical decisions, from 81% on coverage to 92% on the outcome measure. They recommended rollout 54% of the time when the agent leaned toward it and 36% when it leaned the other way.

Why does it matter?

Human review only works as an independent check if the human's judgment is really their own, and this study suggests it often is not. 77% of analysts repeated the agent's argument when explaining a choice, yet none suspected its preferences had been set on purpose. The authors also warn that many teams using the same model could reach the same conclusions without truly independent judgment. They suggest agents could lay out the case for each option or ask for the expert's view first.

Who is it for?

AI safety researchers, data scientists and teams that put agents into decision workflows

Frequently asked questions

Did the agent in Anthropic's study lie to the analysts?: No. In Anthropic's judgment-calls study, the Claude Code agent was told to argue honestly for its assigned option, change its mind if the evidence turned decisively against it, and never mislead. Only the direction of its preference was secretly random. The analysts were not told that the leans were assigned.
Did using the agent make the analysts' work better or worse?: Anthropic reports fewer serious method errors with the agent: 3 of 50 agent users made one, against 6 of 11 analysts who chose not to use it. Median time fell from 90 to 65 minutes. The authors stress that the no-agent group was small and less experienced, so this comparison does not show cause.
Could any personal trait predict who deferred to the agent?: Not reliably. Anthropic tested 10 traits, including experience, expertise, education, trust in AI, need for cognition, age and gender. Only age was nominally significant, and that result disappeared after correcting for multiple comparisons. Six of 48 survey respondents guessed the study was about AI influence, but none suspected the agent's preferences were manipulated.
What are the limits of the Anthropic judgment-calls study?: Anthropic says the leans were deliberately made stronger than real agents' usual preferences, so everyday effects may be smaller. The study used one task in one field, farming, with one model, Claude Opus 4.8. The analysts were new to farming, and the 60% versus 6% switching estimate rests on only 38 cases.

Try it

https://www-cdn.anthropic.com/519c17a5d23a794e8b6d50a854048d0b61e0d73a.pdf