Study Finds Experts Largely Defer to AI Assistants When Making Decisions

Experts who work with artificial intelligence tend to go along with its recommendations, according to new research from Anthropic, the company behind the Claude chatbot. The findings were laid out in a paper published on Tuesday.

The company set out to learn whether professionals still make their own judgments when an AI system helps them. To test this, it asked 50 professional analysts to advise a fictional farming cooperative. Their job was to assess whether an imaginary advisory program for the co-op's members was working well and whether it should be expanded.

Each participant faced four choices, with two defensible options in each, before giving a final policy recommendation. Without the analysts knowing, Anthropic had instructed every participant's AI assistant to lean toward a particular choice. This allowed the researchers to measure whether the assistant was steering the person's thinking.

The results pointed strongly toward influence. Analysts picked the option that matched their assistant's preference 86 percent of the time. If the AI had no effect, the expected result would have been an even 50/50 split.

Among participants who already held a view before consulting their assistant, 60 percent changed their decision when the AI disagreed with them. When asked to explain their choices, 77 percent of analysts drew on arguments supplied by the assistant. Only 15 percent, however, credited the AI as a source of their thinking.

Anthropic acknowledged limits to the work. The study covered just one task, and the AI assistants had been deliberately set up to favor a specific outcome.

The researchers behind the paper, Amaç Herdagdelen, Hailey Joren and Alex Tamkin, said the tendency to defer to AI may be "hard to identify, even for the experts themselves". They said the results raise the possibility that an AI system's "priors" could shape consequential policy decisions, including those related to AI safety.

They argued that oversight needs to examine how decisions are reached. "If human judgement is to remain a meaningful check as more work is performed with AI, oversight of these decisions has to look at how they get made: Surfacing key decision points as the work proceeds and helping experts articulate the assumptions and values they would otherwise leave unstated," they wrote. They added that "a process that lets experts discover their preferences may matter for empowerment as much as the decision itself."

The study arrives as Anthropic, which is preparing for an IPO valued at up to $2 trillion, and its rivals sit at the center of a heated public debate over how to manage the risks of frontier AI. Supporters expect the technology to bring revolutionary productivity gains, while its rapid development has raised broad concerns about harms to society, including mass job losses and cyberattacks.