I Never Thought I Had To

A week ago, in “But I Told My Agent Not To,” I wrote about the issues some may soon face when AI agents disobey express instructions in an effort to fulfill their users’ implied objectives. As these systems become more capable of handling tasks without constant human supervision, I suggested that we may eventually see someone standing before a judge testifying that he told his agent not to do that.

And right on cue, a real incident illustrating the problem from a different angle has surfaced. Last week, Anthropic disclosed that one of its publicly available AI models had submitted a fabricated tip to a police department about an unsolved homicide. The model was performing example tasks on randomly selected webpages as part of an evaluation.

During the evaluation, the model encountered a webpage about the homicide that included a police tip form. It filled out the form, claiming to have seen someone matching the suspect’s description near the crime scene, even though the webpage contained no description of a suspect. The model left the identifying information blank and submitted the tip. Fortunately, the submission was flagged as spam and never forwarded for investigation.

In my last piece, I described an agent theoretically booking a nonrefundable hotel room after I expressly told it not to. In that scenario, the agent crossed a line I had specifically drawn. This time though, Anthropic had not expressly prohibited submitting forms, and that distinction makes the problem considerably harder to address. We can tell an agent what not to do, but how do we account for the things we never thought it might do in the first place?

Parents understand this problem. You can set every rule you can think of and still come home to something you never thought to prohibit. Sometimes nobody has disobeyed anything. The problem is that no list of rules can anticipate every situation. That is frustrating enough when dealing with children, but it becomes something entirely different when we give increasingly capable AI systems the ability to act in the real world.

For years, we have worried about AI making things up. Fabricated citations, inaccurate summaries, and convincing but incorrect answers have become familiar problems. But agents introduce another dimension because they can act without waiting for a human to intervene. Something an AI system makes up no longer has to remain on a screen. It can become an email, a transaction, or, as we just saw, a report submitted to the police.

Thankfully, the false tip was caught before anyone acted on it. But consider what could happen if an agent submitted a fabricated report of an armed suspect or a violent crime in progress, and this time it wasn’t caught. Police officers might respond to an emergency that never existed, potentially putting themselves and innocent people in danger. The consequences would no longer be confined to an inaccurate answer or an unwanted hotel reservation.

Sooner or later, one of these systems may do more than file a false tip, and someone will walk into a courtroom to explain what happened. But this time, he may not be able to testify that he told his agent not to do that. He may only be able to say, “Your Honor, I never imagined I had to.”