Florida woman allegedly treats Claude like a personal diary. Then Anthropic reads her entry and calls the police: ‘Threat of violence’

A woman in Bonita Springs, Florida, is facing a criminal charge after a tip reached the Lee County Sheriff’s Office from an unusual place. As first reported by TechSpot, the tip came after Claude flagged the woman’s messages.

According to the arrest report, a user identified as Carli Michelle Heller wrote on Sept. 26 that she was going to “shoot up” the sheriff’s office. Claude’s safety systems flagged the entry and escalated it to a human reviewer, who determined it was a credible threat and reported it to law enforcement. 

Deputies identified Heller, went to her home, and detained her without incident. An LCSO intelligence detective then took over the case. She is charged with making a written threat of violence under Florida law.

Heller said she used Claude like a diary

Sheriff Carmine Marceno told WINK News that Heller later said she uses AI like a “diary.” Heller’s description suggests she viewed the chatbot as a private place to write.

Marceno said in a statement, “Artificial intelligence is a powerful tool, and like any technology, it can be misused. “When someone uses AI to make or facilitate a threat, we have to take that threat seriously.” He added that users are “never truly anonymous” in AI chatrooms and searches.

One person said the case “straddles the murky line between Winston writing in his diary, coupled with Minority Report pre-crime, and preventing actual lunatics from committing a crime.” Another pointed to a gap in the reporting. “Strange we didn’t hear about any specifics of her prior prompts.” A third wrote. “I’m certainly not going to ‘make friends’, with a Chat Bot.”

Anthropic‘s published Usage Policy says its Safeguards Team uses detection and monitoring to enforce its rules. The policy bans using its products to facilitate or promote violence or intimidation. Its government request policy says user information may be shared with law enforcement in limited emergencies involving danger of death or serious physical injury.

Other AI companies have faced the same question from the opposite side. OpenAI, for one, banned an account months before the Tumbler Ridge shooting in British Columbia without calling police. It has since said it will loosen its threshold for alerting authorities.

Experts told WINK News that the conversational nature of chatbots can make users forget that their conversations are being recorded. Ian Marlow, CEO of FITECH, said the conversational tone hides the fact that a record exists: “Something that feels that it is conversational in nature doesn’t immediately feel like, ‘Oh, this is something that is being recorded,’ because they’re just having a conversation,”

Dr. Chrissann Ruehle, an assistant professor of AI and management at Florida Gulf Coast University, said people may open up because a chatbot seems nonjudgmental. “There’s also a sense of, if I interact with a machine, the machine isn’t going to judge me,” Ruehle said. “AI is not your best friend,” she added. “We need to keep in mind this is a technology.”

Chatbot logs have come up in Florida cases before, such as what the Florida State University shooter asked ChatGPT. Retired FBI Special Agent Richard Kolko told the outlet that the legal side is still unsettled.  “This is all new stuff into the world and courts are still figuring it out,” Kolko said.

As of now, Heller has a court date set for November. WINK News said it asked Anthropic for comment on how its safeguards handle threats, but the outlet had not received a response at the time of publication.