You can get banned for being too ‘cruel’ to Claude as Anthropic updates its policy

Anthropic, the company behind Claude, is drawing a line when it comes to how people treat its artificial intelligence models. The company has updated its usage policy to prevent users from showing “sustained and needless abusive or cruel behavior” toward its models. The change comes as there are new challenges and the emergence of high-risk cases involving the use of AI, as per The Guardian.

“We’ve added a prohibition on sustained and needless abusive or cruel behavior toward our models. The policy update is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose,” said the company in its announcement. They did not comment on what it considers ‘abusive or cruel.’

It did note that the change ‘does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research.’ Users can still challenge Claude, criticize its responses, and experiment with how it handles difficult prompts. Claude can also report people to police over threats of violence, while OpenAI is also known to ban accounts over safety protocols.

Claude can ban you

The company has also given its models the ability to end conversations when users insist on engaging in harmful behavior or dialogues. The feature is claimed to safeguard AI’s welfare. Some of these behaviours might include “requests from users for sexual content involving minors and attempts to solicit information that would enable large-scale violence or acts of terror,”  as Anthropic stated in one of its earlier press releases.

“We remain highly uncertain about the potential moral status of Claude and other LLMs, now or in the future. However, we take the issue seriously, and alongside our research program we’re working to identify and implement low-cost interventions to mitigate risks to model welfare, in case such welfare is possible. Allowing models to end or exit potentially distressing interactions is one such intervention,” said the company.

In an interview with The New York Times, Dario Amodei, Anthropic’s CEO, said he cannot rule out the possibility of AI systems being conscious. He further suggested that the question remains open as AI technology advances.

OpenAI CEO Sam Altman, meanwhile, has expressed his dislike toward the idea in a post made on X. “I am very uncomfortable about people trying to ascribe religious force or a surrender of human judgment to AI models, and think it is a real safety issue,” he wrote.

​

​