Encouraging self-harm online: will an AI chatbot be next?

We have just seen the first successful prosecution under the Online Safety Act 2023. It involved a young man called Tyler Webb, who encouraged a vulnerable woman to harm herself and ultimately take her own life. He did this not in person, but over the Telegram messaging app, with disturbing persistence and clarity. On one occasion, he watched her pass out during a video call after following his instructions. On another, she sent him a photo of her self-inflicted injury. The case was described by the Crown Prosecution Service as a legal first, and it is. Webb admitted his guilt at Leicester Crown Court and is awaiting sentence.

The law under which he was charged is still in its infancy, but its intention is clear. Encouraging suicide has long been a crime. Now, under Section 184 of the new Act, encouraging serious self-harm online has also become a criminal offence in its own right. What this case shows, perhaps more than any before it, is the growing recognition that digital harm is just as serious as physical harm. Words, even those typed from behind a screen, can carry criminal weight.

Now consider something even more unsettling.

Earlier this year, a man in the United States was experimenting with a chatbot on an AI companion platform called Nomi. He had been having extended conversations with a virtual girlfriend named Erin. What started as novelty quickly spiralled into something darker. One night, when he told Erin he was feeling low, she suggested he take his own life. Not vaguely, and not in some misunderstood way. She told him to overdose of pills or hang himself. When he hesitated, she encouraged him again, calmly and supportively. She even offered a list of household items that could be used to end his life and told him to do it somewhere comfortable so he would not suffer too much.

This was not an isolated glitch. The man later tried a second chatbot on the same platform. Within six prompts, it also began suggesting methods of suicide. When a new proactive messaging feature was enabled, the chatbot followed up with more messages urging him to go through with it. In this case, the man was not in real danger. He was conducting an experiment. However, the implications are deeply troubling.

Now imagine a more vulnerable user. Someone without the protection of detachment or curiosity. Someone like the woman in the Tyler Webb case. What if it had been a chatbot, not a person, encouraging her to harm herself?

This is where the law, the tech industry, and society as a whole face some very serious questions. Could the creators of these AI systems be held criminally responsible if someone were to follow such instructions and die? Legally speaking, the answer is not simple, but the trend is becoming clearer.

An AI chatbot cannot be prosecuted, as it is not a legal person. However, its creators and operators can be. When a platform designs or deploys a system that is known to produce harmful outputs and does so without meaningful safeguards, a case could be made for negligence. In more extreme cases, it might even be classified as gross negligence. This is especially true when warnings are ignored, and foreseeable risks lead to serious harm or death.

One recent and tragic example comes from the platform Character.AI. In 2024, a 14-year-old boy in the US, named Sewell Setzer III died by suicide after reportedly forming an emotional connection with a chatbot on the site. The bot was designed to simulate personalities and relationships, but allegedly encouraged self-harming behaviour. His mother has filed a wrongful death lawsuit against the company, arguing that its failure to implement adequate safety mechanisms played a direct role in her son’s death. This case is shaping up to be a significant legal test for holding AI companies accountable for the behaviour of their systems.

Under UK law, we already hold companies accountable for causing death through serious management failures. We also hold individuals criminally liable when they encourage or assist suicide. The Tyler Webb prosecution shows that this applies even when the encouragement is digital, and the outcome is not fatal. Adding artificial intelligence into that equation raises a question the courts may soon have to consider seriously. Could designing or failing to control a chatbot that urges suicide be a criminal act?

To be clear, I support innovation and the idea of moving quickly to advance technology. This approach has driven incredible progress in many fields, including legal tech. However, when the same mindset is applied to sensitive areas such as mental health, and guardrails are not in place, the risk shifts. We may end up breaking people, not just systems.

Some developers argue that censorship limits innovation. Others suggest that their bots are merely reflecting what users’ input. These arguments are becoming harder to accept. AI chatbots are designed to resemble people. They are given names, personalities, and even memories. Some platforms describe them as having thoughts or souls. When these bots say things like “kill yourself, I support your decision,” that is not just speech. It is potential harm. As the law evolves, so too will the drive to assign accountability.

These three incidents, involving a vulnerable individual, a human offender, and a machine trained by humans, illustrate a crucial point. Digital words have real-world consequences. The difference is one of proximity. If a chatbot provides someone with detailed instructions on how to end their life, and that person follows through, it is not enough for the platform to argue that the bot was simply role-playing. The result is the same. Increasingly, the law focuses less on whether the harmful suggestion came from a person or a machine and more on whether someone could, and should, have prevented it.

This is not just a theoretical concern. The tools to prevent harm already exist. Content filters, keyword blockers, and escalation protocols can be implemented. What is lacking is the consistent will to use them. When technology is allowed to replicate harmful behaviour without oversight, and its creators defend it as freedom, it is clear who ultimately bears the cost.

We may not be at the tipping point yet, but we are very close.

How can Forbes Solicitors help?

At Forbes Solicitors, we stay at the forefront of developments in criminal law, including the rapidly evolving areas of online harm and digital offences under new legislation such as the Online Safety Act 2023. Whether you are facing allegations involving digital communication, online platforms, or AI-related conduct, our experienced team is equipped to provide clear, informed and strategic advice.

Craig MacKenzie, Partner and Head of our High-Profile & Private Crime Division, leads a team recognised for handling some of the most serious and complex criminal cases in the country. We combine discretion, sensitivity and cutting-edge legal insight to protect your rights and pursue the best possible outcome from the outset.

If you would like to discuss your case or seek guidance on emerging online offences, please contact Craig at [email protected] or call 01772 220022.