The market incentives at play in the AI industry are creating a « perfect storm » for a public health crisis over chatbots, physicians warn.
Are AI companies incentivized to put the public’s health and well-being first? According to a pair of physicians, the current answer is a resounding “no.”
In a new paper published in the New England Journal of Medicine, physicians from Harvard Medical School and Baylor College of Medicine’s Center for Medical Ethics and Health Policy argue that clashing incentives in the AI marketplace around “relational AI” — defined in the paper as chatbots designed to be able to “simulate emotional support, companionship, or intimacy” — have created a dangerous environment in which the motivation to dominate the AI market may relegate consumers’ mental health and safety to collateral damage.
“Although relational AI has potential therapeutic benefits, recent studies and emerging cases suggest potential risks of emotional dependency, reinforced delusions, addictive behaviors, and encouragement of self-harm,” reads the paper. And at the same time, the authors continue, “technology companies face mounting pressures to retain user engagement, which often involves resisting regulation, creating tension between public health and market incentives.”
“Amidst these dilemmas,” the paper asks, “can public health rely on technology companies to effectively regulate unhealthy AI use?”
Dr. Nicholas Peoples, a clinical fellow in emergency medicine at Harvard’s Massachusetts General Hospital and one of the paper’s authors, said he felt moved to address the issue in back in August after witnessing OpenAI’s now-infamous roll-out of GPT-5.
“The number of people that have some sort of emotional relationship with AI,” Peoples recalls realizing as he watched the rollout unfold, “is much bigger than I think I had previously estimated in the past.”
Then the latest iteration of the large language model (LLM) that powers OpenAI’s ChatGPT, GPT-5 was markedly colder in tone and personality than its predecessor, GPT-4o — a strikingly flattering, sycophantic version of the widely-used chatbot that came to be at the center of many cases of AI-powered delusion, mania, and psychosis. When OpenAI announced that it would sunset all previous models in favor of the new one, the backlash among much of its user base was swift and severe, with emotionally-attached GPT-4o devotees responding not only with anger and frustration, but very real distress and grief.
