US researchers have warned that growing reliance on artificial intelligence-powered chatbots for health information and advice could lead to harm that is difficult to detect or trace. Their concerns come amid a lack of independent systems to monitor the information these applications provide and clear mechanisms for reporting errors they generate.
اضافة اعلان
In a scientific paper published in Nature Health, the researchers said the risk goes beyond the possibility of inaccurate medical information. It also includes what they described as the “structural invisibility of harm,” whereby conversations that may lead to dangerous health decisions remain within technology companies’ platforms, beyond the reach of doctors, researchers and regulators responsible for protecting public health.
The paper was prepared by researchers from Stanford University, Binghamton University, Texas A&M University and Indiana University. It examines the risks associated with using generative AI systems to provide health information, whether through direct conversations with users or automated summaries appearing in search engines and on social media platforms.
The researchers explained that applications powered by large language models can offer simplified medical explanations and answer health questions around the clock, potentially helping people who face language barriers or have limited access to healthcare services. However, these benefits come with risks, including incorrect answers presented confidently and persuasively, misleading simplifications of complex medical issues, and reliance on outdated information or false claims.
Challenges in investigating incidents
The researchers cited the case of a man in his 60s who was hospitalized with bromide poisoning after saying he had used ChatGPT to obtain information about reducing chloride in his diet. He subsequently replaced table salt, or sodium chloride, with sodium bromide, a chemical compound that is not a safe dietary substitute for salt, for three months.
Consuming the compound led to bromide poisoning and hospitalization after the man developed neurological and psychological symptoms, including hallucinations and paranoia. Doctors were unable to access the original conversation he had with the application, preventing them from determining precisely what information he had received or reconstructing the sequence of events that led him to make the decision.
The researchers said the incident, which had previously been documented in a medical report published in 2025, highlights the difficulty of investigating health incidents associated with AI.
The inability to retrieve the conversation also makes it impossible to establish whether the application directly recommended using the compound or how the user interpreted the information he received.
Study co-author Kai Cheng Yang, an assistant professor in the School of Computing at Binghamton University in the United States, said millions of users are now receiving medical information from chatbots, while insufficient oversight makes it difficult to trace errors arising from these interactions or determine the extent of the harm they may cause.
He added that the problem lies in the fact that medical advice provided by these systems is not subject to the documentation and review procedures used by healthcare institutions. Such interactions are generally not recorded in patients’ medical files, and independent researchers lack sufficient data to assess their effects on treatment decisions.
Researchers have also developed an AI technique that analyzes just 20 seconds of a person’s voice to identify those most at risk of developing type 2 diabetes.
The researchers believe this situation creates a serious gap in patient-safety systems. Detecting conventional medical errors depends on the availability of records that can be reviewed, complaints that can be investigated, and data that can reveal recurring patterns of harm.
With chatbots, however, users may make health decisions based on incorrect answers without their doctors knowing where the information came from or any independent body being able to identify that an error occurred.
The researchers also warned that exposure to misleading health information is no longer limited to people who actively seek medical advice. AI-generated summaries have become part of search-engine results, while automated content-generation tools have spread across social media platforms.
Yang said users may encounter AI-generated health information while browsing their social media accounts, even when they are not searching for medical topics. He cautioned that such information could influence people’s behavior and health decisions, particularly when it contains misleading claims or material published by groups seeking to manipulate public debate.
The researchers added that the ease and low cost of producing personalized health information using AI also enable malicious actors to spread misleading messages targeting specific groups of users. This could make it harder to distinguish reliable medical guidance from content designed to influence behavior.
Recommendations to address the risks
To address these risks, the researchers recommended requiring AI companies to provide users with access to their complete health-related conversation histories, allowing them to share the records with doctors when necessary and helping identify information that may have contributed to harm.
They also called for clear systems to report dangerous or incorrect medical advice so that incidents can be documented, investigated and systematically analyzed, rather than leaving their handling solely to internal procedures established by technology companies.
The researchers proposed that social media platforms assume greater responsibility for verifying AI-generated health content by clearly labeling it and subjecting medical information to review by qualified specialists before making it available to users. They also recommended that search engines rely only on scientifically verified information in their automated health summaries.
At the legal level, the researchers urged policymakers to consider extending professional liability rules relating to medical errors to companies that develop chatbots when information provided by their systems causes harm to patients, rather than limiting accountability to traditional healthcare providers.
Yang stressed the need for independent evaluation and oversight systems, arguing that voluntary initiatives by technology companies would not be sufficient given the potential conflict between commercial interests and the requirements of transparency and accountability.
He said companies naturally seek to increase profits, while establishing oversight systems that expose errors and trace harm may conflict with those economic incentives, making independent monitoring essential to protecting users.
The researchers emphasized that the paper does not provide a new measurement of the rates of medical harm caused by chatbots. Instead, it analyzes the risks and regulatory gaps that hinder the detection of such harm and presents recommendations for addressing them.
They concluded that improving the accuracy of AI-generated medical answers, although important, would not be enough to ensure user safety without systems that document the information provided, detect errors, measure their health effects and establish accountability.
The paper argues that the central challenge is no longer simply preventing chatbots from giving incorrect medical advice. It is also ensuring that errors that do occur do not turn into invisible health harms that doctors cannot explain, regulators cannot investigate and researchers cannot quantify.
Resource: Agencies
