BibTeX Citation
@misc{moore_open_2026,
  title = {Open Problems in Chatbots and Mental Health},
  author = {Moore, Jared and Anthis, Jacy Reese and Agnew, William and Mock, Andrea and Klyman, Kevin and Koyejo, Sanmi and Haber, Nick and Chancellor, Stevie and Lin, Eric and Louie, Ryan and Ong, Desmond C.},
  year = {2026},
  month = {nov},
  url = {https://arxiv.org/abs/2611.10997},
  note = {Preprint},
}

How can we tell whether chatbots help or harm users’ mental health?

General-purpose chatbots are increasingly used for emotional and mental health support. They can be useful, but issues such as sycophancy, reinforcement of delusional beliefs, and poor crisis response create risks that are difficult for researchers to study and evaluate.

We identify open problems in evaluating and improving the effects of general-purpose chatbots on users’ mental health and psychological well-being. Throughout the paper, we distill 12 recommendations and 12 open questions to provide accessible takeaways for researchers and other stakeholders.

Connections among controlled evaluation, real-world evidence, and policy.
We survey problems in controlled evaluation, real-world evidence, and policy. Addressing these problems may support safer chatbot use and user well-being. Select any figure to view it at full size.

Controlled evaluation

For example, conventional benchmarks that primarily rely on static datasets cannot establish chatbot safety, and novel simulation methods need validation.

Controlled evaluations test chatbot behavior under specified conditions, but better benchmark performance does not necessarily make a system safer or prove that it will improve well-being. Researchers should link evaluation measures to validated measures of real-world outcomes.

The same response may be appropriate or dangerous depending on the preceding conversation, the user’s intent, or their vulnerability. For instance, when exploring the possibility of faster-than-light travel, is a chatbot encouraging a delusion, or is it appropriately role-playing science fiction? Appropriate behavior also depends on clinical judgment, user goals, and social and cultural norms. Clinicians and affected users should help define these criteria.

Test conversations, model internals, and measures validated against real outcomes.
A controlled evaluation has several interacting parts.

Evaluations should cover multi-turn and multi-session interactions with systematic variation in the conversation history.

A simulator that realistically portrays a mental health condition can help test how a chatbot responds to it, but does not show how often the chatbot would fail in real use.

Example open questions:

  • Can simulator-based evaluations predict chatbot behavior in deployment?
  • Which conversational signals in chat logs indicate real-world risk?

Real-world evidence

The full chatbot product, including its interface and memory, affects user interaction and well-being. Evaluations should test deployed chatbots, not only their foundation models. A single conversation can involve system prompts, different underlying models, safety classifiers, memory and personalization, retrieval and tools, interface cues, human escalation, and more. Researchers should measure how interfaces, memory, personalization, and other components affect user well-being, both together and separately.

Hypothetical chatbot components and optional human review and care.
One visible chatbot interaction involves multiple systems. This hypothetical diagram illustrates some of the possible components of chatbot systems. One of our central questions is how these components affect user well-being, separately and together.

Human review practices raise a variety of questions: what kinds of human-in-the-loop interventions are more or less effective, for what downstream measure, using which kinds of human reviewers, and more. We should also seek to understand what kinds of messages are flagged at all.

Example open questions:

  • Which review and escalation practices help, and how often do users receive care?
  • How should general-purpose chatbots engage with users in mental health crises?

Policy

Policy can translate evidence on chatbots’ mental health effects into user protections and standards of care.

Example open questions:

  • What are the impacts of enacted policy mitigations that restrict chatbot behavior?
  • How can policy mitigations protect adults, whose needs differ from those of minors?
  • What standards should we hold general-purpose chatbots to?
  • How can researchers organize and share user chat logs while keeping data private?

Data from real interactions can also help validate simulations and support clinical research on symptom disclosure and risk signals, but their use raises substantial privacy concerns.

Chat log access should also give participants control over secondary uses and protect the privacy of third parties mentioned in logs.

Conclusion

Users and survivors should help shape chatbot evaluation. Researchers should collaborate with users and survivors on research questions, study design, and interpretation of findings, with fair compensation and control over participation and use of their data. Meaningful participation requires influence over research goals and decisions.

Behavioral evaluations, studies of real-world outcomes, and data governance depend on each other and must be addressed in tandem We hope this survey will help establish an evidence base such that future chatbots help and do not harm users’ mental health.

Read the paper on arXiv or download the local PDF.