When the Advisor Is a Bot. Conversational AI Without Breaking People.

What can’t AI do? We ask it questions about how to spend our money wisely, it tells us about available tax-efficient vehicles. We ask it about relationships, and it offers empathy shaped by pattern recognition. Ask it what to do with your life at 2 a.m., and it will give you an answer… because that’s what it’s built to do.

The emerging concern isn’t the failure of these tools, but their fluency. They’re so reassuringly certain that what is designed to support us can, with enough certainty and repetition, start distorting how we think, feel, and act in ways we never planned for.

Headlines are flashing red. OpenAI recently revealed that in any given week, hundreds of thousands of ChatGPT users might display signs of severe emotional distress, including suicidal ideation. Meanwhile, mental health professionals warn of “  a phenomenon where users develop delusions or dependency through prolonged, emotionally charged chatbot conversations. States in the U.S. are already limiting how bots can be used in therapy as a result.

These stories unsettle us because they challenge the core assumption that AI is just a tool. When the advisor becomes a confidant or feels like a friend, what happens to real human connection?

Developers aren’t just creating fun features anymore; they’re shaping interactions that can influence how people feel and think. That’s why it’s essential to design conversational AI that delivers value without undermining users’ mental well-being.

1.   Narrow the Intent

A recent Harvard study warns that conversational bots tend to agree even when users are wrong, because that kind of reinforcement keeps users engaged. However, it also opens doors to “sycophantic” affirmation. If a chatbot is not meant to be a therapist or intimate friend, you should resist designing it to give that level of emotional affirmation.

The first step is intentionality: To define exactly what your bot is meant to do and what it should avoid. Is it a customer-support assistant, a productivity guide, a career coach, a financial helper, a conversational companion, a recipe creator? Clarity at this stage draws the boundary lines that keep the system from drifting into unwanted territory.

Conversation types, such as open-ended, personal, and non-personal, and modalities like voice or text, influence emotional and problematic use. The study proves that high daily usage is correlated with greater loneliness and AI dependence.

Developers have to ask themselves: How do you keep conversations open enough to be useful, but closed enough to avoid emotional entanglement? For instance, a customer support bot might allow open-ended explanations of the user’s problem but avoid emotionally validating phrases such as, “That sounds really tough, I’m here for you…”.

When the purpose is too broad, the risk of unintended emotional attachment or harmful overreach grows. By narrowing intent, you minimize the chance that people begin to treat the bot as a therapist, or a soul mate.

2.   Verify Knowledge Base

According to a 2025 hallucination report, some LLMs still hallucinate up to nearly 30% of responses. Even top-tier models don’t eliminate the risk entirely. The lowest hallucination rates among tracked AI models were still around 3–5%.

Once you’ve set your purpose, make sure the bot’s knowledge base is grounded in reliable, expert-verified sources. If you’re building something with mental health or emotional support aims, involve clinicians, psychologists, or subject-matter experts in curating the content.

Our medical advisor, Dr. Miguel Villagra, told QuickBlox that, “When we outsource too much of our decision-making and emotional processing to AI, we lose the mental muscle that helps us reality-test and self-correct.” More recently, large models like OpenAI suggest that chatbots introduce intentional “breaks” or small conversational pauses that nudge users back into their own judgment instead of letting the system carry the emotional load.

Still, breaks rely on the bot knowing when to stop and when to redirect. That judgment depends on a solid, vetted knowledge base to anchor it in facts rather than flattery. Gaps or inaccuracies in the database are the easiest and most avoidable gateways to hallucinations, where the AI confidently gives users misleading or dangerous advice.

When the underlying information is tightly curated, regularly updated, and structured around verified sources, the model is far less likely to invent answers or emotionally echo whatever it hears. Instead, it’s forced to pull from grounded material, redirect when something falls outside that domain, and challenge assumptions.

3.   Integrate Safety Checks

Just 48 hours after its AI companions went live, Grok climbed to the number-one app in Japan. Users can talk to these characters through voice, while lifelike avatars mirror expressions and gestures. It’s a level of immersion that’s impressive, but also scarily relatable.

Safety checks are your guardrails. They should include:

• Reality reminders: prompts that remind users they’re talking to an AI, not a human.
• Crisis detection: mechanisms to identify language that signals severe distress, suicidal thoughts, or delusional ideation.
• Escalation protocols: when risk is detected, the bot should gently steer users toward human help, such as professional resources, hotlines, or advise them to reach out to trusted friends.

Without these checks, developers risk enabling echo chambers that reinforce harmful thinking. Experts have explicitly warned that AI’s agreeableness can validate unhealthy belief loops.

4.   Red-Team Dialogues

After testing major bots, a study led by researchers from Stanford University found GPT-4o showed stigma in 38% of responses, and Meta’s Llama 3.1-405b did so 75% of the time. If top-tier models from world-class labs still show measurable stigma, then smaller teams building domain-specific bots are almost guaranteed to have hidden safety failures.

Before launching, run adversarial testing. Engage a red team, it could be internal or external, with the specific job of probing the bot with risky, emotionally fraught conversations. Their sole purpose is to test the bot against the hardest, messiest human scenarios, to prevent genuine potential harm to users once the product is live.

Red teams can ask the bots to role-play edge cases. For customer services, this would be someone in crisis, for companion bots, someone lonely, or someone with distorted beliefs. Evaluate how the bot responds. Does it stay grounded? Does it encourage realism instead of delusion? This phase helps uncover blind spots that your safety checks or knowledge base alone cannot catch.

5.   Initiate Canary Launch

The 2025 International AI Safety Report, published by a panel of 96 global experts, calls out monitoring and intervention as critical for risk mitigation in AI deployments. The report identifies systemic risks, such as loss of control, reliability failures, or bias, that are hard to detect in controlled environments but can emerge only when models interact with real users.

Deploying your bot to a small, controlled group first, also known as a “canary” audience, helps developers monitor how actual users interact. Experts would review the interactions to gauge whether users were becoming emotionally overattached.

It is important to involve relevant advisors, including psychologists, at this stage, as they can understand more deeply what trigger words and phrases might be leading users down a risky path.

Developers should gather both qualitative and quantitative feedback from the control group, like conversation length, sentiment shifts, boundary-testing prompts, repeated emotional disclosures, user-reported comfort levels, and any patterns psychologists flag as signs of over-reliance or distress. This initial rollout is to validate assumptions and refine safety architecture in a tightly scoped launch, rather than in a full-scale release.

6.   Ongoing Monitoring and Iteration

In 2024, experts from nine countries and the European Union met to discuss international cooperation on AI safety science. The summary report stressed the need for scalable, iterative AI governance. The leaders argued for real-world testing frameworks, third-party evaluation, and ongoing assurance beyond pre-deployment checks.

Following the guidance from the report, developers must be vigilant to continuously monitor user interactions and track safety metrics such as crisis triggers or repeated high-risk dialogues. These might include phrases or behaviors that hint at self-harm, hopelessness, suicidal intent, extreme loneliness, or delusional beliefs.

In these instances, developers must update knowledge bases by adding clearer refusal rules and refining crisis-response templates, correcting any factual gaps the bot mishandled. They should also consider incorporating new guidance from psychologists or domain experts to help the system steer conversations safely the next time those triggers appear. If patterns emerge, such as users increasingly relying on the bot for emotional support, you may need to tighten constraints or re-evaluate your design philosophy.

Conversational AI has transformative potential. Used thoughtfully, it can extend access, scale empathy, and reduce friction in coaching or basic counseling-like support. As someone deeply invested in this space, my bet is not on replacing humans but on augmenting them; giving people more tools, not fewer, and doing so responsibly.