AI Fundamentals
How to Build a Chatbot: Architecture, Data, Safety, and Evaluation
Add Unite.AI to your preferred sources on Google
A chatbot is an application that receives a message, determines what the user needs, and returns a response through text or speech. Modern systems may combine rules, retrieval, classifiers, transformers, tools, and large language models rather than relying on one model.
Building a useful chatbot is therefore a product-and-systems problem. The dialogue layer must connect to trustworthy knowledge and business actions, while identity, permissions, logging, evaluation, fallback, and human escalation constrain what the bot is allowed to do.
Key takeaways
- Start with a narrow user task and a measurable success criterion.
- Separate language generation from retrieval, tools, permissions, and business rules.
- Test complete conversations, including ambiguity, interruption, refusal, and recovery.
- Treat prompts and model outputs as untrusted data; monitor production and preserve escalation paths.
Define the job before choosing a model
Write down who the user is, what they are trying to accomplish, which data the system may access, and which actions require confirmation. A frequently asked question bot, an order-status assistant, and an account-management agent have very different risk profiles.
Create a non-AI baseline and an acceptance set of representative conversations. Measure task completion, answer support, latency, abandonment, escalation, and the cost of harmful errors. A fluent demo is not evidence that the workflow works reliably.
Use a layered architecture
A typical pipeline includes a channel adapter, session state, input validation, intent or routing logic, retrieval, a response or policy model, tool adapters, and observability. Retrieval can ground answers in approved documents; tools perform controlled actions through explicit schemas.
Keep deterministic checks outside the language model. Authentication, authorization, inventory limits, refunds, and irreversible actions should be enforced by application code. Prompt engineering can shape behavior, but it is not an access-control system.
Design dialogue, knowledge, and recovery together
Good conversations handle incomplete requests, corrections, multiple intents, and references to earlier turns. Store only the context needed for the task, make retention visible, and distinguish a user statement from a trusted fact returned by an approved system.
When confidence or evidence is insufficient, the bot should ask a focused question, offer a safe alternative, or transfer to a person with a concise summary. Recovery is part of the core experience—not an edge case added after launch.
Evaluate and operate the complete system
Test retrieval quality, tool selection, argument accuracy, policy compliance, prompt-injection resistance, privacy leakage, and end-to-end outcomes. Red-team adversarial inputs and verify that a malicious document cannot silently override system instructions.
Version prompts, indexes, models, policies, and tools. Review sampled conversations with privacy controls, watch drift and failure clusters, and maintain rollback. This operational discipline links chatbot development to AIOps and incident response.
Core chatbot components in more detail
The channel layer normalizes input from web chat, mobile apps, messaging platforms, or speech. A session layer associates messages with an authenticated or anonymous conversation, enforces expiration, and stores only the state needed for the task. Input controls limit size and file types, detect unsafe payloads, and remove markup that downstream systems should not execute.
A router then decides whether the request belongs to a deterministic flow, search, generation, or a human queue. Classical intent classifiers remain useful when the label set is stable; language models are more flexible but harder to calibrate. Hybrid routers can reserve regulated or high-volume tasks for tested workflows and use a general model for open-ended explanation.
The response layer should carry evidence and state separately. A generated sentence can cite a retrieved passage, but the application must preserve which source and version supported it. Conversation memory should distinguish user preferences from verified account data, and should never allow an earlier user message to grant new permissions.
Retrieval, tools, and transactions
Retrieval quality begins before vector search. Documents need ownership, access labels, canonical versions, useful chunks, and removal dates. Query rewriting, keyword search, embeddings, filters, and reranking can be combined. Evaluation should measure whether the necessary evidence was retrieved, whether irrelevant passages were excluded, and whether the answer actually follows the evidence.
Tools convert a model suggestion into a typed request to application code. Each tool needs a narrow purpose, an explicit schema, server-side validation, least-privilege credentials, timeouts, idempotency where possible, and a clear result. The model should not construct raw database queries or arbitrary URLs when a bounded business operation can be exposed instead.
Transactions require confirmation at the point of commitment. Show the user the material fields—recipient, amount, address, date, or access change—and do not treat an old ‘yes’ as approval for a new action. For multi-step work, keep a state machine outside the model so a retry or reordered message cannot skip a required gate.
A practical build and evaluation plan
Begin with twenty to fifty representative tasks and include unsuccessful, ambiguous, and out-of-scope requests. Mark the expected action, evidence, escalation, and prohibited behavior. Implement the simplest viable flow, then add retrieval or generation only where it improves a measured outcome. This produces a reusable regression suite before the interface becomes complicated.
Evaluate components and conversations separately. Retrieval metrics, tool-call accuracy, policy checks, and response support diagnose specific failures; task completion and user effort reveal system-level quality. Use multi-turn tests that correct earlier details, interrupt a flow, switch topics, withhold required information, and trigger dependency failures.
Production rollout should be staged by user group, task, and permission. Monitor unsupported claims, repeated clarification, tool rejection, escalation, latency, and abandonment. Review privacy-safe samples, maintain an emergency disable path for each tool, and use incident findings to update prompts, data, code, and the test set together.
Worked example: a support chatbot from prototype to production
Suppose a retailer wants a chatbot that answers order and return questions. Define supported intents, escalation conditions, approved knowledge, authentication rules, and prohibited actions first. Build a test set from de-identified historical questions, including vague requests, misspellings, multilingual input, angry users, prompt injection, and questions with no answer. A retrieval baseline should return evidence before any generative response is allowed to claim policy or order status.
The runtime can classify intent, retrieve policy passages, request identity verification only when account data is needed, call a narrowly scoped order API, compose an answer, and attach citations. Each tool call needs an explicit schema, authorization check, timeout, retry policy, and idempotency key. The model should never construct raw database queries or decide its own permissions. High-impact actions such as cancellation or refunds require confirmation and, above defined limits, human approval.
Evaluate intent accuracy, answer correctness, evidence support, refusal quality, successful containment, escalation precision, latency, and cost per resolved conversation. Review results by intent and user group rather than one average. In production, log consent-aware traces, tool outcomes, retrieved document versions, and user corrections. Roll out gradually, compare with the existing channel, and disable capabilities when error, abuse, or dependency thresholds are exceeded.
Practical implementation checklist
Turn the concept into a bounded, testable workflow: define task → route → retrieve → generate → use tools → evaluate. Name an accountable owner, document the data and dependencies, establish a simple baseline, set acceptance and stop criteria, test representative failures, and define monitoring, rollback, and review before expanding scope. Record versions and assumptions so another team can reproduce the result and understand what changed.
Before launch, run a documented readiness review with the people who build, operate, secure, and are affected by the system. Test normal cases, boundary conditions, dependency failures, and misuse; preserve the evidence and unresolved risks. Define who can approve release, change a threshold, override an output, or stop operation. Revisit the decision after real-world data arrives, because a technically successful pilot does not guarantee reliable performance at broader scale.
- KNOWLEDGE: approved sources and citations.
- ACTIONS: typed tools with least privilege.
- RECOVERY: clarify, refuse, or escalate.
Frequently asked questions
Does a chatbot need a large language model?
No. Rules, search, forms, and small classifiers can be safer and cheaper for narrow tasks. An LLM is useful when flexible language understanding or generation produces measured value.
What should be tested before launch?
Representative tasks, unsupported requests, ambiguous language, tool failures, privacy boundaries, adversarial prompts, human handoff, latency, and the accuracy of every consequential action.
Primary references
