Chatbots are easy to demo and hard to run in production. The difference is not “better prompts”—it is architecture, evaluation, safety, and operational discipline. From our work delivering AI assistants for Nordic B2B companies, the pattern is consistent: leadership buys outcomes (deflection, conversion, faster resolution), while engineering must ship systems that degrade gracefully when models drift, documents change, or users adversarially probe the UI. This guide translates what matters for CTOs and VPs: technical choices, realistic budgets, and the failure modes that actually show up after launch.
We write from a European delivery perspective: GDPR, vendor subprocessors, and enterprise procurement are not edge cases—they are gating items on the roadmap. If your security questionnaire is an afterthought, your launch will wait for Legal anyway.
Clarify the job: assistant, agent, or workflow bot?
Assistants answer questions and draft content with human oversight—lower risk, faster iteration. Agents call tools (APIs, ticketing systems, CRM) to complete actions—higher value, higher security surface. Workflow bots follow deterministic steps with LLM help for language only—best when compliance demands auditability.
Pick one primary mode. Teams that blend “full autonomy” with “enterprise data access” in v1 usually ship late and incident-prone. A pragmatic first release is often assistant + human approval for any action with financial or legal impact.
Architecture: the boring diagram that saves you
A production chatbot typically needs:
- Gateway for auth, rate limits, session management, and tenant isolation.
- Orchestration service that selects prompts, tools, and retrieval strategies.
- Knowledge layer (if needed): connectors to Confluence, SharePoint, Notion, or custom CMS—plus chunking, metadata, and re-ranking.
- Model routing to use cheaper models for classification and premium models for final answers.
- Observability: structured logs, traces, cost per conversation, and latency percentiles.
For 500–5,000 internal users, cloud-native containers + managed Postgres + object storage is usually enough. For 50,000+ consumer-facing users, expect CDN, edge caching, queue-backed async jobs, and regional deployments—engineering effort scales with concurrency and compliance, not message count alone.
LLM selection: match model to task
Frontier models excel at complex reasoning and long-context synthesis; smaller models excel at classification, intent routing, and high-volume drafts. A common pattern is two-tier routing: a small model decides whether to answer from FAQ, retrieve RAG, or escalate to a human; a large model only runs when needed.
Latency budgets matter: P95 above 6–10 seconds for interactive chat often drives abandonment. Streaming tokens improve perceived speed even when total time is similar—budget frontend engineering for streaming UI, retries, and partial rendering.
Cost scales with tokens and tool calls. For a customer support bot with 20,000 conversations/month averaging 12 turns and 1,200 tokens/turn, inference can land in the low thousands of euros monthly at public API rates before optimization—and 2–4× that if you naively send full documents every turn.
Retrieval-augmented generation: when it is mandatory
If your answers must cite internal policies, product specs, or regulated text, RAG is not optional. Budget EUR 25,000–70,000 for a solid first implementation: ingestion pipelines, PDF parsing, table extraction, language detection, chunking strategy, hybrid search (keyword + vector), and re-rankers.
Also budget content ops: owners who update documents and deprecate stale pages. Without governance, RAG becomes “confidently wrong with citations.”
Tools and actions: the API is the product
When agents call tools, you are building integrations—with idempotency, retries, OAuth refresh, and permission checks that mirror your authorization model. A typical first integration (for example create ticket in Zendesk/Jira) is EUR 15,000–40,000 when done safely—not a weekend script.
Prompt injection is real: users will try to exfiltrate secrets or trick the model into unauthorized actions. Mitigations include least-privilege tool scopes, human confirmation for destructive operations, allowlists for domains, and output filtering—but defense in depth beats any single trick.
Safety, privacy, and EU expectations
GDPR requires clarity on purpose limitation, retention, and subprocessors. If you log prompts for debugging—and you will—implement TTLs, redaction for PII, and role-based access to logs. For Swedish and EU enterprises, EU data residency and DPA terms are common gating items in security reviews.
Accessibility matters for public-facing bots: keyboard navigation, screen reader compatibility, and contrast—often 10–20% additional frontend effort when done properly, far cheaper than retrofitting after a procurement challenge.
Evaluation: what “good” means
Define success metrics before launch:
- Resolution rate or deflection for support bots.
- Task completion rate for internal assistants.
- CSAT and escalation quality (are humans getting cleaner tickets?).
Maintain a golden set of 200–1,000 questions with graded answers. Re-run on every prompt, model, or retrieval change. Add adversarial cases monthly from production failures—this is how you prevent silent regressions.
Budget EUR 8,000–25,000 per quarter for evaluation and human review in active programs—less than the reputational cost of a viral bad answer.
Deployment and SLOs
Ship with feature flags, shadow mode, and canary releases. Define SLOs: availability (99.5–99.9% is typical for business chatbots), P95 latency, and error budgets for upstream model outages.
Have a kill switch that disables tools or falls back to static FAQ when providers degrade. Incidents will happen—runbooks and on-call rotation are part of the product.
Team and timeline: European cost anchors
A credible v1 for an internal enterprise assistant often takes 12–20 weeks with a senior-heavy team. In blended EU rates (EUR 110–150/hour), EUR 120,000–240,000 is a common build range excluding long-term content ops and inference. Customer-facing bots with SSO, brand-grade UX, and multi-language support trend higher.
Add 10–15% for hard QA cycles when your brand is on the line—visual regression, accessibility checks, and load tests on peak traffic scenarios (for example campaign launches). Internals can tolerate rougher edges early; externals cannot.
Operations: what happens after launch
Plan hypercare: 2–4 weeks of elevated support after go-live—daily triage of bad responses, hotfix path for prompt tweaks, and rapid KB updates when users surface gaps. Budget EUR 15,000–40,000 for a structured hypercare slice if you outsource; internal teams should still block capacity—opportunity cost is real.
Longer term, assign an owner for conversation quality and content health. Bots rot when wikis rot—governance is not glamorous, but it is the difference between automation and automated embarrassment.
Common failure modes (and how leaders prevent them)
Scope creep toward “general AI for everything.” Underpowered content governance. Ignoring latency and mobile realities. Treating prompts as the whole system—while integrations silently dominate risk. Skipping evaluation until Twitter notices.
Executive antidote: tie roadmap to one measurable workflow, fund evaluation as Opex, and assign clear ownership for knowledge quality.
If marketing promises “human-like” conversation while engineering ships strict retrieval with citations, users feel the gap. Align externally visible claims with internal architecture—credibility is part of the system.
Finally, plan for seasonality: many B2B bots see Monday morning and month-end spikes. Load testing should reflect real schedules, not average traffic.
Conversation design: scripts, fallbacks, and tone
Great chatbots guide users. Invest in conversation design: sample flows for top intents, disambiguation prompts when confidence is low, and graceful handoff messages to human agents with full transcript context. For B2B, tone should match brand—but precision beats wit in regulated domains.
Plan fallbacks explicitly: when retrieval returns low-confidence chunks, say so and offer next steps (narrow the question, pick a product line, attach a document). Silent guessing is how reputations die.
Analytics and continuous improvement
Instrument funnel metrics: sessions → resolved tasks → escalations → CSAT. Tag conversations with intent labels (even if initially imperfect) so PMs can prioritize roadmap fixes. Feed top failure clusters back into evaluation sets monthly—production is the best dataset you have, provided privacy rules allow redacted storage.
For cost control, track tokens per successful outcome, not per session. A longer conversation that solves the problem may be cheaper than a short one that loops and escalates.
Multilingual and regional considerations
Nordic enterprises often need Swedish and English at minimum; pan-European rollouts add German, French, and Polish faster than teams expect. Machine translation of KB content without human review often poisons RAG—budget localization QA as part of content readiness, not as an afterthought.
Vendor vs build: how to decide
Buy a customer-support platform with AI add-ons if your primary goal is ticketing workflows and vendor innovation pace matches your needs. Build when differentiation is in deep integrations, proprietary workflows, or data residency requirements that platforms cannot meet without expensive enterprise tiers.
Hybrid is common: build orchestration + buy voice or telephony connectors. Price integration honestly—EUR 20,000–60,000 per non-trivial enterprise system is still typical when security and edge cases matter.
Bottom line
A working chatbot is a software service with an LLM inside—not an LLM with a website wrapped around it. Invest in routing, retrieval, permissions, and measurement. The teams that win treat conversation quality as a KPI owned by product and ops, not as a model parameter owned only by engineering.
