How I Built a Bilingual RAG-Powered AI Calling & Chat Agent (With a Full Admin Ops Center)

Most "AI chatbot" projects stop at a demo: a widget, an OpenAI/Gemini call, a canned prompt. Getting one into production for real businesses — across voice calls, web chat, and WhatsApp, in two languages, without hallucinating — is a different problem. Here's how I approached it. The Core Challenge Three requirements shaped the whole architecture: Multi-channel, single brain — phone calls, web chat, WhatsApp, and Instagram all needed to hit the same knowledge base and produce consistent answers. No hallucination tolerance — a wrong answer on a live sales call is worse than no answer. Non-technical operators — the business owner using the admin panel should never need to touch a config file, an API key in code, or a terminal. The RAG Pipeline The knowledge layer runs on a fairly standard but carefully tuned RAG stack: Ingestion: a website crawler pulls headers, paragraphs, and lists from a given URL (or a pasted/uploaded document), and normalizes it into clean text chunks. Embedding: chunks are embedded using gemini-embedding-001. Storage: vectors are indexed into ChromaDB for fast cosine-similarity retrieval. Retrieval-time logic: on every incoming query, the system computes similarity scores against the knowledge base. If the top match clears a confidence threshold, it answers from the KB (source: knowledge_base). If not, it falls back to a live, relevant web search (source: search_fallback) rather than letting the LLM freestyle an answer from parametric memory. That fallback tagging turned out to be one of the most useful design decisions — every logged interaction carries a source field (knowledge_base or search_fallback), which means you get a real-time KB Hit Rate metric for free: how often the system is confidently answering from your own data vs. reaching outside it. Observability Was Not an Afterthought A lot of RAG demos skip this, and it's the first thing that breaks trust once you hand a system to a real business. I built a diagnostic layer that surfaces, per query: Cosine distance / similarity scores against retrieved chunks Which knowledge base sections were actually matched End-to-end latency (ms) Source attribution (KB vs. fallback) This is exposed directly in a "Diagnostic Playground" in the admin UI — type a query, and see exactly what the retrieval layer matched and why, before it ever reaches a real customer. Aggregate metrics (Total Conversations, Average Latency, KB Hit Rate, Search Fallback %, Error Rate) roll up from the same interaction logs into an overview dashboard — no separate analytics pipeline needed. Lead Extraction Without a Structured Form Because conversations happen in free text (and voice-to-text), lead capture couldn't rely on form fields. The system parses conversational turns for identifiers — name, phone number, email — as they're mentioned naturally ("my name is X and my number is Y"), writes them to a leads table, deduplicates against existing entries, and assigns the lead to a sales rep via round-robin rotation. This runs as a lightweight side-effect of the main conversation loop, not a separate workflow the user has to trigger. Multi-Tenant Branding Without Multi-Tenant Infra Complexity Rather than spinning up separate deployments per client, business identity (company name, agent persona, tone, contact details, tagline) is stored as a set of "Business Variables" that get interpolated into the system prompt and voice/chat responses at runtime. One codebase, many brands — which matters a lot if you're an agency or planning to white-label this. No-Code Data Layer The default store is local (SQLite) for simplicity, but the admin UI also supports connecting an external Postgres-compatible database (Supabase, Neon, or vanilla Postgres) via a connection string pasted directly into the UI — no backend redeploy required. This was a deliberate trade-off: less "clever" than an ORM migration system, but it means a non-engineer can point the whole system at their own cloud database in under a minute. Bilingual by Default Since the target users span Hindi and English speakers, language handling isn't a toggle — the model detects and responds in whichever language the user used, per message, in both voice (STT/TTS) and text channels. What I'd Do Differently Next Real-time in-call escalation to a human when confidence is low (currently: unresolved queries become a lead for post-call follow-up — live handoff is the next milestone) Per-channel confidence thresholds (a WhatsApp typo tolerance vs. a live voice transcript need different tuning) Try It If you're building something similar or want to see this running on a real business's data, I opened up a free 10-day trial — happy to walk through the architecture in more depth too. 📧 nexopersupport@gmail.com · 🌐 nexoper.in Would genuinely love feedback from anyone who's tuned RAG confidence thresholds for production voice use cases — what's worked for you?