Vijay Mohan Dhanwant

+91-7768888995 · vijay.mohan.dhanwant@gmail.com · linkedin.com/in/vijaydhanwant

About

Built a production voice AI pipeline handling live phone calls at sub-2-second latency, handling 5,000+ monthly callers — then migrated the entire system from AWS to Azure in one week when the client's licensing changed three weeks before handoff. 3+ years shipping at the intersection of Generative AI, voice interfaces, and cloud infrastructure. Also delivered an AI-powered restaurant SaaS platform as sole engineer, piloted across 15–20 US restaurants. Now looking to go deeper — building AI-native products and the infrastructure layer underneath them.

Technical Stack

Real-Time Voice AI Pipeline for a Genesys Contact Center

GenAI Engineer · e-Zest (Accion Labs) · Dec 2025–Present e-Zest is an Accion Labs company, a technology services firm. Owned the complete cloud pipeline within a cross-functional team — BA, tester, Genesys dev (team lead), project manager, client manager, solutions architect — as the sole AI/cloud/voice engineer.

• Production voice assistant handling live phone calls, built from scratch — audio in from the caller, transcribed in real time, reasoned over by an LLM, and spoken back.
• End-to-end round trip: 1.0–2.1 seconds.
• MVP phase at ~5,000 calls/month, targeted for ~2 million calls/year at full rollout.
• Running on ECS Fargate with load balancing and auto-scaling — context preservation across concurrent calls designed and validated during MVP rather than deferred.

flowchart TD
    A[Caller] --> B[Genesys]
    B --> C[Audio Connector]
    C --> D["Decrypt & Stream PCM Audio"]
    D --> E["Azure STT (Streaming)"]
    E --> F{"Needs LLM?"}
    F -- Yes --> G["Prompt LLM\n(streaming out)"]
    F -- No --> H["Workflow Logic\n(deterministic)"]
    G --> I["TTS\n(parallel streaming)"]
    H --> J[Cached TTS]
    I --> K[Stream Audio Back to Caller]
    J --> K
    K --> L[Conversation Logs + Outcomes]
    L --> M["Eval Sets ← Human Review"]
    M --> N[Prompt & Flow Refinement]
    N --> G
      

How it started — and why the architecture changed.

• The initial design was Lambda-based, routing through Genesys's web flow and workflow designer — latency from that approach wasn't meeting client expectations.
• Rather than waiting for direction, I worked with the solutions architect to prototype a streaming alternative using Genesys's audio connector.
• Built a browser-based PCM16 stream as a working demo for the team and stakeholders before presenting the approach to the client.
• The architecture was collaboratively shifted to streaming based on that prototype.
• The finished pipeline supported two separate audio paths simultaneously — PCMU over Genesys's AudioHook protocol for live calls, and PCM16 over a browser WebSocket for development — sharing the same underlying call-handling logic rather than maintaining two systems.

Constraining the LLM for a live phone line.

• Putting an LLM on a live phone call means operating under constraints that don't exist in text-based applications — there's a hard latency budget where every second of silence feels like a system failure, no opportunity to retry or revise a response once speech synthesis has started, and a caller who will simply hang up if the experience feels broken.
• Within those constraints, LLM responses were scoped to approved intents and topics via system prompts — the model couldn't go off-script or hallucinate actions outside the defined conversation flow.
• PII was excluded from model context and stripped from logs, keeping caller data out of inference and audit trails.

Keeping it fast without burning inference cost.

• Not every turn of the conversation needed the LLM — caller authentication, appointment booking, satisfaction surveys, and call transfers were routed through deterministic logic that skipped the model entirely, cutting inference cost on the majority of turns without touching response quality.
• A separate watchdog layer — 800ms end-of-speech detection paired with a 5-second silence timeout and short filler responses — kept the conversation from stalling during slower operations like a lookup or a slot resolution.

Choosing the production model with data.

• Nova Pro and Nova Lite were benchmarked head-to-head across 80 real turns, measuring both average and p95 latency.
• Lite came out 27% faster on average and 66% faster at the tail (1.7s vs. 5.0s p95) — the number that actually decided the production model, since tail latency is what a caller notices.
• The system was validated across 50–200 real test sessions before handoff.

Shipping iteratively, not just once.

• The BA and tester evaluated the bot through continuous real conversations throughout development — what shipped wasn't driven by latency metrics alone but by conversational quality and use-case fit across real caller scenarios.
• Each version was tracked through git branching and Docker image versioning.
• Azure Log Analytics fed back into every iteration, surfacing latency regressions and conversation failures as they happened rather than after the fact.

Making the system better over time.

• BA and tester conversations generated eval data naturally — every real interaction was a potential test case.
• Conversation logs surfaced failure patterns: misunderstood intents, wrong slot fills, unexpected caller phrasing that broke the flow.
• Prompt refinements were driven by real call outcomes, not guesswork — each change addressed a specific observed failure.
• Eval sets were built from production conversations to regression-test prompt changes, ensuring fixes didn't break previously working scenarios.
• This closed the loop between what callers actually said and how the system responded — the feedback cycle shown in the architecture diagram above.

Hiding a 12-second upstream bottleneck.

• The upstream API providing caller data took up to 12 seconds to respond, which would have stalled the opening of every call.
• The fix was to pre-fetch the client data on call connect and mask the lookup behind the intro and authentication flow, so the caller never experienced the delay.
• The same filler-based pattern was applied across other slow operations throughout the pipeline.

Migrating the entire pipeline in one week — with AI-accelerated development.

• Three weeks before handoff, a late client-side licensing change forced the pipeline off AWS and onto Azure.
• The STT, LLM, and TTS layers were re-implemented against Azure Container Apps, Cosmos DB, Azure Speech SDK, and Azure OpenAI — shipped in one week, solo.
• AI-assisted development across the codebase made this velocity possible — using AI tooling for rapid API migration, configuration generation, and debugging unfamiliar Azure services.
• This wasn't a sandbox migration — it meant working inside the client's live production Azure account, provisioning and securing every service in coordination with their infrastructure team, matching their existing IAM and networking policies rather than standing up a clean environment.
• During the migration, Azure Container Apps' native WebSocket endpoint rejected the Genesys AudioHook connection despite valid DNS and SSL — the fix was an Azure API Management gateway fronting the container, secured with API-key auth and Key Vault-managed secrets.
• End-to-end latency held at the same 1.0–2.1s range post-migration with no measurable regression.

Other work under the same engagement.

• ASP.NET MVC to .NET + React migration pipeline architected.
• Multi-provider LLM gateway (LiteLLM proxy across AWS Bedrock, Azure OpenAI, and hosted models) built and used internally.
• RAG evaluation playground for benchmarking embedding strategies and LLM configurations built and used internally.

AI-Powered Restaurant Management Platform

Data Scientist → Full-Stack AI Engineer · Nice Software & Solutions · Oct 2024–Nov 2025 · Nagpur · dineai.ai

• Full-stack SaaS product for restaurant operations, built and shipped alone from an empty repository to a live pilot with 2,000+ active users across 18 restaurants.
• Worked directly with the founder as his implementation partner, introducing AI-driven development practices into the entire product lifecycle to accelerate delivery.
• Platform remains live post-handoff; the founder continues to operate and iterate on it independently.

flowchart TD
    CA["Customer App\n(iOS/Android)"] --> BE
    WA["Waiter App\n(iOS/Android)"] --> BE
    OP[Owner Management Portal] --> BE

    subgraph BE["FastAPI Backend (Multi-tenant)"]
        subgraph REC[Recommendation Engine]
            Q[Query] --> ID["Intent Detection\n(guardrails)"]
            ID --> RH[Request Handlers]
            RH --> DN[Date Night]
            RH --> AF[Allergen Filter]
            RH --> GP[Group Party]
            DN --> DBF["DB Flags + LLM\n(cached prompts + Redis)"]
            AF --> DBF
            GP --> DBF
        end
        subgraph OM[Order Management]
            OMD["Tables, preferences,\nstatus, invoicing"]
        end
        subgraph AN[Analytics]
            CL[Claude AI] --> ANR["Dish performance,\nregional taste trends, revenue"]
        end
        subgraph MR[Multi-Restaurant]
            MRD["Chain → Branches → Menus"]
        end
        IMG["Image Generation · Payment Gateway"]
        AUTH["Auth: JWT · Roles · Subscription-tier access"]
    end

    BE --> PG["PostgreSQL\n(unified DB)"]
    BE --> RD["Redis (Menu Cache)"]
    PG --> DO["DigitalOcean (Nginx, SSL)"]
    RD --> DO
      

flowchart TD
    PDF[PDF Menu] --> EXT[Text-Level Extraction]
    EXT --> FIL[Menu-Relevant Filter]
    FIL --> CT{"Content Type?"}
    CT -- Image --> OCR["OpenCV + Tesseract\n(OCR pipeline)"]
    CT -- Text --> LLM["LLM Extraction\n(pattern-free parsing)"]
    OCR --> ENR["LLM Enrichment\n(spicy, allergen, dietary,\ncuisine type, flavor profile)"]
    LLM --> ENR
    ENR --> FC{"Flags complete?"}
    FC -- Yes --> OUT["Structured Menu + Flags\n→ DB + Redis"]
    FC -- No --> OWN["Owner Portal\n(manual flag entry)"]
    OWN --> OUT
      

AI layer across the product.

• Recommendation engine generating personalized dish suggestions inside the customer-facing iOS and Android apps. Recommendations are driven by structured DB flags (spicy, allergen, dietary, cuisine type, flavor profile) enriched by LLM during onboarding — not raw text similarity.
• Claude-based analytics layer surfacing dish-performance insights back to restaurant management.
• Image-generation pipeline producing visuals for menu items that didn't have photography. Owners can also upload existing photos through the management portal.

Menu digitization and data enrichment.

• Restaurant onboarding required digitizing existing paper and PDF menus with no consistent format across customers.
• Computer-vision pipeline (OpenCV + Tesseract OCR) parsed 50+ real menus across 10+ distinct layout styles without manual re-entry per restaurant.
• After extraction, an LLM enrichment step generates structured flags for each dish — spicy, allergen, dietary restrictions, cuisine type, flavor profile. These flags drive the recommendation engine rather than raw text similarity.
• When menu descriptions are too sparse for reliable flag inference, owners fill the gaps through the management portal.

Multi-restaurant and menu management.

• Multi-restaurant chain support: separate dashboards per restaurant in a chain, gated by subscription tier.
• Flexible menu structure: one branch can have multiple menus (seasonal, special occasion), and multiple branches can share a single menu.
• Owner management portal for adding, editing, removing, and hiding dishes and their details, including uploading or generating dish images.

Backend and infrastructure.

• Multi-tenant FastAPI service with JWT-based auth, subscription-tier access control, and role separation across customer, waiter, and management — each with different insight levels and feature access based on the restaurant's plan.
• Owner-controlled access grants per user. Ordering, cart, and payment gateway integrations built end to end.
• Deployed on DigitalOcean behind Nginx with SSL.

Shipping and feedback.

• Dev, staging, and production environments with CI/CD per branch.
• Staging environment went to peers and a close circle of real users whose feedback shaped what shipped to production.
• Separate admin users tracked every server-side change by author.
• DigitalOcean log monitoring during client demos and live usage surfaced what was working and what wasn't — the kind of feedback loop that only exists when the product is actually in someone's hands.

Conversational IVR Assistant Over a Real Phone Line

Independent Build · Aug 2026 · GitHub

flowchart TD
    PC[Phone Call] --> VP["Vapi (Telephony)"]
    VP --> LG["LangGraph (Stateful Flow)"]
    LG --> VER[Verification]
    LG --> ACT["Account Actions\n(password reset,\ncontact update,\nticket status)"]
    VER --> MCP["MCP Server\n(CRM Access)"]
    ACT --> MCP
      

• Phone-accessible voice assistant that authenticates a caller, looks up their account, and completes account actions — password reset, contact update, support ticket status — entirely by voice, backed by a live CRM.
• LangGraph structured the conversation as an explicit, stateful flow rather than a single open-ended prompt — caller verification had to complete before any account-specific action was accessible, and sensitive actions required a spoken confirmation step.
• CRM operations went through an MCP server rather than a bespoke integration layer — tool access to contact records and support tickets described declaratively instead of hand-wired.
• Vapi handled telephony, covering the harder edges of a real phone conversation: barge-in when a caller interrupts mid-response, silence timeouts, and re-prompting when input was unclear or misheard.

Key Highlights from Earlier Freelance Work

All clients are available to provide referrals and recommendations on request.