Build a Hindi-First Chatbot With OpenAI + Sarvam ASR: A Weekend Tutorial

Most "Hindi chatbots" we test fail the same way: a customer says "mera order kahan hai bhaiya," and the bot — built on an English-first speech engine — transcribes garbage and replies in confused English. Over a weekend in July 2025 we built one that doesn't. Sarvam's ASR handles the Hindi and Hinglish speech, OpenAI handles the reasoning and the reply, and the round trip stayed under 2 seconds on a Jio 4G connection in a Lucknow test. Here's the full build — code, the code-switching gotcha that breaks naive setups, and the numbers.

The 60-Word Answer

Use Sarvam's Speech-to-Text API to transcribe Hindi/Hinglish audio — it's trained on Indian speech, so it beats English-first ASR on code-switched sentences. Pass the transcript to OpenAI's chat.completions with a system prompt that says "reply in the same language the user used." For voice replies, send OpenAI's text back through a TTS engine. The whole pipeline is three HTTP calls. The hard part is code-switching, not the wiring.

Why Not Just Use OpenAI's Whisper for the Speech Part?

Whisper is excellent, but it's trained predominantly on English and major world languages. On real Indian speech — where a sentence switches between Hindi and English mid-clause ("bhaiya mera delivery kab tak aayega") — it tends to either over-translate to English or mistranscribe the Hindi. Sarvam AI, a Bengaluru company, trains specifically on Indian languages and accents. In our side-by-side on 40 recorded support calls, Sarvam's transcripts needed far fewer corrections on Hinglish utterances. That's the entire reason to add a second vendor instead of going all-OpenAI.

The Pipeline (3 Pieces)

What You'll Need

Step 1: Transcribe Hindi/Hinglish Audio With Sarvam

Sarvam's STT endpoint takes an audio file and returns the transcript plus a detected language code. The key parameter is asking it to keep code-switched output faithful rather than forcing a single script.

// transcribe.js
  import fs from "fs";
  const SARVAM_KEY = process.env.SARVAM_API_KEY;
  export async function transcribe(audioPath) {
    const form = new FormData();
    form.append("file", new Blob([fs.readFileSync(audioPath)]), "audio.wav");
    form.append("model", "saarika:v2");        // Sarvam's STT model
    form.append("language_code", "hi-IN");      // hint; auto-detect also works
    const res = await fetch("https://api.sarvam.ai/speech-to-text", {
      method: "POST",
      headers: { "api-subscription-key": SARVAM_KEY },
      body: form
    });
    const data = await res.json();
    return {
      text: data.transcript,
      language: data.language_code || "hi-IN"
    };
  }

Step 2: Reason and Reply in the Caller's Language

The transcript goes to OpenAI. The single most important line in the system prompt is the language-matching instruction. Pass the detected language explicitly — don't make the model guess from the text alone.

// reply.js
  import OpenAI from "openai";
  const openai = new OpenAI();
  export async function generateReply(transcript, language, history = []) {
    const system = You are a support assistant for an Indian D2C brand.
  The caller's language code is "${language}".
  Reply ONLY in that language, in a warm, natural, spoken style.
  If the caller mixes Hindi and English (Hinglish), reply in Hinglish too — match their register.
  Keep replies to 1–2 short sentences; this will be spoken aloud.;
    const res = await openai.chat.completions.create({
      model: "gpt-4o-mini",
      messages: [
        { role: "system", content: system },
        ...history,
        { role: "user", content: transcript }
      ],
      temperature: 0.4
    });
    return res.choices[0].message.content;
  }

The "1–2 short sentences" instruction matters more than it looks. Models love long, written-style answers. For a spoken bot, a 60-word reply feels like a lecture. Cap it hard.

Step 3: Speak the Reply Back (Voice Only)

For a voice bot, the reply text goes through TTS. Sarvam offers Indian-language TTS with natural-sounding voices; you can also use any TTS that supports Hindi.

// speak.js
  const SARVAM_KEY = process.env.SARVAM_API_KEY;
  export async function speak(text, language = "hi-IN") {
    const res = await fetch("https://api.sarvam.ai/text-to-speech", {
      method: "POST",
      headers: {
        "api-subscription-key": SARVAM_KEY,
        "Content-Type": "application/json"
      },
      body: JSON.stringify({
        inputs: [text],
        target_language_code: language,
        speaker: "meera",
        speech_sample_rate: 16000
      })
    });
    const data = await res.json();
    return Buffer.from(data.audios[0], "base64"); // WAV bytes
  }

Step 4: Verify the Code-Switching Actually Works

Common Mistakes

When NOT to Build This

Real Example: A Lucknow Sweet-Shop Voice Line

We built the voice pipeline above, plugged into a Twilio number. Callers spoke naturally — mostly Hinglish — and the bot took the order, confirmed quantity and pickup time, and logged it to a Google Sheet the counter staff watched. The code-switching handling was the difference: an English-first stack we'd prototyped first mistranscribed "do kilo kaju katli" as nonsense; Sarvam got it right. We later extended this exact pattern into a full WhatsApp order bot for a sweet shop and a Twilio + Sarvam voice IVR for a Tally helpdesk.

The same Indian-speech-first thinking runs under TalkDrill, our in-house English-fluency app — where we cut voice round-trip latency to 740ms on Indian 4G using a similar streaming architecture. Our mobile development team reuses these voice patterns across client apps.

Frequently Asked Questions

Is Sarvam ASR better than OpenAI Whisper for Hindi?

For code-switched Hindi-English speech, yes — Sarvam is trained on Indian languages and accents, so Hinglish survives transcription instead of collapsing into broken English. For clean single-language English audio, Whisper is fine and one fewer vendor. Pick Sarvam specifically when callers mix languages.

How many Indian languages does Sarvam support?

Sarvam's speech models cover the major Indian languages — Hindi, Tamil, Telugu, Kannada, Malayalam, Bengali, Gujarati, Marathi, Punjabi, Odia, and English, with more added over time. Always check their current docs, since coverage and model versions change.

What's the latency of a voice chatbot on Indian mobile networks?

In our Jio 4G test in Lucknow the full round trip was about 1.8 seconds — roughly 600ms for speech-to-text, 700ms for the OpenAI reply, and 500ms for text-to-speech. Keep total under 3 seconds or the conversation feels broken to the caller.

Can I build a text-only Hindi chatbot without the voice parts?

Yes, and you should if your users type. Drop the Sarvam STT and TTS calls and send the typed message straight to OpenAI with the "reply in the user's language" prompt. That's cheaper, faster, and simpler than the full voice pipeline.

How do I handle a caller who switches languages mid-conversation?

Auto-detect the language on the first turn, then lock to that code for the rest of the call. Switching detection every turn causes flapping and broken transcripts. If a caller genuinely changes language, a fresh call or a "switch language" command resets it cleanly.

How much does this cost to run per conversation?

A typical 4–6 turn voice conversation costs very little: Sarvam STT/TTS charge per second of audio, OpenAI per token. At gpt-4o-mini rates, the model is the cheapest piece.

As Hrishikesh, our CTO, says: in India, "speaks Hindi" is not a feature — "understands the way people actually mix Hindi and English" is. That's the bar.