A 22-person Bengaluru B2B SaaS company was answering the same 40 support questions every week by hand. We built them a RAG support bot over their 180-page help-doc set in a weekend using Claude and pgvector — no Pinecone, no LangChain, no vector-DB subscription. It answered 71% of incoming questions correctly on the first try, and — this is the part that matters — it said "I don't know, here's a human" on the rest instead of inventing answers. Here's the full build, with the chunking config, retrieval settings, and eval harness you can copy.
What's a RAG support bot, in plain terms?
A RAG (retrieval-augmented generation) bot answers questions using your own documents instead of the model's general knowledge. When a user asks something, you find the most relevant chunks of your help docs, hand those chunks to Claude as context, and ask it to answer using only that context. The result is a support bot that's grounded in your actual docs and can be told to say "I don't know" when the docs don't cover the question — which is what keeps it from making things up.
Why build this now (June 2025)
Two things changed. First, pgvector matured to the point where, for anything under a few million chunks, you don't need a dedicated vector database at all — your existing PostgreSQL handles embeddings storage and similarity search with one extension. Second, Claude's instruction-following got good enough that a well-written prompt reliably refuses to answer outside its context. Together, that collapses what used to be a multi-tool, multi-subscription stack into Postgres plus one API. We've shipped variants of this for several clients; the foundational version is in our weekend RAG chatbot tutorial with Claude and Pinecone, and this post is the leaner, Pinecone-free take.
The architecture (deliberately small)
How to build it: the weekend walkthrough
You'll need a PostgreSQL database (managed is fine), the pgvector extension, an Anthropic API key, and an embeddings provider. Every step has a verification check.
CREATE EXTENSION IF NOT EXISTS vector; then create a table: chunks(id, doc_url, section, content text, embedding vector(1536)). Add an HNSW index: CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);. Verify: \d chunks shows the vector column and the index exists.INSERT the text plus its vector into the chunks table. Batch the embedding calls to stay under rate limits. Verify: SELECT count(*) FROM chunks WHERE embedding IS NOT NULL; equals your chunk count.SELECT content, section, doc_url, 1 - (embedding <=> $1) AS score FROM chunks ORDER BY embedding <=> $1 LIMIT 5;. Drop any chunk scoring below your floor (we started at 0.35 and tuned). Verify: ask a question your docs clearly cover and confirm the top chunk is the right section.If you'd rather we build and tune this against your real support inbox, we ship a working v1 in about a week.
The retrieval settings that actually moved the needle
The default settings gave us 54% on our eval. Three changes took it to 71%.
| Change | Before | After | Eval score |
|---|---|---|---|
| Chunk size | 1,000 tokens, no overlap | 500 tokens, 80 overlap | 54% → 62% |
| Chunks retrieved | Top 3 | Top 5 | 62% → 66% |
| Similarity floor | None (always answered) | 0.35 cutoff | 66% → 71% (and refusals became honest) |
The similarity floor is the one that matters most for trust. Before it, the bot answered everything — including questions the docs didn't cover — by stretching unrelated chunks. After it, the bot refused cleanly on out-of-scope questions. The raw "correct" number only rose 5 points, but the wrong-and-confident answers dropped to near zero. That's the difference between a bot users trust and one they learn to ignore.
When NOT to build a RAG bot
Three more situations where RAG is the wrong tool:
Questions that need live data. "What's my account balance?" or "Where's my order?" aren't documentation questions — they need an API call, not retrieval. RAG answers static knowledge; route account-specific questions to your backend.
Highly regulated answers. If a wrong answer has legal or financial consequences (tax advice, medical guidance, compliance), a 71%-correct bot is a liability, not an asset. Keep a human in the loop, and use the bot only to draft, never to send.
Tiny doc sets. If you have 8 FAQ entries, you don't need RAG — just put all 8 in the prompt directly. RAG earns its complexity when the doc set is too big to fit in context.
A real example: from 40 weekly tickets to 12
The Bengaluru SaaS client was fielding about 40 repeat questions a week — password resets, plan-limit explanations, integration setup steps — all answered in their docs but easier to ask support than to search. After we shipped the bot on their website widget and inside their Slack community, repeat tickets dropped to roughly 12 a week. The bot handled the 71%; the 29% it refused went to a human with the conversation already attached, so the support rep had context instantly. The founder's framing: "It didn't replace support. It deleted the boring half of support."
We applied the same chunking-plus-floor pattern when we built an internal knowledge bot described in our weekend internal-wiki RAG build, and the "I don't know" guardrail came directly from lessons in our insurance-broker RAG bot where we cut wrong-confident answers from 38% to 11%. This is core work for our AI and automation team, and the same retrieval patterns power features in our in-house English-practice app TalkDrill.
A pre-launch checklist for your RAG bot
- Help docs reviewed and current before indexing — RAG amplifies whatever you feed it
- Chunks are self-contained, heading-aware, and carry source URLs
- A similarity floor is set so the bot can refuse out-of-scope questions
- The answer prompt forbids answering outside the retrieved context
- A 30-question eval harness exists, with a baseline score recorded
- Account-specific and live-data questions route to your backend, not RAG
- Refused questions hand off to a human with the conversation attached
Our take
The interesting part of RAG in 2025 isn't the retrieval — pgvector and Claude make that almost boring. It's the guardrail. A support bot that confidently answers 90% of questions but invents the other 10% is worse than no bot, because users can't tell which answers to trust. A bot that correctly answers 71% and honestly refuses 29% is something a team will actually deploy. Build the eval harness, set the similarity floor, and optimise for honest refusals before you optimise for raw coverage. We crosschecked the chunking and floor approach against r/LocalLLaMA RAG threads, where the "small chunks + overlap + retrieval floor" recipe is the repeated community finding.
Frequently asked questions
Do I need a vector database like Pinecone for a RAG bot?
Not for most use cases. The pgvector extension turns your existing PostgreSQL into a vector store with similarity search and indexing. For anything under a few million chunks, that's plenty, and it saves you a separate subscription and service to operate. Reach for a dedicated vector DB only at large scale or with specialised needs.
How do I stop a RAG bot from making up answers?
Two mechanisms. First, a similarity floor in retrieval — if no chunk clears the threshold, the bot gets told there's no relevant context. Second, an explicit prompt instruction to answer only from the provided context and reply "I don't know" otherwise. Together they turn confident hallucinations into honest refusals.
What chunk size works best for help docs?
We landed on roughly 500 tokens with 80 tokens of overlap, split on headings. Larger chunks (1,000+ tokens) diluted relevance; smaller ones lost context. The right size depends on your docs, which is why you tune it against an eval harness rather than guessing once.
How accurate can a RAG support bot realistically be?
For well-maintained help docs, 65–80% first-try correct on real questions is a realistic range, with the rest handled by an honest refusal and human handoff. Chasing 95% usually means the doc set or the question routing needs work, not the model. Measure it with your own eval set.
How long does it take to build a RAG support bot?
A focused v1 over an existing doc set is genuinely a weekend for an experienced engineer: enable pgvector, chunk and embed the docs, build the retriever and answer prompt, and write a 30-question eval. Tuning retrieval and wiring it into your website or Slack adds a few more days.
Can the bot cite where its answer came from?
Yes, and it should. Keep each chunk's source URL and section title, and instruct the model to cite the section it used. Citations let users verify the answer and let you spot when the bot is pulling from the wrong place — which is invaluable while tuning.
Want a RAG Support Bot Trained on Your Actual Docs?
We build grounded support and knowledge bots on Claude + pgvector — with the similarity-floor guardrail and an eval harness so it refuses honestly instead of guessing. Typical v1: about a week, fixed scope. Suitable if your team answers the same 40 questions weekly. First call is technical, with the engineer who'd build it.
Book a 20-min Call
