Posted on Jun 25, 2026 · Updated Jun 25, 2026 · 10 min read
What a Production RAG System Costs Per Month (2026)
Most teams budget for a RAG system by pricing the vector database — and then get blindsided. A production retrieval-augmented-generation system in 2026 runs roughly $150–$400/month small, $600–$1,500 mid-size, and $5,000–$15,000 at a million queries a month. The twist is where the money goes: the vector database is usually the smallest line, embeddings are nearly free, and LLM inference is 60–75% of the bill. Get the model and context budget right and everything else is rounding error.
This guide prices a real RAG stack component by component, then walks three concrete scenarios — an internal tool, a SaaS feature, and a high-volume product — so you can place your own system on the curve. It pairs with our vector database pricing comparison and LLM inference cost per 1M tokens, which expand the two biggest lines.
TL;DR — production RAG cost (2026)
- Small (internal tool, ~10K queries/mo): $150–$400/mo
- Mid-size (SaaS feature, ~150K queries/mo): $600–$1,500/mo
- High-volume (1M queries/mo, premium model): $5,000–$15,000/mo
- LLM inference is 60–75% of the bill; vector DB ~15%, embeddings <5%, infra ~10%
- Embeddings are cheap: indexing a 50M-token corpus with text-embedding-3-small costs ~$1
- Biggest levers: a smaller generation model (15x), semantic caching (40% hit = 40% inference saved), and trimming retrieved context
Table of contents
What you actually pay for
A RAG system has five recurring cost lines. Here is what each runs at 2026 list prices for a typical mid-size deployment:
| Component | What it is | Monthly cost |
|---|---|---|
| LLM inference | Generating answers from retrieved context | $200–$10,000+ |
| Vector database | Storing and querying embeddings | $25–$500 |
| Embeddings | Turning documents + queries into vectors | $5–$50 |
| Orchestration / compute | App servers, API layer, re-ranker | $50–$200 |
| Document processing | Parsing, chunking, metadata, re-indexing | $20–$100 |
Notice the spread. Four of the five lines are bounded — even a large system rarely pays more than a few hundred dollars for the vector DB, embeddings, and orchestration combined. It's the inference line that ranges across two orders of magnitude, entirely driven by query volume, model choice, and how much context you stuff into each prompt.
Where the money goes
For a representative mid-size RAG system — a SaaS support assistant doing ~150K queries a month on gpt-4o-mini with a re-ranker — the bill splits roughly like this:
This is the headline finding: optimizing the vector database to save money is mostly a distraction. Cutting your $135/mo Weaviate bill to $65 with Qdrant saves $70. Switching the generation model from gpt-4o to gpt-4o-mini, or caching 40% of queries, saves hundreds to thousands. Spend your optimization time on the inference line.
Three real scenarios
Same architecture, three different scales. Models priced at 2026 list rates (gpt-4o-mini at $0.15/$0.60 per 1M input/output tokens; gpt-4o at $2.50/$10).
| Scenario | Volume / model | Inference | Total / mo |
|---|---|---|---|
| Internal tool | 10K queries · gpt-4o-mini · 1M vectors | ~$60 | $150–$400 |
| SaaS feature | 150K queries · gpt-4o-mini · 10M vectors | ~$500 | $600–$1,500 |
| High-volume product | 1M queries · gpt-4o · 50M vectors | ~$9,000 | $5,000–$15,000 |
The same 1M-query workload on gpt-4o-mini instead of gpt-4o costs roughly $600 instead of $9,000 — a 15x swing from one decision. That is why "which model generates the answer" dominates every other choice in the stack.
The cost formula
You can estimate any RAG system's monthly inference cost — the line that matters — with one formula:
monthly inference ≈ queries × (context_tokens × input_price + answer_tokens × output_price)
Worked example for the SaaS feature: 150,000 queries, each sending ~2,000 tokens of retrieved context in and getting ~400 tokens out, on gpt-4o-mini ($0.15/1M in, $0.60/1M out):
- Per query: (2,000 × $0.00000015) + (400 × $0.0000006) = $0.0003 + $0.00024 = ~$0.00054
- × 150,000 queries = ~$81/month on gpt-4o-mini
- The same on gpt-4o: per query ~$0.009 → ~$1,350/month
The two numbers you can move are context_tokens (how much you retrieve and stuff in the prompt) and the price (which model). Both are why context trimming and model selection are the highest-leverage cost work in RAG — far more than the database.
Six levers that cut the bill
Roughly in order of payoff:
- Right-size the generation model. Use a small model (gpt-4o-mini, Claude Haiku, Llama 3) for most queries and escalate to a flagship only when needed. This is the single biggest lever — up to 15x.
- Add semantic caching. Many RAG queries repeat. A 40% cache hit rate eliminates ~40% of inference cost — on a $20K/mo inference bill, that's $8K saved.
- Trim retrieved context. Sending the top 3 chunks instead of the top 10 cuts input tokens dramatically. A re-ranker lets you retrieve broadly but send only the best few.
- Batch embeddings. The OpenAI Batch API is 50% cheaper; use it for initial indexing and bulk re-indexing where latency doesn't matter.
- Quantize the vector index. Cuts the vector DB's RAM (and bill) 4–32x — see the vector database pricing guide.
- Consider self-hosting at scale. Above a few billion tokens a month, a dedicated GPU can beat per-token API pricing — we work the break-even in self-hosting Llama vs the OpenAI API.
For the wider picture on AI infrastructure spend and where teams waste it, see the State of AI Infrastructure Costs 2026 and our guide to cloud cost management for AI/ML startups.
Stop your RAG bill from surprising you
spendark tracks your AI line items — inference APIs, vector database, and GPU compute — across AWS, Azure, and GCP in one view, so you see the inference creep before it shows up on the invoice.
Frequently asked questions
How much does a RAG system cost per month?
In 2026, roughly $150–$400/month for a small internal tool (~10K queries), $600–$1,500 for a mid-size SaaS feature (~150K queries), and $5,000–$15,000 for a high-volume product at 1M queries on a premium model. LLM inference is 60–75% of that; the vector database, embeddings, and infrastructure are the small lines.
What is the biggest cost in a RAG system?
LLM inference — generating the answer from retrieved context. It's typically 60–75% of the monthly bill and is the only line that ranges across two orders of magnitude, driven by query volume, model choice, and how much context each prompt carries. The vector database, by contrast, is usually under 15%.
How much do embeddings cost for RAG?
Very little. With OpenAI's text-embedding-3-small at $0.02 per 1M tokens, embedding a 50M-token corpus costs about $1, and ongoing query embeddings are pennies. Embeddings are usually under 5% of a RAG bill, which is why optimizing them rarely moves the total.
How can I reduce RAG inference costs?
In order of impact: use a smaller generation model for most queries (up to 15x cheaper), add semantic caching (a 40% hit rate cuts ~40% of inference), and trim retrieved context so each prompt carries fewer input tokens. A re-ranker lets you retrieve broadly but send only the top few chunks to the model.
Is it cheaper to self-host the LLM for RAG?
Only at high volume. A dedicated GPU running an open model beats per-token API pricing roughly above 50% GPU utilization — on the order of a couple billion tokens a month per H100. Below that, a hosted API is cheaper and far less operational work. See our break-even analysis for the full math.
Estimate your cloud costs — for free
Compare AWS, Azure, and GCP pricing side by side with our free calculator, and dig into the guides to learn how to cut cloud waste. No sign-up required.