RAG in 2026: long context did not kill retrieval, it raised the bar
Models now read a million tokens, and GraphRAG, agentic search and multimodal embeddings have matured. Yet long-context research, cost and latency keep retrieval at the center of any serious knowledge assistant. Here is the pipeline we recommend, what to test in Spanish and French, and what it costs.
Million-token windows are now standard on several frontier models, yet retrieval remains the backbone of serious knowledge assistants. Long-context research (Lost in the Middle, RULER, NoLiMa, Chroma's context rot) shows accuracy falling as inputs grow, and every query over a full corpus pays for all its tokens.
The pipeline that works in 2026: careful parsing, contextual chunks, hybrid search, a reranker, citations and access filters, plus agentic search for multi-step questions. Under about 200,000 tokens of stable content, Anthropic's advice holds: skip retrieval and cache the whole base.
Does a 1M-token window make RAG obsolete?
No, but it moves the threshold. Since Claude 4.6, a 900,000-token request costs the same per token as a 9,000-token one, and GPT-5.4 and DeepSeek V4 also offer a million tokens; from GPT-5.5, OpenAI charges 2x input and 1.5x output above 272,000 tokens. Capacity is no longer the limit. Cost, latency and accuracy are.
Accuracy is the least intuitive part. Lost in the Middle (2023) showed that models use information at the start and end of the context better than in the middle. In RULER (2024), only half of 17 models claiming 32K tokens or more held up at 32K. NoLiMa (2025) removed literal word overlap between question and answer: 11 of 13 models fell below half their short-context score at 32K. Chroma's context rot study (2025, 18 models) saw degradation even on simple tasks.
Newer models do better (OpenAI reports 96.3% for GPT-6 Astra on its 8-needle test between 512K and 1M tokens), but finding needles is not reasoning over a messy corpus, and each query still pays for the whole window.
Anthropic's threshold: under about 200,000 tokens (roughly 500 pages) of rarely changing content, put everything in the prompt with caching. Above that, or with permissions, frequent updates or mandatory citations, build retrieval.
The 2026 pipeline, step by step
- Parse before you embed. Many failures start here: two-column PDFs, flattened tables, scanned pages. Keep headings, table rows and page numbers; send scans to OCR or to page-image retrieval.
- Chunk by structure. Split on sections and paragraphs, keep tables whole and store the metadata you will filter on: source, date, language, owner, access groups.
- Add context to each chunk. Anthropic's Contextual Retrieval prepends 50 to 100 tokens that situate the chunk in its document before embedding and BM25 indexing; with prompt caching, Anthropic put the one-time cost at $1.02 per million document tokens.
- Search hybrid. BM25 catches exact terms (product codes, article numbers, names), vectors catch paraphrase. BEIR showed BM25 is a robust zero-shot baseline while dense retrievers often generalize poorly out of domain.
- Rerank. Retrieve a wide candidate set (we start at 50 to 100), rerank it and pass the best 5 to 20 passages to the model.
- Cite. Return passage identifiers with the answer and check automatically that every quote exists in the retrieved text.
The steps stack up. In Anthropic's tests, top-20 retrieval failures fell from 5.7% to 3.7% with contextual embeddings, to 2.9% adding contextual BM25 and to 1.9% adding a reranker: 67% fewer in total.
Embeddings and rerankers available in 2026
| Model | Type and access | Price | What stands out |
|---|---|---|---|
| text-embedding-3-large (OpenAI) | Embedding, API | $0.13 | Up to 3,072 dimensions; 54.9% on multilingual MIRACL (ada-002: 31.4%) |
| gemini-embedding-2 (Google) | Multimodal embedding, API | $0.20 text, $0.45 images | Text, images, video, audio and PDF in one space |
| voyage-3.5 and 3.5-lite (Voyage AI) | Embedding, API | $0.06 and $0.02 | Voyage 4 adds one space across sizes and an open nano model |
| Qwen3-Embedding-8B (Alibaba) | Embedding, open (Apache 2.0) | Self-hosted | No. 1 on MTEB multilingual on June 5, 2025; 32K context |
| EmbeddingGemma (Google) | Embedding, open | Self-hosted | 308M parameters, under 200 MB quantized, 100+ languages |
| Rerank 4 and 3.5 (Cohere) | Reranker, API | See vendor | Multilingual; Rerank 4 (December 2025) is Cohere's most accurate |
| rerank-2.5 (Voyage AI) | Reranker, API | $0.05 | 32K-token context, follows instructions |
| Qwen3-Reranker 0.6B to 8B (Alibaba) | Reranker, open (Apache 2.0) | Self-hosted | 32K context, 100+ languages |
Voyage reports that general LLMs used as rerankers cost 25 to 60 times more than rerank-2.5 and run 9 to 48 times slower (a vendor claim). Changing embedding model means re-embedding the corpus, so for Spanish and French, shortlist multilingual models and decide on recall@k over your own documents.
Scans, slides and graphs: beyond text chunks
Visual documents. ColPali (2024, ICLR 2025) embeds page images with a vision-language model and matches them by late interaction; on the ViDoRe benchmark it beat OCR-based pipelines while being simpler and faster to index. gemini-embedding-2 and voyage-multimodal-3.5 also bring images and video into the same space as text. Use them for slides, forms, charts and scans.
GraphRAG. Microsoft's GraphRAG (2024) builds an entity graph and community summaries at indexing time, and beat conventional RAG on comprehensiveness and diversity for global questions over million-token corpora, at a high indexing cost. LazyGraphRAG cut indexing to the cost of vector RAG (0.1% of full GraphRAG) and, at 4% of its global query cost, outperformed the methods Microsoft compared.
Our rule: GraphRAG pays off for questions about the whole corpus (recurring themes across 5,000 contracts, links between suppliers), not for lookups.
Agentic search: the model runs the queries
In agentic search the model decides what to look for, reads the results, reformulates and searches again, mixing keyword search, vectors, SQL, APIs and file listings. It suits multi-step questions and structured sources.
Cursor reports that adding semantic search to its agent's grep raised answer accuracy by 12.5% on average (6.5% to 23.5% depending on the model) in codebases of over 1,000 files, with the best results from combining both. Applied to tool catalogs, Anthropic's Tool Search Tool cut token usage by 85% and lifted Opus 4.5 from 79.5% to 88.1% on an MCP evaluation.
The price is cost and latency: every round is another model call over a growing context, and web search adds fees ($10 per 1,000 searches at Anthropic). Cap the rounds, log every query and keep single-shot hybrid retrieval for FAQ-style questions (see AI agents in production).
Evaluate retrieval and answers separately
Measure three layers: retrieval (recall@k: whether the right passage is in the top k), faithfulness (whether every claim is supported by the retrieved passages) and answer quality (correct, complete, cited, in the right language). The Ragas paper (2023) formalized reference-free metrics for these layers; Ragas, DeepEval, promptfoo and the UK AI Security Institute's Inspect are open-source options.
LLM judges scale the work but show position, verbosity and self-preference biases. OpenAI recommends pass/fail or pairwise judgments, controlling for length and trusting a judge only once it agrees with human labels. Method in our LLM evals guide.
Spanish and French: what to test
- Accents and elisions in keyword search: información versus informacion, l'entreprise versus entreprise.
- Regional vocabulary: celular and móvil, computador and ordenador.
- Questions in one language over documents in another, and English terms inside Spanish or French text.
- Local identifiers and number formats: NIT, SIRET, 1.000,50 versus 1,000.50.
Put these cases in your golden set: a good multilingual leaderboard score does not guarantee them.
Permissions and data governance in Colombia and Europe
Filter before you retrieve. Store access groups with every chunk, sync them from the source systems (SharePoint, Drive, the ERP) and apply them as filters inside the vector and keyword queries. Keeping rights and freshness in sync is data platform work as much as AI work.
Treat the index as a copy of the data. Deletions and corrections must reach chunks, embeddings and caches. Retrieved text is also untrusted input: in 2025 Brave showed a browser agent that passed page content to the model without separating it from user instructions (see prompt injection and agents).
Colombia. Law 1581 of 2012 applies to AI processing: prior authorization, explicit authorization for sensitive data, a transmission contract with foreign processors and Article 26 for international transfers. Europe. The GDPR applies, and residency costs extra: Claude's regional endpoints on Bedrock and Google Cloud cost 10% more, as does OpenAI's regional processing for models released since March 5, 2026. More in our 2026 regulatory map.
The model is not an access control. Never ask it to hide what a user may not see: anything that reaches the prompt can reach the answer.
What it costs: a model with explicit assumptions
Assumptions: 100,000 questions a month; a mid-tier model at $2 per million input tokens and $10 per million output tokens (list price of both GPT-6 Sol and Claude Sonnet 5); cached input at 10% of the input price; Voyage rerank-2.5 at $0.05 per million tokens; 500-token passages. Arithmetic on list prices, not a measured project.
| Scenario | Tokens per query | Per query | Per 100,000 queries |
|---|---|---|---|
| Hybrid RAG with reranking | 6,000 in, 500 out, 50 passages reranked | $0.018 | $1,825 |
| Agentic search, 4 rounds | 50,000 in, 1,500 out over 5 calls | $0.12 | $12,000 |
| Whole 200K-token base, cached | 200,000 cached plus 500 in, 500 out | $0.046 | $4,600 |
| Same base, no caching | 200,500 in, 500 out | $0.406 | $40,600 |
| 1M-token prompt, cached | 1,000,000 cached plus 500 in, 500 out | $0.206 | $20,600 |
Three readings. Indexing is cheap: embedding a 100-million-token corpus costs about $13 with text-embedding-3-large, and contextualizing it about $102 at Anthropic's estimate, half that with batch APIs. Even under the 200K threshold, RAG costs about 2.5 times less than a cached full prompt at this volume: stuffing wins on simplicity, not price. And agentic search costs several times classic RAG, so route to it only the questions that need it. More levers in the real cost of AI.
Key takeaways
- A 1M-token window raises the threshold for RAG without removing it: accuracy degrades with length and every query pays for the whole window.
- Under about 200,000 tokens of stable content, cache everything in the prompt; above that, or with permissions and frequent updates, retrieve.
- Contextual chunks, hybrid search and a reranker cut retrieval failures by 67% in Anthropic's tests.
- Keep agentic search and GraphRAG for multi-step and whole-corpus questions, with capped rounds.
- Evaluate retrieval, faithfulness and answers separately, test Spanish and French edge cases, and enforce access rights before retrieval, never in the prompt.
Sources
- Introducing Contextual Retrieval
- Lost in the Middle: How Language Models Use Long Contexts
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- NoLiMa: Long-Context Evaluation Beyond Literal Matching
- Context Rot: How Increasing Input Tokens Impacts LLM Performance
- BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- ColPali: Efficient Document Retrieval with Vision Language Models
- From Local to Global: A Graph RAG Approach to Query-Focused Summarization
- LazyGraphRAG: Setting a new standard for quality and cost
- Semantic search for Cursor's agent
- Ragas: Automated Evaluation of Retrieval Augmented Generation
- Pricing
Editorial note: this analysis reflects the public information available on the review date. Models, prices and rules change fast; every third-party figure links to its source, and our opinions are labeled as such. Spotted an error? Write to contact@slash-digital.io.
The questions we hear often
Is RAG still necessary with 1M-token context windows?
For most companies, yes. Long-context accuracy degrades with length, each query pays for every token, and permissions or daily updates are easier to handle in an index. Below about 200,000 tokens of stable content, a cached prompt is a valid alternative.
Which embedding model is best for Spanish and French?
There is no universal answer, and we could not verify the current MTEB leader. Shortlist multilingual models (Qwen3-Embedding, Gemini, Voyage, OpenAI) and compare recall@k on a few hundred questions over your own documents.
Do I need a reranker?
Almost always. In Anthropic's tests a reranker took retrieval failures from 2.9% to 1.9% on top of contextual hybrid search, and dedicated rerankers cost far less than using a general LLM for the job.
When is GraphRAG worth it?
When users ask about the whole corpus: themes, patterns or relationships across thousands of documents. For fact lookups, hybrid search with reranking is cheaper and usually enough; LazyGraphRAG lowers the indexing cost if you need both.
How do I stop a RAG assistant from leaking confidential documents?
Store access rights with each chunk, sync them from the source systems and filter at query time, before anything reaches the model. Deletions must reach the index and caches, and retrieved text should be treated as untrusted input.
More analysis to read next
Fine-tuning vs RAG: the criterion
When RAG, when fine-tuning and when a hybrid: the decision order before paying for any training.
Agents and engineeringEvals: how to test an LLM system
Why eyeballing fails: a golden set in Spanish and French, calibrated LLM judges, CI regression gates, tooling and what running evals costs.
Agents and engineeringAI agents in production: what works in 2026
Workflow or agent, the eight building blocks, adoption and failure data, computer use, design rules, costs, security and a launch checklist.
Let's put it in production
Tell us your challenge. We reply within 24 business hours with an honest first read: if we can help, we'll say how; if not, we'll say who can.
I reply personally. No endless forms, no canned replies.