RAG relevance: the boring 35% that actually moved the needle

Most RAG content online is about the exciting 5%: agents, re-rankers, exotic retrieval tricks. In production, over 10K+ enterprise documents, the answer-relevance gains I actually shipped (35–45%) came from the boring 95%.

Here's what actually moved the needle.

1. Chunking is a product decision, not a config value

The default "split every 1000 characters" destroys meaning. Tables get cut in half. A heading ends up orphaned from the paragraph it introduces.

What worked: chunk on structure first (headings, sections, table boundaries), then fall back to size. And carry a little context in each chunk — the document title and the section path — so a retrieved chunk still knows where it came from.

type Chunk = {
  text: string;
  // context that survives retrieval
  docTitle: string;
  sectionPath: string[]; // ["Onboarding", "Step 2: KYC"]
};

2. Retrieval quality is measurable — so measure it

If you can't say "relevance went up 35%," you're guessing. Build an eval set early, even a small one. A few hundred real questions with known-good answers beats a vibe check every time.

I ran evals as a gate: retrieval changes had to beat the current baseline on the eval set before they shipped. That one habit is responsible for most of the gains, because it turned "this feels better" into "this is +8% or it doesn't merge."

3. Latency is part of relevance

An answer that's 10% better but arrives 3 seconds later often feels worse. Redis caching on embeddings and retrieval results, plus trimming the context you actually send to the model, cut response latency 25–40%. Users read that as "the system got smarter."

4. Fail gracefully or fail everything

LLM and downstream services fail. If one flaky call takes down the whole answer, your relevance number is irrelevant because the user got an error. Fallback, retry, and orchestration around every external call is what kept cascading failures from happening.

The takeaway

The hard part of production AI isn't capability — the models are good. It's making the system trustworthy and consistent inside a real business. That's almost entirely retrieval, evals, latency, and failure handling. Boring. Effective.

I'm building AgentTrace partly because of this: you can't improve what you can't see, and LLM/agent observability is still mostly missing.