How to build a RAG system that answers from your own documents, cites the page it got each claim from, says "I don't know" when the documents don't say, and proves with numbers that it beats the naive version. Ruby for the plumbing, PostgreSQL for the index, every model behind an adapter.
The Ruby examples on this page that carry tests (the chunker and the metrics) were run with Ruby 3.3 and pass. The SQL and model calls are written against the documented APIs but have not been run against our own corpus yet; treat them as a starting point to check, not a finished system. Numbers below are either worked arithmetic or targets, never measurements we haven't made.
Naive RAG is three steps: embed the question, take the five nearest chunks, paste them into a prompt. It works in a demo and fails quietly in use: it misses exact terms like part numbers, ranks a vaguely related chunk above the right one, answers confidently when nothing relevant came back, and can't say where an answer came from. Each section below fixes one of those failures. The full pipeline is:
ingest: PDF → pages → chunks (500 tokens, 50 overlap) → embeddings + full-text index → PostgreSQL
query: question → rewrite → expand → hybrid search (dense + sparse, tenant-scoped)
→ rerank top 50 to top 10 → parent sections → answer with page citations
→ grounding check → abstain, or fall back to web search, when confidence is lowEvery model in that pipeline is a swappable part (Own your stack): one small Ruby interface, one class per provider, chosen by an environment variable.
| Part | Interface | Self-hosted first | Hosted alternative |
|---|---|---|---|
| Embedder | embed(texts) → [[Float]] | BAAI/bge-m3 or nomic-embed-text, served by Hugging Face Text Embeddings Inference (TEI) or Ollama, or in-process with the informers gem | Voyage, OpenAI, Cohere |
| Vector store | SQL | PostgreSQL with pgvector; ParadeDB pg_search for BM25 | Any Postgres host with pgvector |
| Reranker | rerank(query, texts) → [Float] | bge-reranker-v2-m3 on TEI, or informers | Cohere Rerank, Voyage rerank |
| Generator | answer(question, chunks) → Answer | A local model through Ollama | Claude (our default), Codex, Kimi |
| Web search | search(query) → [Result] | SearXNG | Claude's web search tool, Brave Search API |
An adapter is a plain class. Here is the embedder over TEI, which serves POST /embed; the
others look the same.
# frozen_string_literal: true
require "json"
require "net/http"
module Embedders
# Text Embeddings Inference on our own machine. EMBEDDINGS_URL comes from .env.
class Tei
def initialize(url: ENV.fetch("EMBEDDINGS_URL"), model: ENV.fetch("EMBEDDINGS_MODEL"))
@uri, @model = URI.join(url, "/embed"), model
end
attr_reader :model
def embed(texts)
response = Net::HTTP.post(@uri, { inputs: texts, normalize: true }.to_json,
"Content-Type" => "application/json")
response.value # raises on non-2xx
JSON.parse(response.body)
end
end
endA citation is only as good as the page number stored with the chunk, so keep pages from the first step.
Poppler's pdftotext puts a form feed (\f) between pages, which gives a page array
for free:
require "open3"
def pdf_pages(path)
text, status = Open3.capture2("pdftotext", "-layout", path, "-")
raise "pdftotext failed for #{path}" unless status.success?
text.split("\f") # pages[0] is page 1
endocrmypdf first; it adds one and keeps pages intact.file.pdf#page=14) and the printed label for
citing ("p. 12"). pdfinfo and most PDF libraries expose page labels.Our default is windows of 500 tokens that overlap by 50. The overlap means a sentence cut at a boundary still appears whole in one of the two neighbouring chunks. The arithmetic, with size S = 500, overlap O = 50 and stride S − O = 450:
tokenizers gem loads any Hugging Face tokenizer.The chunker records which pages each window spans, so a chunk that crosses a page break cites "pp. 12–13":
# frozen_string_literal: true
# Splits a document's pages into overlapping token windows and remembers
# which pages each window came from, so answers can cite "p. 12-13".
class Chunker
Chunk = Data.define(:ordinal, :text, :token_count, :page_start, :page_end)
def initialize(tokenizer:, size: 500, overlap: 50)
raise ArgumentError, "overlap must be smaller than size" unless overlap < size
@tokenizer, @size, @overlap = tokenizer, size, overlap
end
# pages: page texts in order, page 1 first.
def call(pages)
tokens = pages.each_with_index.flat_map do |text, i|
@tokenizer.encode(text).map { |id| [id, i + 1] }
end
windows(tokens.size).each_with_index.map do |range, ordinal|
ids, page_numbers = tokens[range].transpose
Chunk.new(ordinal:, text: @tokenizer.decode(ids), token_count: ids.size,
page_start: page_numbers.first, page_end: page_numbers.last)
end
end
private
def windows(total)
return [] if total.zero?
starts = 0.step(by: @size - @overlap).take_while { |s| s.zero? || s + @overlap < total }
starts.map { |s| s...[s + @size, total].min }
end
endWith the tokenizers gem, wrap Tokenizers.from_pretrained("BAAI/bge-m3") in a
small object whose encode returns .ids and whose decode calls the
tokenizer's own. The tests use a one-word-one-token tokenizer so the expected values can be worked by hand:
def test_windows_overlap_by_the_configured_amount_and_track_pages
pages = [(1..6).map { "a#{_1}" }.join(" "), (7..10).map { "a#{_1}" }.join(" ")]
chunks = chunker(size: 5, overlap: 2).call(pages)
# stride 3: tokens 0-4, 3-7, 6-9
assert_equal ["a1 a2 a3 a4 a5", "a4 a5 a6 a7 a8", "a7 a8 a9 a10"], chunks.map(&:text)
assert_equal [[1, 1], [1, 2], [2, 2]], chunks.map { [_1.page_start, _1.page_end] }
end
def test_chunk_count_for_a_100k_token_document
words = (1..100_000).map { "w#{_1}" }.join(" ")
# ceil((100_000 - 50) / 450) = 223
assert_equal 223, chunker(size: 500, overlap: 50).call([words]).size
endFixed windows are the baseline, not the ceiling. Two refinements usually help and are worth measuring (section 20): snap window edges to the nearest paragraph or heading break within ±50 tokens, and never merge two documents or two top-level sections into one chunk. Keep the 500/50 sizes as the minimum the rest of the pipeline assumes; go smaller only with eval numbers that say so.
Chunking is the decision that silently caps every metric downstream, so write it down next to the code as
CHUNKING.md and update it whenever a number changes. A useful template:
# Chunking
Tokenizer: BAAI/bge-m3 (same as the embedder; counts must match its input limit)
Window: 500 tokens, 50 overlap (stride 450, ~11.5% overhead)
Boundaries: snap to paragraph/heading within ±50 tokens; never cross a document
or a top-level section
Metadata: document_id, sha256, page_start, page_end, page_label, section path,
organization_id, rooftop_id, kind, published_on
Parents: the enclosing section, capped at 2,000 tokens (see parent retrieval)
Why: Answers in our documents are usually one to three paragraphs long.
500 tokens holds a full answer plus its heading; 50 overlap keeps
a sentence cut at a boundary whole in one neighbour.
Evidence: sweep on the golden set (date, commit):
size 256 / 500 / 1000 → hit@10, MRR, NDCG@10 for each
Revisit when: the document mix changes, the embedder changes, or hit@10 drops.The "Evidence" line is what turns a habit into a decision. A sweep is cheap: re-chunk into a scratch table, re-embed (the cache in section 5 makes repeat runs fast), and run the metrics in section 17.
Embed every chunk once at ingest and the question once per query. Both are worth caching, keyed by what the vector depends on, so entries expire themselves when anything changes (Caching in Rails):
require "digest"
class CachedEmbedder
def initialize(embedder, store:) # store: Rails.cache, a Postgres table, or Redis
@embedder, @store = embedder, store
end
def embed(texts)
keys = texts.map { |t| key(t) }
cached = @store.read_multi(*keys)
missing = texts.zip(keys).reject { |_, k| cached.key?(k) }
fresh = missing.empty? ? [] : @embedder.embed(missing.map(&:first))
missing.map(&:last).zip(fresh).each { |k, v| @store.write(k, v) }
keys.map { |k| cached[k] || fresh[missing.index { _2 == k }] }
end
private
# Model name and version are in the key, so switching models never serves stale vectors.
def key(text) = "emb:#{@embedder.model}:#{Digest::SHA256.hexdigest(text.unicode_normalize(:nfkc).strip)}"
endOn "reduce latency 80%": that is a target, and whether you hit it depends on the hit rate, which you have to measure. With an embedding call of 120 ms, a cache read of 1 ms and a hit rate h, mean latency is h × 1 + (1 − h) × 120. At h = 0.8 that is 24.8 ms, a 79% cut; at h = 0.3 it is only 30%. Question caches pay off for repeated questions (support, internal search); the chunk cache pays off on every re-index and chunking sweep, where the hit rate is close to 100%. Record cache hits as a span attribute so the dashboard shows the real rate.
Dense vectors find meaning ("how do I reset my password" matches "recovering account access"). Sparse, keyword search finds exact strings: error codes, SKUs, names, statute numbers. Each misses what the other finds, so run both and fuse the rankings. Everything lives in one PostgreSQL table:
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE chunks (
id bigserial PRIMARY KEY,
organization_id bigint NOT NULL,
rooftop_id bigint NOT NULL,
document_id bigint NOT NULL REFERENCES documents,
parent_id bigint REFERENCES sections,
ordinal integer NOT NULL,
page_start integer NOT NULL,
page_end integer NOT NULL,
page_label text,
kind text NOT NULL,
published_on date,
body text NOT NULL,
embedding vector(1024) NOT NULL, -- bge-m3
tsv tsvector GENERATED ALWAYS AS (to_tsvector('english', body)) STORED
);
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON chunks USING gin (tsv);
CREATE INDEX ON chunks (organization_id, rooftop_id, kind);Fuse with Reciprocal Rank Fusion (RRF). Each list contributes 1 ÷ (k + rank) for each chunk it returned, with k = 60 by convention. RRF uses ranks, not scores, so it needs no tuning to combine a cosine distance with a text-rank score that live on different scales.
WITH dense AS (
SELECT id, row_number() OVER (ORDER BY embedding <=> $1) AS rank
FROM chunks
WHERE kind = ANY($3)
ORDER BY embedding <=> $1
LIMIT 50
), sparse AS (
SELECT id, row_number() OVER (ORDER BY ts_rank_cd(tsv, q) DESC) AS rank
FROM chunks, websearch_to_tsquery('english', $2) AS q
WHERE tsv @@ q AND kind = ANY($3)
ORDER BY ts_rank_cd(tsv, q) DESC
LIMIT 50
)
SELECT id, sum(1.0 / (60 + rank)) AS score
FROM (SELECT * FROM dense UNION ALL SELECT * FROM sparse) AS both_lists
GROUP BY id
ORDER BY score DESC
LIMIT 50;The same fusion in Ruby, for when the lists come from different places (or from several query variants, section 9):
def reciprocal_rank_fusion(*rankings, k: 60)
scores = Hash.new(0.0)
rankings.each { |ids| ids.each_with_index { |id, i| scores[id] += 1.0 / (k + i + 1) } }
scores.sort_by { |_, score| -score }.map(&:first)
end
# dense [a b c], sparse [c a d] → a: 1/61 + 1/62, c: 1/63 + 1/61, b: 1/62, d: 1/63
reciprocal_rank_fusion(%w[a b c], %w[c a d]) # => ["a", "c", "b", "d"]PostgreSQL's built-in ts_rank_cd is a decent keyword ranker but not BM25. For true
BM25, use ParadeDB's pg_search extension. bge-m3 can also emit learned sparse vectors, which
pgvector stores as sparsevec; try them as the sparse leg once the basic version is measured.
Filters do two jobs. The first is security, and it is not optional: every chunk carries
organization_id and rooftop_id, and row-level security makes it impossible to
retrieve another tenant's text, whatever the query says (Identity).
ALTER TABLE chunks ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_scope ON chunks
USING (organization_id = current_setting('app.organization_id')::bigint
AND rooftop_id = ANY (string_to_array(current_setting('app.rooftop_ids'), ',')::bigint[]));The second is relevance: let a question scope itself ("in the 2025 handbook", "only policies"), and put
those filters in the WHERE clause of both legs, as above, rather than filtering after retrieval.
Filtering afterwards can leave you with two results out of fifty.
One trap: an HNSW index scan returns its nearest candidates before the WHERE clause
runs, so a selective filter can return fewer than LIMIT rows. pgvector 0.8 and later fix this
with iterative scans: SET hnsw.iterative_scan = relaxed_order;. Test it: a filter matching 1% of
rows should still return 50 results.
Users ask follow-ups ("what about the second one?") and vague questions ("the limit?"). Retrieval can only match what the query says, so rewrite it into a standalone question first, using the conversation so far. When the question is genuinely ambiguous, ask instead of guessing. A small, cheap model does this well; keep it behind the generator adapter so it can be swapped.
REWRITE_PROMPT = <<~PROMPT
Rewrite the user's last message as one standalone search question, using the
conversation for any missing context. Keep names, numbers and codes exactly.
If it could reasonably mean two different things, reply with
CLARIFY: followed by one short question to ask the user instead.
Reply with the question only.
PROMPT
def rewrite(history, message)
text = generator.complete(system: REWRITE_PROMPT,
prompt: "#{history.last(6).join("\n")}\nUser: #{message}")
text.start_with?("CLARIFY:") ? Clarify.new(text.delete_prefix("CLARIFY:").strip) : text.strip
endLog the original and the rewrite on the same span. When answers go wrong, the rewrite is the first thing to check.
Rewriting makes one good query; expansion makes several, because the right chunk may use words the user didn't. Generate two or three alternatives, run hybrid search for each, and fuse all the lists with RRF.
Expansion raises recall and costs precision and latency, since each variant is another search. That is fine because the reranker (section 10) restores precision. Cap it at three variants and watch the latency panel.
An embedding model encodes the question and each chunk separately and compares the vectors. A cross-encoder reads the question and one chunk together and outputs a relevance score. That is far more accurate and far too slow to run over the whole corpus, so use it on the candidates: hybrid search returns 50, the reranker orders them, and the top 10 go forward.
module Rerankers
# TEI serves cross-encoders at POST /rerank and returns [{index:, score:}, ...].
class Tei
def initialize(url: ENV.fetch("RERANKER_URL"))
@uri = URI.join(url, "/rerank")
end
def rerank(query, texts)
response = Net::HTTP.post(@uri, { query:, texts:, raw_scores: false }.to_json,
"Content-Type" => "application/json")
response.value
JSON.parse(response.body).sort_by { _1["index"] }.map { _1["score"] }
end
end
end
scores = reranker.rerank(question, candidates.map(&:body))
top10 = candidates.zip(scores).max_by(10) { |_, score| score }Self-hosted options: BAAI/bge-reranker-v2-m3 or mixedbread-ai/mxbai-rerank-base-v1,
on TEI with the GPU, or in-process with the informers gem
(Informers.pipeline("reranking", ...)) when the volume is low. Hosted: Cohere Rerank or Voyage.
Track two things: whether top-10 accuracy rises (hit@10, NDCG@10 in section 17), and the added latency for 50
pairs. The reranker's top score is also the best confidence signal you have (section 15).
Late interaction sits between the two. Like an embedder, it encodes the question and the document separately, so document vectors can be precomputed. Unlike an embedder, it keeps one vector per token and scores with MaxSim: for each query token, take its best match among the document's tokens, and add those up.
score(q, d) = Σi ∈ q maxj ∈ d (qi · dj)
This keeps exact-term matching that a single pooled vector loses, at the cost of much more storage: a 500-token chunk is about 500 vectors of 128 dimensions instead of one of 1,024. Two sensible ways to use it:
Models: jina-colbert-v2, answerdotai/answerai-colbert-small-v1. This is one of the
places where Python is the right tool: the training and indexing libraries have no Ruby equivalent, so run the
encoder as a small HTTP service behind the same reranker interface.
MaxSim itself is a hot loop. Reranking 100 chunks of 180 tokens against a 32-token query is
100 × 180 × 32 × 128 ≈ 74 million multiply-adds. That is seconds in plain Ruby and milliseconds in C. Measure
first, then reach for a C-backed library such as numo-narray before writing a C extension
(Ruby and C).
Small chunks retrieve precisely but answer poorly: the matching paragraph often depends on the table above it or the definition two paragraphs earlier. Parent retrieval separates the two jobs. Search over the 500-token chunks, then hand the model each hit's enclosing section, capped at about 2,000 tokens.
def contexts_for(hits, budget: 8_000)
hits.map(&:parent).uniq.each_with_object([]) do |section, picked|
break picked if picked.sum(&:token_count) + section.token_count > budget
picked << section
end
endsections table with a path like
"3 Benefits › 3.2 Leave"). Without headings, use a window of the neighbouring chunks.Claude has a citations feature built for this. Send each retrieved passage as a document block with
citations: {enabled: true}, and the response comes back as text blocks, each carrying the exact
passage it relied on and which document it came from. Because document_index points back at our
own chunk, the page number comes from our database, not from the model.
require "anthropic"
class Generators::Claude
SYSTEM = "Answer only from the documents provided. If they don't contain the answer, " \
"say so plainly and don't guess."
def initialize(client: Anthropic::Client.new, model: ENV.fetch("RAG_MODEL", "claude-opus-5-5"))
@client, @model = client, model
end
def answer(question, chunks)
documents = chunks.map do |chunk|
{ type: "document",
source: { type: "text", media_type: "text/plain", data: chunk.body },
title: "#{chunk.document_title}, #{chunk.pages_label}",
citations: { enabled: true } }
end
message = @client.messages.create(
model: @model.to_sym, max_tokens: 16_000,
system_: [{ type: "text", text: SYSTEM }],
messages: [{ role: "user", content: [*documents, { type: "text", text: question }] }]
)
Answer.new(message.content.select { _1.type == :text }.map { |block| sentence(block, chunks) })
end
private
def sentence(block, chunks)
sources = Array(block.citations).map do |c|
chunk = chunks.fetch(c.document_index)
Source.new(quote: c.cited_text, document: chunk.document_title, pages: chunk.pages_label,
chunk_id: chunk.id)
end
Sentence.new(text: block.text, sources:)
end
endRendered, each sentence gets a footnote such as "Employee Handbook 2025, p. 14", linking to
handbook.pdf#page=16 (the physical page) with the quoted text on hover.
page_location
with start_page_number and end_page_number; plain-text documents report
char_location. Sending our own chunks keeps retrieval, tenancy and page labels under our
control, which is why it's the default here.output_config.format); the request is
rejected. Keep structured output for the rewriter, not the answer.stop_reason before reading the content, and handle "refusal"; production
code should enable Anthropic's server-side refusal fallbacks.RAG_MODEL), so another Claude model, or another provider's adapter,
is a change to .env, not to code (Twelve-factor apps). A
provider without native citations gets numbered sources in the prompt and asks for [n] markers;
the grounding check below then does the verifying.A citation feature tells you what the model points at. Grounding checks that the pointing is honest and complete. Run these checks on every answer, in code, before the user sees it:
[n] markers from other providers it often doesn't, which is why the check exists.def grounded?(answer, chunks_by_id)
answer.sentences.all? do |s|
next true unless factual?(s.text)
s.sources.any? && s.sources.all? do |src|
squish(chunks_by_id.fetch(src.chunk_id).body).include?(squish(src.quote))
end
end
end
def squish(text) = text.gsub(/\s+/, " ").stripTreat retrieved text as data, never as instructions. A document saying "ignore previous instructions" is a test case (section 18), and the system prompt should say plainly that documents can't change the rules.
Confidence comes from retrieval, not from the model's tone. Use the reranker's top score (as a probability, 0 to 1) and how many candidates cleared a floor:
def route(question, ranked)
top = ranked.first&.score.to_f
return :answer if top >= ENV.fetch("RAG_ANSWER_THRESHOLD", "0.5").to_f
return :web if web_allowed?(question)
:abstain
endweb_allowed? a deliberate rule (per tenant, per collection), default it to off, and log every
fallback./search?q=…&format=json on our machine. Hosted: Claude's
server-side web search tool ({ type: "web_search_20260209", name: "web_search", max_uses: 3 }),
which returns its own cited URLs, or the Brave Search API, each behind the same search(query)
interface.Every improvement on this page is a claim, and the golden set is how you check it. Fifty cases is the floor where a change of a few points starts to mean something; more is better. Each case says what a correct retrieval is, not just a correct answer, so retrieval and generation can be judged separately.
# test/golden/rag.yml
- id: leave-001
question: How many weeks of parental leave do full-time staff get?
relevant: # chunk ids or (document, page) pairs, with grades
- { document: handbook-2025, page: 14, grade: 2 }
- { document: handbook-2025, page: 15, grade: 1 }
answer_must_include: ["16 weeks"]
tags: [lookup, single-hop]
- id: leave-002
question: Does that apply to part-time staff too?
history: ["How many weeks of parental leave do full-time staff get?"]
relevant: [{ document: handbook-2025, page: 15, grade: 2 }]
tags: [follow-up, rewrite]
- id: none-001
question: What is the company's policy on Mars relocation?
relevant: []
expect: abstain
tags: [adversarial, no-answer]bin/rag-eval), writing one JSON line per case and one
summary, with the commit, chunking settings and model ids, so any two runs can be compared.For a query q, let rankq be the position of the first relevant chunk in the top k, and reli the graded relevance of the chunk at position i (0 for irrelevant, 1 relevant, 2 the best answer). Averaged over the set of queries Q:
A worked example: the system returns [c7, c2, c9, c4] and the relevant chunks are c2 and c4
(grade 1 each). Hit@1 = 0, hit@2 = 1. Reciprocal rank = 1 ÷ 2 = 0.5. DCG@4 = 1 ÷ log23 +
1 ÷ log25 = 0.63093 + 0.43068 = 1.06161; IDCG@4 = 1 ÷ log22 + 1 ÷ log23 =
1.63093; NDCG@4 = 0.65092.
# frozen_string_literal: true
# Retrieval metrics for one query. `ranked` is the list of retrieved chunk ids, best first;
# `relevant` maps chunk id => graded relevance (1 = relevant, 2 = the best answer).
module RetrievalMetrics
module_function
def hit?(ranked, relevant, k:)
ranked.first(k).any? { |id| relevant.key?(id) }
end
def reciprocal_rank(ranked, relevant, k:)
rank = ranked.first(k).index { |id| relevant.key?(id) }
rank ? 1.0 / (rank + 1) : 0.0
end
def ndcg(ranked, relevant, k:)
ideal = dcg(relevant.values.sort.reverse.first(k))
ideal.zero? ? 0.0 : dcg(ranked.first(k).map { |id| relevant.fetch(id, 0) }) / ideal
end
def dcg(gains)
gains.each_with_index.sum { |rel, i| (2**rel - 1) / Math.log2(i + 2) }
end
endThe test uses the hand-worked numbers above, not numbers produced by running the code:
def test_ndcg_matches_the_hand_worked_example
# DCG = 1/log2(3) + 1/log2(5) = 0.63093 + 0.43068 = 1.06161
# IDCG = 1/log2(2) + 1/log2(3) = 1.00000 + 0.63093 = 1.63093
assert_in_delta 0.65092, RetrievalMetrics.ndcg(%w[c7 c2 c9 c4], { "c2" => 1, "c4" => 1 }, k: 4), 1e-5
endReport hit@10, MRR@10 and NDCG@10 for retrieval, and separately for the answers: correct, grounded (section 14), abstained, and wrong. Queries with no relevant chunks are left out of the retrieval averages and scored in section 18 instead.
A RAG system that always answers is a hallucination engine with extra steps. Keep at least 10 golden cases, and ideally 20% of the set, where the right behaviour is to say the documents don't cover it:
Score two numbers and keep both visible, because improving one tends to hurt the other: the false answer rate on these cases (target: close to 0) and the over-abstention rate on answerable cases (how often it says "I don't know" when the answer was there). The thresholds in section 15 are where you trade one against the other.
RAG data goes to the same place as everything else: OpenTelemetry spans and metrics to the local collector,
shown in our Grafana (Observability). Register the service in
observability/services.yml like any other. Each query is a trace with a span per stage
(rewrite, expand, search, rerank, generate, ground) and attributes such as:
rag.query.rewritten = true
rag.search.candidates = 50
rag.rerank.top_score = 0.83
rag.route = "answer" | "web" | "abstain" | "clarify"
rag.grounding.cited_ratio = 1.0
rag.embedding.cache_hit = true
gen_ai.request.model = "claude-opus-5-5"
gen_ai.usage.input_tokens = 9120Panels worth having, top to bottom:
Never put question text or document text in metric labels. Questions can hold personal data, and the privacy rules for logs apply to traces too; keep text in the trace only where the retention policy allows it.
The point of all this is a number that goes up. Define the baseline precisely, then add one component at a time on the same golden set, the same generator and the same k, so each row shows what that component bought:
| Configuration | Hit@10 | MRR@10 | NDCG@10 | False answers | p95 latency |
|---|---|---|---|---|---|
| Naive: 500/50 chunks, dense only, top 5, no rerank, no abstain | – | – | – | – | – |
| + hybrid search (RRF) | – | – | – | – | – |
| + query rewriting and expansion | – | – | – | – | – |
| + cross-encoder rerank, 50 → 10 | – | – | – | – | – |
| + parent sections | – | – | – | – | – |
| + confidence threshold and grounding | – | – | – | – | – |
| + late-interaction rerank (instead of or after the cross-encoder) | – | – | – | – | – |
The table is empty on purpose: we haven't run it on our own documents yet, and published gains
from other corpora don't transfer. Fill it from bin/rag-eval and record the date and commit.
CHUNKING.md with the sweep results.Provided as is, for information only, with no warranty and no liability. Not legal or professional advice. Security and radio material is for authorized, lawful use only. Full disclaimer.
Visits are logged on our own server: IP address, browser and device details, and pages viewed. We use them to understand traffic and spot attacks, and never share or sell them.