A grounded assistant pulling from a corpus of versioned SOPs or policy documents will, sooner or later, return an answer sourced from an obsolete version while a current one sits in the same index. The operator sees a fluent citation and no warning. This note sets up the failure mode and the test we plan to run against it.
The failure mode
In a corpus with versioned documents, retrieval returns the top k chunks by semantic similarity. If a v2 clause and a v4 clause of the same procedure both score high, naive RAG picks whichever ranks higher and generates from it. The user is given a confident answer with a citation. Nothing in the pipeline knows the two chunks contradict each other, and nothing tells the user that an older version was consulted.
Published work puts this failure in sharp relief. On version-specific retrieval tasks, naive RAG drops to 55% accuracy against a version-aware baseline that reaches 100%, with 25% of baseline failures attributed to "version mixing" where chunks from several versions get blended into one answer (VersionRAG, arXiv 2510.08109). A separate production account of a mid-size corpus found the same manual existed in four versions across three document stores, with the retriever happily returning conflicting procedures from all of them.
In a plant, a lab or a treasury function, this failure does not look like a hallucination. It looks like a correct answer that is quietly wrong.
The measurable question
On a mixed-version corpus, how often does a production assistant surface a stale clause as authoritative, and does a conflict-detection pass (semantic diff plus version metadata) reduce that rate without pushing end-to-end latency past the point where operators abandon the assistant?
Conditions we plan to publish with the result
- Corpus: a de-identified slice of the chemicals and fertilizer client's SOP and compliance document set, deployed on the Sherlock/Launchpad stack. Target size: 2,000 to 5,000 documents, with at least 15% carrying more than one active version.
- Retriever: the current Launchpad retrieval pipeline (embedding model and reranker versions to be pinned at test time).
- Baseline: naive top-k retrieval, k=5, with no conflict handling.
- Candidate interventions: version-aware filtering using document metadata; a cross-encoder NLI pass (DeBERTa-v3 nli variant) over retrieved pairs; a lightweight LLM judge over flagged pairs only.
- Hardware: CPU inference for retrieval and NLI; GPU for generation. Exact SKUs to be pinned.
- Evaluation set: 300 to 500 synthetic question-answer pairs generated from the corpus and reviewed by a subject-matter expert, with planted version conflicts in at least 30% of items.
- Acceptance thresholds for shipping: stale-answer rate on the planted-conflict subset reduced by at least 40 percentage points against baseline; added latency at p50 under 150 ms; added latency at p95 under 500 ms.
A number without these conditions is not a result.
Method we plan to test
- Baseline: run the current pipeline against the eval set. Capture retrieved chunks, generated answer, and the version of each cited document.
- Version-aware retrieval: filter retrieved chunks by an "effective date" and "superseded by" metadata field, preferring the current version. Fall back to older versions only when the current version lacks coverage.
- Conflict-detection middleware: between retrieval and generation, pass every pair of retrieved chunks through an NLI classifier. If any pair scores above a contradiction threshold, flag the retrieval.
- Resolution policy: on a flagged retrieval, resolve by recency for versioned documents; for non-temporal contradictions, generate an answer that names both claims and their sources rather than picking one silently.
- End-to-end measurement: stale-answer rate, groundedness (RAGAS-style), citation correctness, p50 and p95 latency, and operator-visible warning rate.
The design is informed by the ContRAG-Med formulation of weighted contradiction detection, the version-aware graph structure in VersionRAG, and the practical NLI gates in work such as Meridian and Director-AI. We are not inventing a new method here. We are measuring whether the combination holds its latency budget on a real enterprise corpus.
Observations
None yet. This note is published at Hypothesis stage.
Where this fails
We expect the approach to break or degrade in at least four conditions, and we will report each honestly:
- Metadata-starved corpora. If "effective date" and "superseded by" fields are missing or inconsistent, version-aware filtering reduces to similarity ranking, and the NLI pass carries the entire load.
- Semantically equivalent but syntactically different clauses. Two clauses can say the same thing in different words. A naive NLI pass will flag these as neutral rather than entailing, inflating the warning rate until operators stop reading warnings.
- Numeric conflicts that are not textual. A threshold change from 50 ppm to 100 ppm is a contradiction, but it is not well-detected by generic NLI. A number-aware checker is a separate module we have not yet scoped.
- Latency pressure in high-volume ITOps. The NLI pass on a top-5 retrieval is 10 pairs. On busy endpoints, that budget may not survive, and the pass will have to run only when retrieval confidence is low.
If the result holds under one condition and fails under another, we will say which, and we will not generalise.
What delivery teams can take from this
A conflict-detection middleware module designed to drop into any Sherlock-backed Launchpad engagement:
- Deterministic version-aware filter driven by document metadata.
- Optional NLI pairwise pass over the retrieved set, with a tunable contradiction threshold.
- A resolution policy that surfaces conflicts to the operator rather than hiding them.
- A benchmark harness that reports stale-answer rate, groundedness, and latency against a baseline.
The module ships only if the controlled test clears the acceptance thresholds above. If it does not, we will publish the negative result instead.
Open questions
- Does the chemicals and fertilizer client's SOP corpus carry usable version metadata, or will we need to synthesise it from file naming conventions and change logs?
- Does Enola, which already reviews proposed fixes in the Launchpad stack, already perform any form of retrieval-level deduplication that we can build on rather than duplicate?
- At what retrieval confidence do we skip the NLI pass entirely to protect the latency budget?
- Can Watson, the memory layer, store the operator's past resolution choices to tune the warning threshold per site?
Resolving these moves this note from Hypothesis to Controlled test.
Have a versioned SOP corpus or a compliance assistant whose stale-answer rate you want measured? Propose a challenge →