AI-Ready Context Source Ranking Tests

The most dangerous retrieval result is not always wrong. Sometimes it is just less authoritative than the source it buried.

Retrieval needs source authority

AI-ready context is not only about finding relevant text. It is about preferring the source the business actually trusts. A stale wiki page, a retired metric definition, and an owner-approved catalog record may all look semantically close. Only one should win.

Source ranking tests make that preference explicit. They create cases where the system must choose between popular, recent, stale, official, derived, and restricted sources.

Test the ranking path

Build evaluation sets with competing sources. Include a catalog record, a dbt semantic definition, a runbook, a lineage event, a document, and an older but popular page. The expected answer should cite the authoritative source and explain why lower-ranked sources were not used.

Signals to preserve

Useful signals include owner, freshness, lineage, contract state, policy state, source system, update cadence, and deprecation status. OpenLineage facets, catalog metadata, dbt source freshness, and evaluation records can all contribute evidence.

For related ODI guidance, read context graph source authority ranking, AI-ready context evaluation datasets, and AI-ready context authorization receipts.

What breaks first

• The retriever ranks a stale document above the catalog record because it has better prose.
• The index knows freshness but the prompt path does not expose it.
• Restricted context is correctly denied, but the fallback source is not authoritative.
• Evaluation tests answer wording but not source selection.

Ranking test questions

Ask whether the test set includes conflicts, whether authority is labeled, whether denied sources are tracked, and whether the answer cites the source that the data owner would trust.

Sources to start with

These primary sources anchor the technical claims in this guide.

• OpenAI evals documentation
• OpenLineage object model documentation
• dbt semantic models documentation
• NIST AI Risk Management Framework

Relevance finds candidates. Authority decides what should be believed.