Open Data Infrastructure Evaluation Harnesses for AI

AI evaluations miss the point when they test answers but ignore the data infrastructure behind them.

The data layer needs to be evaluated too

An AI system can pass a response-quality evaluation while depending on stale tables, missing policy checks, broken lineage, unowned metrics, or non-repeatable retrieval. That is not a model problem. It is an infrastructure evaluation problem.

An ODI evaluation harness tests the data-control behaviors that support the AI system. Can the system access only approved data? Can it see freshness? Can it preserve lineage? Can it refuse when evidence is missing? Can the same test run produce reviewable evidence next week?

Evaluation needs policy, lineage, and telemetry signals

The NIST AI Risk Management Framework frames AI risk management as a lifecycle discipline. OpenLineage supplies lineage event concepts, and OpenTelemetry supplies telemetry concepts for system behavior.

An ODI harness ties those ideas to concrete tests. It tests tool calls, catalog reads, policy decisions, table freshness, retrieval context, lineage evidence, refusal behavior, and audit packet completeness.

Test the control path, not only the answer

• Create fixtures for approved, denied, stale, missing, and contradictory data cases.
• Validate that the agent reads catalog, policy, freshness, and lineage evidence before answering.
• Record tool calls, retrieved context, policy decisions, and final answer in one audit packet.
• Run the harness after model, prompt, data, policy, and catalog changes.
• Track failure categories that map back to data owners and platform teams.

This belongs beside the ODI scorecard, AI audit packet design, and negative evidence sets. The harness should prove the system can be trusted for the reasons the architecture claims.

What breaks first

• Evaluation scores improve while data access violations go untested.
• Freshness failures are treated as wrong-answer failures instead of infrastructure failures.
• Lineage is captured in production but absent from evaluation evidence.
• The test set cannot reproduce a past failure because source state was not recorded.

Questions to ask before trusting the harness

• Which ODI controls does the harness actually test?
• Does it include negative evidence, denial, and stale-context cases?
• Can failures be routed to data owners, policy owners, or platform owners?
• Can an auditor rerun or inspect the evidence from a prior test?

Sources to start with

These primary sources anchor the technical claims in this guide.

• NIST AI Risk Management Framework
• OpenLineage documentation
• OpenTelemetry documentation

Do not just evaluate the answer. Evaluate the path that made the answer possible.