Open Data Infrastructure
Agentic Data Quality Feedback Loops
How agent failures, retrieval misses, policy denials, and corrections should feed data quality work instead of staying in AI logs.
Agent failures are data quality signals wearing an AI costume.
AI logs are not enough
OpenLineage facets can attach contextual metadata to jobs, runs, and datasets. DataHub assertions and data contracts give teams a way to represent data quality checks as verifiable conditions.
Agent systems create new signals: retrieval misses, correction events, policy denials, stale context warnings, failed tool calls, and eval regressions. If those signals stay only in AI logs, the data product never gets better.
Feedback belongs in the data product
The feedback loop should route agent failures back to the owner who can fix the underlying data product. A bad answer may indicate missing documentation, weak examples, stale freshness, ambiguous semantics, broken lineage, or a policy rule that needs clarification.
That is different from prompt tuning. Prompt tuning can hide a data-quality problem for a while. A data product feedback loop fixes the thing the prompt was trying to compensate for.
Core idea: Agentic data quality means machine-consumer failures become producer work.
The ODI quality loop
Open Data Infrastructure can connect agent traces, eval results, lineage, assertions, ownership, and contract changes. The loop should be visible enough that owners can prioritize fixes and operators can prove whether fixes improved behavior.
For adjacent context, read agentic data product observability, AI-ready context quality tests, and data product SLAs.
What breaks first
- The AI team fixes prompts when the data product needs better semantics.
- Policy denials are treated as application errors instead of access-contract feedback.
- Corrections improve one agent workflow but never update catalog examples or tests.
- Data product owners do not see eval failures tied to their assets.
Questions to ask
Ask which agent failures become data product tasks. Ask how those tasks connect to assertions, contracts, ownership, and eval improvements. Ask whether denials and misses are classified instead of discarded.
The fastest way to improve agents may be to stop treating every agent failure as an agent problem.
Sources to start with
These primary sources anchor the technical claims in this guide.
- OpenLineage facets documentation
- DataHub assertions documentation
- DataHub data contracts documentation
- OpenAI evals documentation
- NIST AI Risk Management Framework
Better data beats a clever prompt more often than vendors want to admit.