DataFusion TableProvider Boundaries for Governed APIs

The query service boundary is where a data product stops being a table and starts being an API.

TableProvider sits on the boundary

Apache DataFusion exposes extension points for custom data sources through TableProvider. The DataFusion docs describe the TableProvider as sitting between logical and physical planning, where filter and projection pushdown hints move toward the source.

That is exactly why DataFusion TableProvider implementations matter for governed APIs. The implementation decides which schema is visible, which filters can move down, which metrics are captured, which errors are explainable, and which policy checks happen before scan execution.

Pushdown is a control decision

The DataFusion custom TableProvider guide explains that filter and projection hints are passed during planning. The catalog docs connect tables, schemas, and TableProvider behavior inside DataFusion catalogs.

A governed data product API should treat those hints as reviewable decisions. If a filter is pushed to a remote source, the API should know whether policy was checked before pushdown, whether sensitive columns were projected away, and whether the resulting scan can be audited.

Patterns that work

• Expose a stable schema contract and record every schema version served by the provider.
• Evaluate policy before scan construction, not after rows return.
• Log pushed filters, projected columns, row limits, source snapshots, and error classes.
• Return clear denied, stale, and partial-result states instead of generic execution failures.
• Publish provider metrics as data product evidence for latency, scan size, denial rate, and source freshness.

For adjacent ODI context, read DataFusion for agents, ODI reference architecture for agents, policy enforcement across open systems.

What breaks first

• A custom provider hides source-specific behavior until a query fails in production.
• Projection pushdown removes fields but leaves policy decisions outside the API boundary.
• The API returns rows without recording which source version was scanned.
• Error handling collapses denied access, missing data, and stale data into the same exception.

Questions to ask

• Which schema is the provider allowed to expose?
• Where are policy checks performed relative to scan planning?
• Which pushdown decisions are logged for audit?
• Can consumers distinguish no data, denied data, and stale data?

Sources to start with

These primary sources anchor the technical claims in this guide.

• Apache DataFusion custom TableProvider guide
• Apache DataFusion catalogs documentation
• Apache DataFusion introduction

The provider boundary is where openness becomes an API contract.