Open Data Infrastructure

DataFusion CatalogProvider Boundaries for Multi-Tenant APIs

How DataFusion CatalogProvider boundaries separate tenants, schemas, table discovery, policy checks, and query-planning evidence.

A multi-tenant query API fails the moment table discovery becomes a shared junk drawer.

Catalog boundaries are API boundaries

Apache DataFusion is often embedded inside services that expose analytical query capability through an application API. That is powerful, but it makes tenancy a design problem. If every tenant, schema, and table is exposed through one loose catalog surface, the query planner can become the first place a policy mistake shows up.

DataFusion exposes catalog, schema, and table abstractions so applications can decide how tables are registered and resolved. In a multi-tenant API, those abstractions should align with ownership, allowed datasets, policy checks, and evidence collection.

Core idea: CatalogProvider boundaries should encode tenant and policy boundaries before the query planner sees a table.

Table discovery needs an owner

The DataFusion catalog user guide describes catalogs and schemas as the namespace layer for table registration. The DataFusion Python catalog API documentation shows the catalog objects exposed to application code.

That matters for ODI because embedded query services do not get a free governance pass. A table that can be resolved can often be planned. The service needs a boundary that says which tenant can discover which schema, which table provider is allowed, which policy checks ran, and which plan evidence should be kept.

Patterns that work

  • Create catalog boundaries that match tenant, product, or trust-zone boundaries.
  • Register tables through reviewed providers instead of ad hoc service code.
  • Attach policy decisions to table discovery, not only to final query execution.
  • Log catalog, schema, table, and plan identifiers for every API request.
  • Test negative discovery paths so denied tables fail before planning work expands.

For adjacent ODI context, read DataFusion TableProvider boundaries, DataFusion session boundaries, and policy-aware DataFusion query services.

What breaks first

  • Tenants share a catalog because the first version had only one customer.
  • A policy check runs after the table has already been resolved and planned.
  • Temporary tables bypass the same registration path as governed tables.
  • Incident evidence records SQL text but not the catalog boundary that allowed resolution.

Questions to ask

  • Which catalogs and schemas can each tenant resolve?
  • Which table providers are approved for agent-facing query APIs?
  • Where does the service record policy decisions and rejected discovery attempts?
  • Can operators replay the plan boundary for a disputed answer?

Sources to start with

These primary sources anchor the technical claims in this guide.

A catalog boundary is where multi-tenant safety starts becoming real.

ODI hub Article library Use the scorecard TableProvider boundaries
Get started with Apache Iceberg, today! Want to learn more? Visit https://www.opendatainfrastructure.com/