Open Data Infrastructure
DuckDB Replacement Scans and Governed DataFrames
How DuckDB replacement scans should carry provenance, policy, and reproducibility when local DataFrame analytics becomes ODI infrastructure.
A notebook can look harmless until its DataFrame becomes the source of a metric, an evaluation, or an agent answer.
Local analytics still needs governance
DuckDB replacement scans let SQL refer to DataFrames in the Python process by name. That is a great developer experience. It also means a local object can become queryable infrastructure before the platform team has a chance to ask where it came from.
The ODI problem is not DuckDB. The problem is treating DataFrame access as informal just because it happens close to the analyst. If the output feeds an AI evaluation, a data product, or a governed decision, the DataFrame needs source evidence.
Core idea: Replacement scans are safest when DataFrames carry the same source, policy, and reproducibility evidence as governed tables.
The scan boundary is real
The DuckDB SQL on Pandas guide explains that replacement scans replace references to a table name with a table function that reads the DataFrame. The Python overview notes that DuckDB can directly query Pandas DataFrames, Polars DataFrames, and Arrow tables.
That boundary is where governance should attach. The query might be local, but the evidence should not be local-only. Store the input source, extraction time, row-count checks, schema fingerprint, notebook owner, policy context, and any filtering that happened before DuckDB touched the object.
Patterns that work
- Wrap DataFrame creation with a small evidence object that records source URI, snapshot, owner, and policy label.
- Block replacement-scan access to sensitive DataFrames unless the notebook session has an approved purpose.
- Persist row counts, schema hashes, and filter summaries next to evaluation outputs.
- Promote repeated notebook flows into named data products once they become operational dependencies.
- Treat local DuckDB queries as read paths that still need lineage when their outputs leave the notebook.
For adjacent ODI context, read DuckDB at the edge, AI-ready runtime tests, lineage for AI-ready infrastructure.
What breaks first
- A DataFrame is copied from production once and reused for weeks with no freshness signal.
- An AI evaluation cites a result but not the source file, snapshot, or filtering step.
- Notebook users rename columns locally and accidentally change metric meaning.
- Sensitive fields are dropped in one cell, but another cell still has the original object in memory.
Questions to ask
- Which DataFrames are allowed to be queried by name?
- Can the output be traced back to a source snapshot and policy decision?
- Which notebook sessions can access regulated fields?
- When does a local analytic flow need to become a governed data product?
Sources to start with
These primary sources anchor the technical claims in this guide.
Local does not mean informal once the result becomes evidence.