Open Data Infrastructure
Substrait: Portable Query Plans for a Composable Stack
Why an engine-agnostic relational plan format matters for real interoperability across modern data systems.
Most stacks claim interoperability because they can read the same files. That is the easy part. The hard part is making meaning portable across engines, optimizers, and execution layers.
The interoperability problem
Interoperability failures usually look like this: the same query returns different results across engines, optimizations break semantics, or a feature only works inside one system boundary. Files and table formats help, but they do not solve the “compute meaning” problem.
If ODI is the control test, compute interoperability is one of the most expensive places to fail. It forces you to pick one engine forever, or to accept silent semantic drift.
What Substrait is
Substrait is an open specification effort for representing relational query plans in a portable way. The goal is not to replace SQL. The goal is to represent the meaning of a query after parsing, in a way that different systems can exchange.
In other words, Substrait is an intermediate representation (IR) for relational compute. If systems can agree on the IR, they can compose: one system can plan, another can optimize, another can execute.
Core idea: file interoperability is table-level. Substrait is compute-level.
Why query plan portability matters
Query plan portability matters when you want a composable stack:
- Multiple engines: pick the engine that fits a workload without rewriting the world.
- Shared optimizers: reuse optimization work across systems rather than re-implementing it repeatedly.
- Pushdown boundaries: push filters and projections closer to data while keeping semantics explicit.
- Governed compute: enforce policy and auditing at clear control points even as engines change.
This aligns with ODI because it reduces the number of hidden, proprietary contracts that determine meaning.
Limits and realistic expectations
No plan format can magically eliminate differences between engines. Engines have different type systems, different function semantics, and different optimization tradeoffs. A portable plan can still be interpreted differently.
The realistic win is narrower and more valuable: it gives the ecosystem a shared language for describing compute semantics, so interoperability becomes an engineering problem with explicit contracts, not a marketing claim.
Substrait is also only one layer. You still need open table formats, open catalog boundaries, and portable governance signals to pass the ODI control test end-to-end.
Sources to start with
Start with the Substrait specification and the projects implementing it.