The query optimizer

collect(), collect_with(), and explain() all run the same optimize(plan: Plan) -> Plan over the Plan tree a LazyFrame chain built, before anything executes. The result is guaranteed to answer exactly what the unoptimized plan would — every pass here is a rewrite, never a behavior change.

The passes

Four passes run in a fixed loop, up to ten rounds, until a round leaves the plan's rendered explain() string unchanged (the cheapest structural-equality check available, since Plan has no PartialEq):

fold_plan          // constant and identity folding
push_predicates     // predicate pushdown
prune               // projection pushdown
push_slices         // slice/limit absorption

Fold

Walks every expression in the plan and folds what it safely can:

• two literals under an operator fold to one literal (1 + 1 → 2)
• identity rewrites that hand back their operand untouched — x + 0, x - 0, x * 1, 1 * x, x AND true, x OR false, not(not(x)) — so whatever null x carries keeps propagating through it exactly as it would have unfolded
• a Filter whose predicate folds all the way to the literal true disappears from the plan entirely

Absorbing identities (x * 0, x AND false, x OR true) are deliberately not folded to a constant: collapsing them would erase the null propagation a null x is owed, and only the executor — not the optimizer — is allowed to decide that.

Predicate pushdown

Splits a Filter's predicate on its top-level ANDs and tries to sink each conjunct as deep into the plan as it legally can, merging with a scan's own predicate or an existing Filter below it. A conjunct only descends through a Select/WithColumns node when every column it touches passes through that node unchanged (an identity mapping) — anything referencing a computed column stops and gets wrapped in a Filter right above that node instead. A conjunct never crosses a Slice or a GroupBy at all.

A scan that already carries a limit (see slice absorption, below) is treated as a row-count barrier exactly like a Slice would be: nothing sinks into its predicate field, since filtering after that limit would change which rows survive versus filtering before it.

Projection pushdown

Computes the set of column names actually required at the plan's root, then walks back toward the scan translating that set through each node's own renames — a Select/WithColumns/GroupBy only demands the input columns its own expressions actually reference, not everything above it needs. The scan at the bottom narrows its projection to exactly that surviving set.

One guard: an all-literal query (every select entry independent of any column, e.g. select([lit(1).alias("x")])) would otherwise prune the scan to zero columns — but the executor reads a block's row count off its first column, so pruning would leave it nothing to measure length from. One base-schema column is kept alive in that case even though its values are never used.

Slice and limit absorption

Sinks a Slice(offset, len) toward the scan the same way predicate pushdown sinks a filter: merging with a Slice already below it, descending through row-preserving Select/WithColumns, and stopping at anything else (Filter, Sort, GroupBy). A zero-offset slice reaching a Scan is absorbed directly into that scan's own limit field rather than staying a separate plan node — this is why LazyFrame::limit(2) (sugar for slice(0, 2)) renders as limit=2 on the SCAN line itself in explain(), with no separate SLICE node at all.

Reading explain()

explain() renders the optimized plan tree, indented one level per nesting depth, root first:

SORT [sym]
  GROUPBY keys=[sym] aggs=[q10 = sum(q10), n = count(qty)]
    WITH [q10 = (qty * 10)]
      SCAN cols=[sym, qty] predicate=(qty > 1)

SCAN lists its surviving cols=[...] (or cols=* if projection pushdown never ran on it), an optional predicate=, and an optional limit=. WITH/SELECT/GROUPBY's aggs= list each entry as name = expr when the output name differs from a bare column reference. Expressions render fully parenthesized ((qty * 10)), so the same rendering is unambiguous regardless of operator precedence.

The guarantee

The optimizer never changes what a query returns — only how much work it costs. A folded expression evaluates to the same value (nulls included) as its unfolded form would have; a pushed-down predicate sees the same rows it would have seen at its original position, just earlier; a pruned scan reads fewer columns but the same rows in the same order. Comparing collect() against collect() on a plan with every optimizer pass disabled (there is no public switch for this — it's an internal invariant the test suite checks directly) is expected to produce an identical DataFrame, not merely an equivalent one.

Join rules

Predicate pushdown understands join shape. A conjunct referencing only left-side outputs pushes into the left subtree for every join flavor. A conjunct referencing only right-side outputs pushes into the right subtree only for inner joins — under a left join it must stay above, because filtering the build side would resurrect unmatched rows with nulls where the filter should have dropped them. For asof joins, right-side pushdown is refused unconditionally: removing any right row can change which row is "the latest at or before". Mixed conjuncts always stay above the join. Projection pruning reaches through joins keeping key columns — and any left column whose mere existence forces a _right suffix — alive, so output naming stays stable under optimization. All of this is enforced by an equivalence suite that runs every optimized plan against its unoptimized twin.