Documentation

Keyten is a compact Rust dataframe engine with a Python-first surface: a typed Series/DataFrame pair for eager data, a LazyFrame for building a query as a plan instead of running each step eagerly, and an engine that optimizes and executes that plan over compressed, block-encoded columns. Execution is parallel on one machine, spills when state outgrows memory, and can ship serializable plans to workers.

Quickstart
Install the package and run the first query.

Installation
Supported systems, version checks, source builds, and common failures.

Tutorials
Queries, files, joins, and time series — every example runs.

Python API
Every class, method, and function, with its exact contract.

Engine
Cell layout, encodings, the optimizer, and execution.

What the engine covers

• Value kinds: Int, Float, Bool, Str, and first-class Date, Timestamp, and Time, all nullable
• Expressions: arithmetic, comparisons, boolean logic, math, casts, string operations, temporal extraction, conditional chains, recoding, and null/NaN shaping
• Aggregations: totals, extrema, moments, exact distinct counts, interpolated quantiles, and pairwise correlation, per group or whole-frame
• Windows: cumulative, row-count rolling, duration rolling, exponentially weighted statistics, ranking, filling, and partitioned over(...) evaluation
• Joins: inner, left, semi, anti, plus backward, forward, and nearest asof matching with groups and tolerance
• Eager table operations: append, upsert, update, delete, pivot, unique, drop-null, and Arrow PyCapsule interop
• Sources: CSV (strict typed inference), Parquet (files or directories, statistics-pruned), and the engine's native block format (zero-copy, crash-safe, indexed)
• Out-of-core execution against a self-derived memory budget — sorts spill, aggregations and joins partition to disk, map pipelines stream in constant memory
• Remote execution of serializable plans and scatter-gather aggregation over native scans
• A pushdown-and-folding optimizer in front of a block-parallel executor

The Python package is the primary product surface. Local work needs no service: pip install keyten includes the engine. The repository also builds a headless worker for explicit remote or distributed collection; it is optional and never started by importing the package.

Design pillars

• Blocks over one column, not one column over blocks. Every column is a run of fixed-size, independently encoded blocks behind a uniform 64-byte cell header. See cells and encodings.
• Plans, then a plan rewrite, then execution. LazyFrame methods build a plan tree; collect() runs it through a fixed-point optimizer before any data moves. See the query optimizer.
• The optimized plan is not a shortcut, it's the same answer. Every rewrite the optimizer performs is required to leave the result identical to running the unoptimized plan.
• Decisions are derived, not configured. Encodings, parallelism, partition counts, and memory budgets come from measuring the data and the machine — the API has one knob (set_workers), and everything else is the engine's job.

Choose your path

I want to analyze data in PythonI want to work with filesI need mutable tables or workersI want to understand the engine

Start with the quickstart, then the writing queries tutorial; keep the Python API map at hand.

Data in and out covers CSV, Parquet, and the native format, including the ingest-once pattern and what each format can prune.

Use the mutation tutorial for eager append/upsert/update/delete, or the distributed execution guide for workers and plan shipping.

Start with cells and encodings, then the query optimizer, execution and parallelism, and the architecture deep view.

At a glance

Layer	What it provides
Python API	Series/DataFrame for eager and mutable table state, LazyFrame/Expr for queries, scans for CSV/Parquet/native tables, and Arrow interop
Storage	64-byte cell headers over 13 physical encodings with per-block statistics, column-global dictionaries, and per-value block indexes at rest
Optimizer	Constant/identity folding, predicate pushdown (through joins where legal), projection pushdown, slice absorption, run to a fixed point
Execution	A block-parallel pipeline: streamed scans through fused filter/map into aggregation, sort, window, join, and file sinks, with vectorized aggregation and join membership filters
Bigger than memory	A self-derived budget; spilling sorts, partitioned aggregations and joins, constant-memory streaming for map-only plans
Beyond one process	Versioned plan serialization, headless workers, compressed result transport, and scatter-gather partial aggregation