Input, output, and runtime
Three sources feed a query: CSV, Parquet, and the engine's own native table format. All three scan lazily. Building a CSV plan reads its header and a bounded inference sample; execution parses only the columns the query references, in 2,048-row blocks. A bare limit stops the CSV source at the requested prefix. collect() retains the returned table, while collect_to_native() streams blocks to disk.
CSV
scan_csv(path, *, delimiter=",", has_header=True, infer_rows=1024, kind_overrides=None)
csv_to_native(src, dst, *, delimiter=",", has_header=True, infer_rows=1024, kind_overrides=None)
df.write_csv(path)
Types are inferred from the first infer_rows rows — integer, float, boolean, date, timestamp, time, then string — and the inferred kind is then enforced strictly over the whole file: a value that fails its kind raises ComputeError naming the exact line. Use kind_overrides={"price": "float", "order_id": "str"} when a sampled prefix does not represent the whole column. The one frozen exception: only the columns a query references are parsed, so a malformed value in a column nothing touches never fails the query. A bare limit is also honored at the CSV source, so values after the requested prefix are not parsed. Empty fields are nulls; delimiter must be exactly one byte. write_csv writes the forms inference reads back — round-tripping types without a schema file, with the inherent caveats that entails (a string column that looks numeric re-infers as numeric).
csv_to_native is scan_csv(src, ...).collect_to_native(dst) in one call, with the same options — the direct route for the ingest-once pattern, no intermediate LazyFrame to build by hand.
import keyten as kt
kt.DataFrame([
kt.Series.string("sym", ["ev", "od"]),
kt.Series.float("px", [1.5, 2.5]),
]).write_csv("data.csv")
kt.csv_to_native("data.csv", "data.k10dir")
assert kt.scan_native("data.k10dir").collect().shape == (2, 2)
import keyten as kt
kt.DataFrame([
kt.Series.string("sym", ["ev", "od"]),
kt.Series.float("px", [1.5, 2.5]),
]).write_csv("data.csv")
out = kt.scan_csv("data.csv").filter(kt.col("px") > 2.0).collect()
assert out.to_dict() == {"sym": ["od"], "px": [2.5]}
Parquet
Building the lazy plan reads only footer metadata. At collect, projection prunes columns, predicate statistics drop row groups a filter cannot match, and dictionary-coded text arrives as coded columns the engine keeps coded through downstream operations.
Native tables
scan_native(path) # lazy
DataFrame.read_native(path) # eager
df.write_native(path)
df.append_native(path)
lf.collect_to_native(path) # query result straight to disk
The native format persists a table as a directory whose files are the engine's own blocks, serialized verbatim — a persisted table is not an export; it is the in-memory representation, durable. That buys three things for free: zero-copy reads (files map into memory and cells point straight at them; nothing decodes until touched, and results stay valid after their source frame is gone), projection pruning at file granularity (a query over two of ten columns opens two files), and cold statistics (per-block min/max bounds and table-level metadata persist, so a pushed-down predicate skips blocks — and with per-value block indexes, a point lookup skips straight to the blocks holding its value — before their pages are ever faulted in).
Writes are atomic: stage, fsync, rename. Appends require the exact same schema and land data before the manifest swap, so a crash mid-append leaves the previous table intact. The manifest is the only truth — readers walk exactly what it references. One writer at a time; readers always see a consistent snapshot.
import keyten as kt
kt.DataFrame([
kt.Series.string("sym", ["a", "b", "a"]),
kt.Series.int("qty", [1, 2, 3]),
]).write_native("trades.k10dir")
out = (
kt.scan_native("trades.k10dir")
.group_by("sym")
.agg(kt.col("qty").sum().alias("total"))
.sort("sym")
.collect()
)
assert out.to_dict() == {"sym": ["a", "b"], "total": [4, 2]}
Out-of-core execution
Queries are not limited by memory. The engine derives a budget from the effective total (inside a container, the container's allowance) and every stateful stage consults it: sorts spill sorted runs and merge them, aggregations and joins partition to disk, and map-only plans stream source-to-sink in constant memory regardless. Spill files are ordinary native tables in a per-query temporary directory, removed when the query ends however it ends.
Workers
set_workers(n) # 0 restores automatic sizing
effective_workers() # what queries will actually use right now
The engine sizes its worker pool to the available cores; set_workers overrides it process-wide, read fresh at each stage boundary. Everything else about execution — parallelism, budgets, encodings — is derived by the engine from the data and the machine, deliberately without knobs.