Skip to content

Data in and out

Keyten reads CSV and Parquet, and owns a native table format that persists the engine's blocks verbatim. The working pattern for repeated analysis: ingest once into a native table, query it from disk — scans are zero-copy, and only the columns and blocks a query needs are ever touched.

Round-tripping CSV

import keyten as kt

kt.DataFrame([
    kt.Series.string("sym", ["ev", "od", "ev", "od"]),
    kt.Series.int("qty", [100, 250, 300, 150]),
    kt.Series.float("px", [10.0, 20.5, 10.2, 19.9]),
]).write_csv("trades.csv")

out = (
    kt.scan_csv("trades.csv")
    .filter(kt.col("qty") > 100)
    .select(["sym", "px"])
    .collect()
)
assert out.shape == (3, 2)

scan_csv infers each column's type from a sample, then enforces it strictly — a bad value fails with the exact line number. Only referenced columns are parsed: the select above means a malformed value hiding in an unreferenced column would never be read, let alone fail the query. Dates (2026-07-19), timestamps (2026-07-19 09:30:00, fractional seconds to nanoseconds), and times (09:30:00) infer as first-class temporal kinds.

Reading Parquet

kt.scan_parquet("events.parquet")      # one file
kt.scan_parquet("events/")             # a directory of files
kt.DataFrame.read_parquet("events/")   # eager

Scanning is lazy — only footer metadata is read up front. At collect, projection prunes columns, row-group statistics answer pushed-down predicates, and dictionary-coded text columns stay dictionary-coded inside the engine, which is what keeps text-heavy analytics fast.

Native tables

For data you query more than once, convert to the native format:

import keyten as kt

kt.DataFrame([
    kt.Series.string("sym", ["ev", "od", "ev", "od"]),
    kt.Series.int("qty", [100, 250, 300, 150]),
]).write_csv("trades.csv")

# One-time ingest: any query can persist its result directly.
kt.scan_csv("trades.csv").collect_to_native("trades.k10dir")

# From here on, scan the native table.
out = (
    kt.scan_native("trades.k10dir")
    .group_by("sym")
    .agg(kt.col("qty").sum().alias("total"))
    .sort("sym")
    .collect()
)
assert out.to_dict()["total"] == [400, 400]

What the native format buys:

  • Zero-copy reads. Column files map into memory; blocks are used in place, nothing decodes until a query touches it.
  • Pruning at every level. A query opens only the column files it references; persisted per-block statistics let a pushed-down filter skip blocks cold, and per-value block indexes take point lookups straight to the blocks that can match.
  • Crash safety. Writes stage and atomically rename; appends land data before the manifest swap. A crash leaves the previous table intact, never a torn one.

Grow a table in place with append_native — the schema must match exactly:

kt.DataFrame([
    kt.Series.string("sym", ["ev"]),
    kt.Series.int("qty", [500]),
]).append_native("trades.k10dir")

n = kt.scan_native("trades.k10dir").collect().shape[0]
assert n == 5

Bigger than memory

None of the above assumes data fits in RAM. A map-only plan (scan → filter/derive → collect_to_native) streams block by block in constant memory whatever the file size; sorts, aggregations, and joins that outgrow the engine's self-derived budget spill to disk and finish. There is nothing to configure.