Arrow interop

DataFrame speaks the Arrow PyCapsule Interface: __arrow_c_schema__ and __arrow_c_stream__ export a frame, and kt.from_arrow imports from anything on the other side of that protocol. No dependency on any particular Arrow library is required for this to work — any library that speaks the Arrow PyCapsule interface can produce or consume a keyten DataFrame directly, with no intermediate format. The examples below use pyarrow as the reference implementation, since it's the library most consumers already have.

Export: `DataFrame` → Arrow

df.__arrow_c_schema__()                    # ArrowSchema capsule
df.__arrow_c_stream__(requested_schema=None)  # ArrowArrayStream capsule

Both are dunders — you don't call them directly. Any Arrow-consuming library calls them for you when it recognizes the protocol:

import keyten as kt
import pyarrow as pa

df = kt.DataFrame([
    kt.Series.string("sym", ["a", "b", "a"]),
    kt.Series.int("qty", [1, 2, 3]),
])

table = pa.table(df)        # consumes __arrow_c_stream__
schema = pa.schema(df)      # consumes __arrow_c_schema__

assert table.num_rows == 3
assert table.column("qty").to_pylist() == [1, 2, 3]

Copy on export. Every value crossing the boundary is copied: export copies out of keyten's internal blocks into freshly allocated, Arrow-shaped buffers, so nothing aliases keyten's own storage and the exported table stays valid after the source frame is dropped.

Single-batch stream. __arrow_c_stream__ always yields exactly one record batch, built eagerly at call time (so a decode failure raises from __arrow_c_stream__ itself, not from a later pull), then signals end-of-stream. There is no chunking on export.

requested_schema is ignored. It's accepted for protocol compatibility, but v1 always exports the frame's native schema — no column projection or casting through it.

Import: Arrow → `DataFrame`

kt.from_arrow(obj)

Imports any object implementing __arrow_c_stream__ (preferred — handles multi-chunk tables, concatenating every chunk) or, failing that, __arrow_c_array__. Raises TypeError for an object implementing neither.

import keyten as kt
import pyarrow as pa

table = pa.table({"sym": ["a", "b", "a"], "qty": [1, 2, 3]})
back = kt.from_arrow(table)

assert back.to_dict() == {"sym": ["a", "b", "a"], "qty": [1, 2, 3]}

Dictionary-encoded columns and multi-chunk tables both work:

import keyten as kt
import pyarrow as pa

chunked = pa.table({"sym": pa.chunked_array([["a", None], ["b"]])})
assert kt.from_arrow(chunked).to_dict() == {"sym": ["a", None, "b"]}

dict_encoded = pa.table({"sym": pa.array(["a", "b", "a"]).dictionary_encode()})
assert kt.from_arrow(dict_encoded).to_dict() == {"sym": ["a", "b", "a"]}

v1 type map

Export

Every keyten Series kind exports to exactly one Arrow format:

keyten kind	Arrow format	Arrow type
`Int`	`l`	Int64
`Float`	`g`	Float64
`Bool`	`b`	Bool
`Str`	`u`	Utf8 (`i32` offsets)
`Date`	`tdD`	Date32
`Timestamp`	`tsn:`	Timestamp[ns], naive (no timezone)
`Time`	`ttn`	Time64[ns]

A Str column whose total UTF-8 byte length would overflow an i32 offset fails the export with a clear error rather than silently wrapping; there is no 64-bit-offset (U, LargeUtf8) export path in v1.

Import

kt.from_arrow accepts:

Arrow format	Arrow type	Decodes to
`l`	Int64	`Int`
`g`	Float64	`Float`
`b`	Bool	`Bool`
`u`	Utf8 (`i32` offsets)	`Str`
`U`	LargeUtf8 (`i64` offsets)	`Str`
`tdD`	Date32	`Date`
`ttn`	Time64[ns]	`Time`
`tsn:`	Timestamp[ns], naive	`Timestamp`
dictionary-encoded, string values	Dictionary\<Utf8 or LargeUtf8>	`Str` (dictionary discarded, values decoded to plain strings)

Rejected on import

Every other Arrow type is rejected with a TypeError rather than silently widened, truncated, or coerced:

Input	Rejected because	Error names
Timezone-aware `Timestamp[ns, tz=...]`	keyten only has naive timestamps	`"timezone-aware"`
`Int32` (`i`), `Int16` (`s`), `Int8` (`c`)	narrower than keyten's `Int` (Int64)	the Arrow format string, plus the type name (e.g. `"Int32"`)
`Float32` (`f`), `Float16` (`e`)	narrower than keyten's `Float` (Float64)	the Arrow format string, plus the type name
`UInt8`/`UInt16`/`UInt32`/`UInt64` (`C`/`S`/`I`/`L`)	no unsigned keyten kind	the Arrow format string, plus the type name
`Null` (`n`)	no null-only keyten kind	the Arrow format string, plus the type name
Any other unrecognized Arrow format	not in the v1 map	the raw Arrow format string
Sliced top-level struct/table array (`array.offset != 0`)	a struct's row window isn't threaded through independently-offset children in v1	`"sliced"`
Top-level struct/table array with a null row	a null row at the struct level has no keyten representation to fall back to	`"null"`
Struct/table array whose child length doesn't match the parent's length	the parent struct's length governs the row count; a child reported longer or shorter is rejected rather than decoded at its own (wrong) length	`"length"`

The narrow-width rejection is deliberate, not an oversight: a producer that defaults to Int32/Float32 gets a clear error naming the format instead of a silent widen. Concatenate/cast to the wider type before importing (e.g. pyarrow's table.cast(...)) if the source produces narrow columns.

import keyten as kt
import pyarrow as pa

try:
    kt.from_arrow(pa.table({"n": pa.array([1], type=pa.int32())}))
except TypeError as e:
    assert "Arrow" in str(e)

try:
    kt.from_arrow(pa.table({"ts": pa.array([0], type=pa.timestamp("ns", tz="UTC"))}))
except TypeError as e:
    assert "timezone-aware" in str(e)