Cells and encodings

Every column is a run of one or more blocks, each of at most 2,048 rows (BLOCK_LANES). A block is a Cell: one fixed-size, 64-byte, 64-byte-aligned header, plus an optional payload the header points to.

The cell

#[repr(C, align(64))]
struct Cell {
    kind: u8,          // Bool | Int | Float | Str | Table | Date | Timestamp | Time
    enc: u8,            // Plain | Constant | Sequence | ... | Directory
    attrs: u8,           // SORTED | HAS_NULLS | PAYLOAD_MMAP | PAYLOAD_BORROWED | HAS_ZONEMAP
    _resv: u8,
    rc: AtomicU32,       // reference count
    len: u64,            // logical row count
    payload: *mut u8,
    payload_bytes: u64,
    meta: [u8; 32],      // encoding-specific: constants, first+step, bounds, zone map, ...
}

Field	Bytes	Purpose
`kind`	1	`Bool` (0) · `Int` (1) · `Float` (2) · `Str` (3) · `Table` (4) · `Date` (16) · `Timestamp` (17) · `Time` (18)
`enc`	1	Which of the thirteen encodings this block uses
`attrs`	1	Bit flags: sorted, has-nulls, mmap-backed payload, borrowed payload, has-zone-map
`_resv`	1	Padding
`rc`	4	Atomic reference count — a block is shared, not copied, until it's mutated
`len`	8	Row count (≤ 2,048 for a leaf block)
`payload`	8	Pointer to the block's out-of-line bytes, or null
`payload_bytes`	8	Payload length in bytes
`meta`	32	4 general-purpose `u64`/`i64`/`f64` slots — a constant value, a sequence's first+step, or a zone map's min+max

64 bytes total: the struct is asserted at compile time to be exactly one cache line, both in size and alignment. Every block, regardless of its kind or encoding, is described by this one shape — a caller never needs to know which encoding a block picked before reading its header.

Kind and Encoding are orthogonal: any kind may end up in any encoding its data supports (Int can be Plain, Constant, Sequence, For, Delta, Rle, or a multi-block Directory; Str has its own string-only encodings). The kind never changes once a column exists — only its physical encoding can differ block to block.

Encodings

Encoding	Kind(s)	Physical shape
`Plain`	any	Dense, aligned lanes — the fallback for irregular data
`Constant`	any	One value in `meta`, no payload at all
`Sequence`	`Int`	First value + step in `meta`, no payload — an arithmetic progression
`BoolBits`	`Bool`	One bit per row
`For`	`Int`	Frame of reference: a minimum in `meta` plus bit-packed residuals sized to the block's actual range
`Delta`	`Int`	Successive differences, for runs that drift slowly rather than sitting in a narrow range
`Dict`	`Str`	A small distinct-value table plus one code per row
`Rle`	any	Run-length pairs, for long repeated runs that aren't a single constant
`Sym8`	`Str`	Strings up to 8 bytes packed byte-for-byte into a `u64`, so lexicographic string order falls out of plain integer order
`Sym8For`	`Str`	Order-preserving `Sym8` words stored as one base plus bit-packed unsigned residuals
`Views`	`Str`	A 16-byte view per row (length, plus either the string inline or a 4-byte prefix and an offset into a trailing heap) for longer or more varied strings
`Directory`	any	Not a leaf: a vector-level cell whose payload is a list of child block cells, each (but possibly the last) exactly `BLOCK_LANES` rows
`GlobalDict`	`Str`	Codes into a column-global dictionary shared by every block of the column — the value store lives at column scope, in memory as an attached column and at rest as a metablock

A builder (IntBuilder, FloatBuilder, BoolBuilder, StrBuilder) picks a block's encoding from what was actually pushed into it — a run of 0,1,2,3,… becomes Sequence with no payload at all; a column under 2,048 rows never becomes a Directory in the first place. This choice is per block: two neighboring blocks in the same column can use entirely different encodings.

The null mask trailer

A block with HAS_NULLS set carries a fixed 256-byte validity bitmap (MASK_BYTES, one bit per row across all 2,048 possible lanes) appended after its data payload — validity() reads it as the last 256 bytes of payload_bytes. A block without any null in it skips the trailer entirely and clears HAS_NULLS, so an all-valid block costs nothing beyond its data.

Zone maps

Int, Float, and Sym8 blocks may also carry a min/max zone map in two of the four meta slots, flagged by HAS_ZONEMAP. A comparison kernel checks a block's zone map before touching its payload at all: if the map alone proves a predicate true or false for every row in the block, the whole block is resolved without decoding a single value — a Constant-true or Constant-false mask comes back instead, and later stages (see execution and parallelism) can skip re-reading the block's data entirely once its mask says so.

Sequence and Constant blocks never need a stored zone map — their bounds are derivable directly from meta (a Constant's one value, or a Sequence's first value and length) without ever setting HAS_ZONEMAP at all.

Width tiers

When an Int block does have to decode, it decodes at the narrowest width its value range fits, not automatically at 64 bits. The decoded form is a tier of 8-, 16-, 32-, or 64-bit lanes plus one per-block offset, where each value is offset + lane:

A Constant block is all-zero 8-bit lanes with the value as the offset — as narrow as it gets.
A Sequence tiers by its full span (checked in 128-bit arithmetic, so a wrapping sequence safely decodes wide).
A For block is the natural fit: its payload already is a base plus bit-packed residuals, so the residuals stream straight into the tier matching their packed width with the base as the offset — no 64-bit intermediate is ever materialized.
Plain, Delta, Rle, and Dict blocks decode wide as before.

Integer kernels consume the tiers directly. Aggregation folds narrow lanes and factor the offset out of the loop (a sum is offset × valid-count + Σlane, bit-identical to the wide fold); block-vs-constant comparisons rebase the constant once and compare narrow lanes in place; filters widen only the rows they actually select; and arithmetic between two same-tier blocks widens each element in registers rather than materializing either side. The tier is invisible in every result — it only changes how many bytes per row the hot loops touch. It's also why Int is one logical kind rather than a family of fixed-width types: the block already knows how wide its values really are.

For the hottest three operations — sum, compare-against-a-constant, and filter selection — For and Delta blocks go one step further and skip the scratch buffer entirely, computing on the encoded payload. A For sum is base × valid-count plus the sum of the packed residuals as they unpack (null lanes pack residual zero, so they contribute nothing); a For comparison rebases the constant into the residual domain once and never reconstructs a value at all; a Delta block scans its running recurrence in registers, feeding sums, comparisons, or a filter's selected rows directly. Every other operation, and every other encoding, falls back to the tiered scratch above — the surface is deliberately small, extended only where measurement justifies it.

Beyond the block: column-scope metadata

Some truths belong to a column, not to any one block, and the native format stores them as metablocks — a second record class beside the data that a reader who does not know a kind simply skips. Three exist today: an ingest-time distinct-count sketch per column (what the grouping engine's partition decision reads), the column-global dictionary a GlobalDict block's codes point into (measured at ingest — a column only goes global-dict when the dictionary provably pays for itself), and per-value block indexes mapping each distinct value of a high-cardinality column to the blocks containing it, which take an equality or membership predicate straight to the blocks that can match — the needle-in-haystack complement to zone maps. All three are derived by measurement at write time; none is configured.