struct

C0::Tokenizer

Inherits Struct / Value / Object

High-performance zero-copy tokenizer for C0DATA.

Scans a byte buffer for control codes (< 0x20) and emits tokens as offsets into the original buffer. No allocation for data values.

The hot loop is a single comparison: byte < 0x20.

Constructors

new(buf : Bytes)

each

Yields each token to the block. This is the primary interface for streaming consumption.

to_a

Collects all tokens into an array. Convenience method for non-streaming use. Prefer each for performance.