class

Tartrazine::Tokenizer

Inherits Iterator / Enumerable / Tartrazine::BaseTokenizer / Reference / Object

=== 6. Identity fast path in split_tokens (perf only) ================ Stock rebuilt a fresh Array(Token) on every rule hit even though token values only rarely contain a newline. When none does, the split is the identity transform (each token maps to itself in order), so return the input array untouched — the only caller iterates it into the deque and never mutates it.

Instance methods

split_tokens(tokens : Array(Token)) : Array(Token)

If a token contains a newline, split it into two tokens

Source