Tartrazine::Tokenizer
Inherits Iterator / Enumerable / Tartrazine::BaseTokenizer / Reference / Object
=== 6. Identity fast path in split_tokens (perf only) ================ Stock rebuilt a fresh Array(Token) on every rule hit even though token values only rarely contain a newline. When none does, the split is the identity transform (each token maps to itself in order), so return the input array untouched — the only caller iterates it into the deque and never mutates it.
Instance methods
split_tokens(tokens : Array(Token)) : Array(Token)
If a token contains a newline, split it into two tokens