Tartrazine::Tokenizer
Inherits Iterator < Enumerable < Tartrazine::BaseTokenizer < Reference < Object
=== 6. Identity fast path in split_tokens (perf only) ================ Stock rebuilt a fresh Array(Token) on every rule hit even though token values only rarely contain a newline. When none does, the split is the identity transform (each token maps to itself in order), so return the input array untouched — the only caller iterates it into the deque and never mutates it.