Noir::FastSearch
A memchr-driven fast path for String#index(String), String#byte_index(String)
and therefore String#includes?(String).
The stdlib implements substring search as a Rabin-Karp rolling hash: one
multiply-add per byte, no vectorisation, roughly 0.8 GB/s. Every detector
and analyzer gates its regexes with content.includes?("..."), so on a
scan that reads tens of megabytes of source the search itself was a
measurable share of the wall clock (profiled on superset: ~9% of the
Flask analyzer, ~50% of the Express analyzer).
The replacement scans for the rarest byte of the needle with memchr
(vectorised in every libc) and confirms each candidate with memcmp.
It tests every byte offset, so a miss here is a miss in the stdlib too.
A hit is returned directly only when the stdlib would have found the
same one: for byte_index always (both are byte-level), for index
only when every byte before the hit is ASCII, because the stdlib steps
by decoded character and can skip over a match that begins inside a
multi-byte or malformed sequence. Every other case falls through to
previous_def, so the result is identical by construction rather than
by argument. The fall-through is bounded by the old cost: it only runs
after a hit, and the stdlib scan stops at that same hit.
Constants
Rank of each byte value by how often it appears in source code, 0 =
most common. Measured over ~400 MB of Python, TypeScript, Go, Ruby,
Java, PHP, C#, Rust, JSON, YAML and Markdown from real repositories.
Only the ordering matters: the needle byte with the highest rank is
the one memchr looks for, so a needle like .route( is located by
its ( or r, not by the e or t that appear on every line.
Class methods
True when every byte in [0, count) is ASCII, i.e. the stdlib's
character stepping visits every byte offset in that prefix.