Noir::FileUrlScanner
Shared line/URL extraction for the FileAnalyzer hooks
(file_analyzers/string.cr, file_analyzers/base64.cr).
Those hooks are the only analyzers that run against every file in the
project regardless of detected technology, so both what they accept and
what they cost apply to every scan that passes -u/--url. Three things
are centralised here:
- reading through the detector's content cache instead of re-opening the file per hook,
- rejecting lines that are binary payload rather than text,
- turning a raw
https?://…run into a URL, or rejecting it.
Constants
A run of base64 alphabet characters long enough to hold a URL. The surrounding text can make any substring look like one, so decoding is attempted per token and failures are ignored rather than propagated.
Control characters that never appear in a text line. \t (0x09) is
legitimate indentation; \n / \r are consumed by each_line. A line
carrying any of the rest is binary payload that happens to sit in a
file with a text-ish extension — the detector's binary sniff only reads
the first 512 bytes, so protobuf/asset blobs with a clean header still
reach the analyzers. URL-shaped byte runs inside them are noise.
Bracket pairs that may legitimately appear inside a URL
(…/wiki/Foo_(bar)), so only an unbalanced closer is trimmed.
Request files are parsed properly (method, headers, body, {{vars}})
by the http_file specification analyzer, so scraping their bare URLs
here would only add a duplicate GET for every non-GET request.
Trailing sentence/prose punctuation. A URL at the end of a Markdown
link, a comment or a sentence carries the delimiter with it
(see https://x/donate/).), and that delimiter is not part of the URL.
A URL literal, terminated by any character that cannot appear in one.
Besides whitespace this stops at the quoting/markup characters that
actually wrap URLs in the wild — " and ' (string literals), < and
> (XML/HTML markup and RFC 3986 angle-bracket delimiters), and the
RFC 3986 "unwise" set (\ ^ ` |). Without the markup stop,
<string>https://x/install</string> yielded the endpoint
https://x/install</string>.
{ / } are deliberately kept: a templated https://api/x/{id} is a
real endpoint declaration, not markup.
Class methods
True when line is binary payload rather than source/prose text.
Yields each line of path with its 0-based index, reading through the
detector's content cache when the file is there. The hooks used to
File.open the same file once each, so a cached file was read from
disk twice per scan on top of the detector's own read.