MediaFilter

Maximum file size for processing (default 10MB). Specification documents have their own budget — see MAX_SPEC_FILE_SIZE below. Can be overridden with the environment variable NOIR_MAX_FILE_SIZE. Supported formats for NOIR_MAX_FILE_SIZE:

• Plain bytes integer (e.g., 5242880)
• Human-readable with unit suffix (K, KB, M, MB, G, GB) e.g., 5MB, 500K, 1G Invalid / unparsable values fall back to the default (10MB).

Size budget for specification documents, overridable with NOIR_MAX_SPEC_FILE_SIZE (same value syntax as NOIR_MAX_FILE_SIZE).

MAX_FILE_SIZE exists to keep images, archives and compiled blobs out of the scan, and 10MB is a sensible ceiling for that. Applying it to API descriptions was collateral damage: those are plain text, they are the highest-value input noir has, and generated ones routinely pass 10MB. NetBox ships a 12.35MB contrib/openapi.json describing 308 paths, and every one of them was dropped — not because the document was unreadable, but because a filter aimed at image files measured it.

The budget still has to be bounded, because the document is read whole, held in the content cache, and parsed into a JSON/YAML tree that peaks at roughly 5x its size in memory. Measured with a release build: NetBox's 12.35MB document costs 0.18s and 81MB peak RSS for the whole scan; a 76MB variant of it (same document, component schemas multiplied, forced through with the override) costs 0.90s and 392MB. 64MB is the ceiling because it keeps one document under a second and a few hundred MB while leaving 5x headroom over the largest generated spec observed in the wild. Past that a .json is a data dump, not an API description.

Floored at MAX_FILE_SIZE so that raising the general cap past 64MB lifts specification documents with it rather than capping them below everything else. Lowering the general cap does not lower this budget; use NOIR_MAX_SPEC_FILE_SIZE for that.

O(1) lookup set materialized from MEDIA_EXTENSIONS. Used on the hot path — every file in the project is checked once.

Common media file extensions that should be skipped

Extensions that carry an API description or a structured document noir reads as one. These get MAX_SPEC_FILE_SIZE instead of the media cap.

The list mirrors the file gates of src/detector/detectors/specification/* (grep -h "detector_for" src/detector/detectors/specification/*.cr) with two deliberate omissions. Binary container formats get no relief: mitmproxy's .flow dumps are NUL-bearing blobs the walk drops on content regardless of size. Neither do the source extensions the AWS CDK detector reads (.ts, .js, .py, …) — a 12MB .js or .py is a bundled or generated artifact, where a 12MB .json is usually exactly the document the scan is looking for.

The list is by extension rather than by content because a document's own marker ("openapi", "swagger") can sit anywhere in it, and a prefix sniff that misses one is a silent loss again. The cost of being wrong the other way is small: a 32MB package-lock.json matches no specification marker, so the walk reads it and drops every spec detector without parsing — 0.08s more than skipping it outright.

Cheap binary-content sniff. Reads the first 512 bytes and returns true if the buffer contains a NUL byte (\x00), which is the canonical "this is binary, not text" marker — text files in any common encoding (UTF-8, UTF-16-with-BOM, Latin-1, etc.) don't contain interior NULs in practice.

Byte budget read from name, falling back to default when the variable is unset, unparsable or non-positive.

Split out of the constant initializers so both budgets resolve the same way and a spec can exercise the parsing without reloading the module — the constants read ENV once, at first use.

Check if a file is too large to process. max_size defaults to the budget for the file's own kind — pass one only to override it.

The size budget that applies to file_path.

Check if a file should be skipped based on extension

Parse size strings like "10MB", "500K", "1G" or raw bytes ("1048576")

Combined check - returns true if file should be skipped. Prefer {skip_check} on hot paths: it returns the reason in the same call so the caller does not re-stat to log.

Decide whether a file should be skipped and, if so, return the human readable reason in a single pass — avoids re-stat'ing the file just to compose the log message. Returns nil when the file should be processed.

When the caller has already obtained a File::Info (e.g. the detector walker stats each entry with follow_symlinks: false), it can be passed as info to skip the size stat entirely.

max_size defaults to the budget for the file's own kind, so the reported ceiling is the one that was actually applied.

Get a human-readable reason why a file was skipped. Kept for backwards compatibility; new callers should use {skip_check}.

True when the path names a specification document (see SPEC_DOCUMENT_EXTENSIONS), which is read under the larger budget.