Krikri::AptLockRetry

The two reported cmd strings above (apt.py resolves the absolute /usr/bin/apt-get path for its fail_json, this engine runs the bare name through PATH), plus the warning apt.py emits BEFORE attempting the auto-install - module.warn("Updating cache and auto-installing missing dependency: python3-apt") - which shows up as the final "[WARNING]: ..." line after the PLAY RECAP when a stage fails.

After a signature-failed attempt, apt marks the repo untrusted and every LATER python-apt update failure prepends these two diagnostic lines to the exception text (verified live: attempt 0 = GPG+E: lines only, attempts 1+ = the two secure-repo lines then the same GPG+E: lines, all ", "-joined). Emitted in python-apt's own marker form (no space after "W:").

Ansible's apt module cannot run at all in check mode when it can't see the python3-apt bindings: its auto-install fallback (apt_auto_install_python_apt above) is a real, PERSISTENT host mutation, and check mode must never mutate the target - so the module fails fast with this exact message instead. Shared between apt.cr's own update-cache block and package.cr's cache-refresh-only path (real package: delegates to the apt module on apt hosts, so it refuses identically) rather than duplicating the literal in two places and letting them drift.

Default for install/remove/upgrade operations - matches Ansible's apt module default.

Defaults for apt-get update - match Ansible's apt module defaults exactly.

Builds the PluginResult a failed auto-install stage reports, shared by apt.cr's and package.cr's three call sites so the shape can't drift. Shaped like Ansible's own fail_json for a check_rc=True run_command failure (issue #522, observed on the update stage): msg is the stderr with its trailing newline stripped - NO "Failed to auto-install" prefix - the keys dump in fail_json's sorted order, and there is NO cache_updated key (the module never got to touch the cache). The result.extra warnings carry the module's pre-failure warning so the final "[WARNING]: Updating cache and auto-installing missing dependency: python3-apt" line renders like real ansible's.

Emulates Ansible's apt module module-start auto-install of the python3-apt bindings (apt.py's probe_interpreters_for_module + "Updating cache and auto-installing missing dependency" path): a real, PERSISTENT host mutation that changes every later apt invocation's changed-reporting path. When the bindings are missing this runs the same apt-get update prefetch (skipped only when the task explicitly passed update_cache: false, per apt.py's own if module.params.get('update_cache') is False guard), then apt-get install -y python3-apt, exactly the two commands Ansible runs before respawning itself. Returns nil when the bindings were already present (a no-op) or the install succeeded; returns the failed stage's raw command result plus its reported cmd string when either command failed, mirroring Ansible's check_rc=True hard failure.

This engine previously only emulated the FIRST invocation's observable changed behavior (the round-30001 rule) without ever performing the install, so a host that started without the bindings stayed on the "absent → changed=false" cache-refresh path forever, while Ansible moved to the mtime-diff path after its very first apt task. Found via geerlingguy.kubernetes (rounds 65166/65311): the role's "Ensure dependencies are installed." task triggers Ansible's auto-install, so its later "Update Apt cache." task (immediately after deb822_repository added the pkgs.k8s.io repo, whose freshly-fetched indexes genuinely move the lists mtime) reported changed=true there, while this engine stayed on the absent path and reported ok.

force_absent is a spec seam only (never set by real playbooks): it short-circuits the live probe so a spec running on a dev box that really has the bindings can still drive the absent path through the plugin's own call site.

A failed stage returns the raw command result PLUS the cmd string Ansible's own fail_json carries for that stage (issue #522): apt.py resolves apt-get via module.get_bin_path(..., required=True), so the reported cmd is the absolute /usr/bin/apt-get path even though this engine executes the bare name through PATH - the reported string only, what actually runs on the host is unchanged.

Same mtime probe Ansible's get_cache_mtime()/ get_updated_cache_time() use: the update-success-stamp if present, else the /var/lib/apt/lists directory's own mtime. Shared between apt.cr's own before/after cache-update comparison and this module's own should_update_cache? (see there).

Ansible's changed semantics for a cache-refresh-ONLY apt invocation (no name:/upgrade:/deb: alongside it): WITHOUT python3-apt, Ansible auto-installs it before its own measurement window opens (that auto-install runs a full apt-get update first, then respawns and reads mtime entirely AFTER the prefetch) - so it always reports changed: false here regardless of whether anything was actually fetched. WITH python3-apt, it reports changed: true only if the cache mtime genuinely moved. See apt.cr's own #execute for the live verification this mirrors (round 30001) - shared here so package:'s own cache-refresh path (update_cache: true with no name:) doesn't duplicate-and-drift from this logic the way it previously did (found via robertdebock.update_package_cache: this engine hardcoded changed: true for apt unconditionally, Ansible's ok/changed: false on an already-fresh mirror).

Same freshness gate apt.cr's own should_update_cache? implements (see there for the full real-Ansible apt.py rationale): with the default cache_valid_time: 0 the refresh always runs; with a positive cache_valid_time it only runs when the update-success- stamp/lists-dir mtime is older than the validity window.

Returns the refusal message above when a check-mode run must fail because the bindings are absent, else nil (not check mode, or the bindings are already there and nothing needs installing).

Detects the CLI-observable signature of a genuinely corrupt/ unparseable on-disk package index - NOT a plain "package doesn't exist in an otherwise-valid cache" miss.

Ansible's apt module (python-apt-backed) only retries when apt.Cache() itself raises a SystemError whose message mentions /var/lib/apt/lists/ (get_cache() in ansible's apt.py) - that is specifically a cache open/parse failure (corrupt or unreadable index files), not "no candidate for this name". A simple locate-miss on a valid-but-empty or valid-but-outdated cache does NOT trigger it: package_status() fails straight to fail_json("No package matching '%s' is available") with no retry at all - confirmed live (0.9.737) against a genuinely empty /var/lib/apt/lists/, where ansible-playbook failed outright on package: {name: w3m, state: present} with that exact message and krikri (this helper's previous, over-broad "Unable to locate package" gate) silently installed it instead - a real divergence the previous gate introduced rather than fixed.

The corrupt-lists SystemError's apt-get-CLI equivalent was confirmed live by corrupting a downloaded index file (an unreadable .lz4 list) and reproducing the exact python-apt exception text via both apt.Cache() directly and apt-get install: E: The package lists or status file could not be parsed or opened. - genuinely different from, and much narrower than, "Unable to locate package".

python-apt's Cache.update() - what real apt.py's update_cache pass actually runs - raises a bare FetchFailedException() the moment ANY acquire item failed, while the CLI apt-get update this plugin shells out to treats that same partial failure as WARNING-level and still exits 0 (verified on jammy's apt 2.4: a sources.list.d entry pointing at an unresolvable host prints "W: Failed to fetch ..." / "W: Some index files failed to download." and exits 0 - even when it is the only configured source). Trusting the exit code alone made update_cache: true against a bogus repo report success where Ansible fails after its retries (round 1100002 kop_apt_fail apt_fail_bogus_source). The CLI-observable equivalent of python-apt's "res == failed" check is the failed-item markers apt itself prints: an Err: line per failed item plus the W: summary. python-apt raises ONLY on a failed item - the E:/Err: markers; a warning-only line (a standalone "W: GPG error ... NO_PUBKEY ..." with no failed item) exits 0 and python-apt raises nothing, so the update succeeds just as real ansible reports it changed.

apt.py's FetchFailedException retry loop verbatim: for retry in range(update_cache_retries) around cache.update(), exponential backoff (2**retry plus a random sub-second jitter, capped at update_cache_retry_max_delay plus jitter), one warn pair per failed attempt (the fail warn before the sleep, the sleeping warn after it). python-apt raises FetchFailedException() BARE - str() is the empty string - so both the warning's "due to <e>" and the final msg's reason come out EMPTY (round 1100002's py capture: "due to , retrying" / "Failed to update apt cache after 5 retries: "); the 'unknown reason' fallback in apt.py's msg only fires when no attempt ever raised (update_cache_retries=0 leaves err == ''). The caller only enters here after an attempt already failed, so iteration 0's "after 1 retries" warn refers to that attempt - and quotes ITS exception text when the failure was the SystemError- wrapped signature class (see apt_fetch_failure_reason). Each later failed attempt warns with its own text, and once a repo has been refused authentication apt prepends the two untrusted-repo lines to every later attempt's text (APT_FETCH_SECURE_REPO_PREPEND). Returns recovered: true as soon as an attempt succeeds, with the warnings emitted so far (they persist in the registered result exactly like module.warn() texts do in Ansible) and last_reason - the final attempt's exception text, which apt.py's final msg quotes ("Failed to update apt cache after N retries: <reason>", empty for the bare-exception class).

The str() text of python-apt's SystemError-wrapped FetchFailedException: the W:/E: diagnostic lines of that update attempt, with the space after the letter marker stripped (python-apt reports "W:GPG error ...", the CLI prints "W: GPG error ..." - verified live in a jammy container: attempt 0's exception text was exactly the two CLI W:/E: lines so transformed), joined with ", ". Empty for the bare-exception class (plain fetch failures), whose warnings and final msg stay empty - the round-1100002-verified shape.

With SEVERAL failing repositories python-apt does not join their diagnostic blocks bare: each block ("W:GPG error ..." + "E:The repository ... is not signed.") is separated from the next by its two untrusted-repo lines ("W:Updating from such a repository ...", "W:See apt-secure(8) ..."), probed live in a GPG-broken trixie container - a three-repo first attempt's exception text was "W:GPG<r1>, E:<r1>, W:Updating..., W:See..., W:GPG<r2>, E:<r2>, W:Updating..., W:See..., W:GPG<r3>, E:<r3>" while the CLI stderr carries the same W:/E: lines with NO untrusted-repo lines at all. The CLI gives no way to tell those lines apart, so re-insert the pair at every GPG block start that follows a REFUSED block - one whose diagnostics carried the "E: The repository ..." refusal line (issue #408: real ansible's retry warnings quote one pair per refused repo, krikri used to quote none until its single from-attempt-2 prepend). Tolerated GPG blocks - a [trusted=yes] repo's "W: GPG error ..." warning carries no refusal line - get NO pair between them (live-verified in-process on jammy 22.04 with two such repos plus a 404: the two GPG lines sit directly adjacent in every attempt's text), so keying the insertion on the refusal line keeps those shapes unprefixed exactly as real ansible quotes them.

One failed attempt's reason bookkeeping, shared by the apt and apt_repository update-cache retry loops: returns the attempt's warning/final-msg text paired with the signature_failed flag for the NEXT attempt. The prepend decision uses the flag as it stood BEFORE this attempt - apt only marks a repo untrusted after its first failed update, so the very first failed attempt's text is quoted bare (verified against ansible-core 2.19.11: attempt 1's warn carries only the GPG/E: lines, the two untrusted-repo lines appear from attempt 2 on). Encapsulated here so both loops structurally cannot compose the text from a flag that already includes the current attempt - the apt_repository loop used to do exactly that and warned with the prepend from its very first failure.

The flag turns on for a REFUSED repo only. apt marks a repo unauthenticated exactly when it refuses the repo's metadata with a top-level "E: The repository ..." line ("is not signed", "does not have a Release file" - both live-verified in-process to gain the pair from the next update on); the tolerated failure shapes never mark anything - a [trusted=yes] repo's standalone "W: GPG error ... NO_PUBKEY ..." warning, the allow-insecure "W: ... potentially dangerous" lines, and plain index 404s keep the identical unprefixed reason on every attempt (all live-verified in-process, jammy 22.04 apt 2.4.14). That tolerated shape is exactly issue #665's real round (xe0nic.ansible_vprotect_server): both of the role's repos are deb [trusted=yes], so real ansible warned with the same unprefixed mixed GPG-warning + 404 text five times and failed with it unprefixed, while the previous flag trigger (any non-empty reason) prefixed the later attempts and the final msg. The marking is in-process state: a separate apt-get update CLI process never prints the pair (live-verified, three consecutive CLI runs), so the prepend stays a synthesis and stays CLI-safe. The reason text arrives marker-space-stripped from apt_fetch_failure_reason ("E: The" -> "E:The"), which is why the probe has no space.

The SECOND FetchFailedException shape python-apt raises: raise FetchFailedException(e) around the SystemError apt_pkg throws for repository-level errors, which the CLI surfaces as a top-level "E:" diagnostic line ("E: The repository ... is not signed."). python-apt raises only on a failed item, and a top-level E: line is exactly how a failed item prints, so the E: form is the whole detection. Every warning-only shape exits 0 and must succeed, the same changed outcome real ansible reports: a standalone "W: GPG error ... NO_PUBKEY ..." line raises nothing, and the W: form of the unsigned-repo message (apt's allow-insecure path, "W: The repository ... is not signed.") likewise never marks the update failed. Unlike the bare raise FetchFailedException() a plain network fetch failure takes, this shape CARRIES the diagnostic lines as its str() text, so the retry warnings and the final failure message quote them (round 5210000, artem_shestakov.nginx: real ansible-core 2.19.11 warned five times with the GPG/E: text and failed with "Failed to update apt cache after 5 retries: W:... , E:..." while krikri classified the case as a plain nonzero-exit failure and failed fast).

Matches Ansible's apt module's update_cache_retries + update_cache_retry_max_delay on apt-get update. Exponential backoff starting at 1s, doubled each attempt, capped at retry_max_delay. Only retries on lock contention - other apt-get update failures (broken repo, network) fail-fast, matching Ansible's selective-retry behavior.

Ansible's apt module silently recovers from an install failure caused by a corrupt/unparseable on-disk package index: get_cache() catches the apt.Cache() SystemError and retries apt-get update (up to twice) before re-opening the cache. Only retries ONCE here, and only on the corrupt-lists stderr pattern - any other failure (broken repo, signature mismatch, a plain locate-miss on a valid cache, a held lock that outlives lock_timeout) fails fast exactly as before. Returns the retry result when the refresh helped, else the original failure.

Detects the dpkg/apt lock contention stderr patterns that apt-get itself emits. Matches Ansible's python-apt-based wait/retry detection (which checks the same three patterns on OSError from apt's cache_lock/system_lock). Conservative: any other stderr fails fast, even if it's "lock-related", so bogus lock messages don't trigger an infinite retry.

Can the target's Python see the python3-apt bindings? Same two interpreters Ansible's apt module probes (probe_interpreters_for_module(['/usr/bin/python3', '/usr/bin/python'], 'apt')) before deciding whether to auto-install python3-apt and respawn under an interpreter that can see it - see apt.cr's own #execute for the long comment on which changed-reporting path that decides.

Ansible's apt.py main() runs the cache refresh BEFORE any install/remove/upgrade whenever update_cache: is set (or any cache_valid_time: is) - unconditionally with the default cache_valid_time: 0 (only the stamp-mtime + cache_valid_time < now staleness gate can skip it), and even when every requested package is already installed. Returns nil when no refresh is needed (check mode - apt.py's own if not module.check_mode: cache.update() guard skips it; no relevant params; still-fresh cache), else runs apt-get update through apt_get_update_with_retry with Ansible's update_cache_retries/update_cache_retry_max_delay semantics, returning the failed result so the caller can fail the task BEFORE touching any package state.

Matches Ansible's apt module's lock_timeout retry behavior on install/remove/upgrade operations. Only retries when stderr indicates dpkg lock contention - other failures (broken repo, missing package, signature mismatch) fail-fast on the first attempt, matching Ansible's selective-retry behavior.

Sleeps in 3-second increments between attempts - bounds total controller-fiber blocking time across multiple retries, well within the user-visible lock_timeout. Returns the last lock-holding error when the budget is exhausted, exactly the way Ansible's apt module does.