Krikri::SSHManager

Default per-command execution timeout for #exec/#exec_script/ #daemon_send. Ansible has NO default command-duration limit at all - a foreground task runs until it completes, however long that takes (only async: tasks get an explicit max duration); what actually detects a genuinely dead/unreachable host is the SSH connection's own keepalive (ServerAliveInterval=60 x ServerAliveCountMax=3 below, ~180s to a hard disconnect), which this timeout duplicates and undercuts. Previously 300s (5 minutes)

• too short for entirely ordinary, legitimately slow real-world tasks: found live re-benchmarking buluma.netdata (round 163 regression check) - its own installer genuinely compiles from source and took a confirmed 1536s (~25.6 minutes) on the ansible-playbook side; crystal's identical task was killed at exactly 300s ("SSH command timed out... did not exit even after being killed") despite the remote command still actively running and eventually would have succeeded. The same 300s cap likely also explains 2 earlier "flaky"-looking incidents this session (robertdebock.luks, geerlingguy.java - both ordinary dnf install calls that occasionally took a bit over 300s on a slow mirror/cold host, not a genuine hang) that were chased down as one-off infra flakiness via isolated re-tests rather than recognized as this same root cause at the time. Raised to a generous but still-bounded 1 hour - long enough for realistic compile-from-source/large- package-install tasks, while the SSH keepalive above still catches a truly dead connection in ~3 minutes regardless of this value.

A frame larger than this is corrupt/hostile input (e.g. a MITM'd or compromised daemon sending a garbage length prefix) - refuse to allocate for it rather than OOM'ing the controller on a 4 GiB Bytes.new. Real payloads (plugin result JSON) are nowhere near this.

The extra ssh option the askpass mechanism needs on every ssh/scp invocation - the same one ansible-core adds under its ssh_askpass password_mechanism - bounding the password prompts to one so a wrong password fails fast instead of cycling through the helper.

Spec seam - the registry is process-global, so tests clear it rather than leaking passwords (or the triple keys) between specs.

Close all connections

Graceful shutdown for every still-open daemon connection, called once at the end of a run. Closes every daemon's stdin first (each one sees EOF and exits cleanly via plugin_daemon.cr's own rescue IO::EOFError - no explicit "goodbye" message needed) and returns without waiting for the ssh clients to exit.

Close specific connection

Exception counterpart of #connection_level_failure? - an scp/rsync/ ssh failure surfaces as a raised "Failed to upload/download ..." whose detail embeds ssh's own stderr, so the same pattern list decides whether the exception means "the transport never got there" (UNREACHABLE in Ansible) or something else that must keep propagating (a missing local file, a staging-dir safety refusal, an engine bug). Message-only, no exit code to gate on.

True when exit_code/stderr look like the SSH transport itself failing - ssh never reached or never authenticated to the host - as opposed to a remote plugin crashing after a successful connection. stdout (optional, empty by default for the message-only exception counterpart) feeds the silent-death rule: bare 255 with nothing in either stream is ssh's own mid-execution death, never a remote plugin's. Public so PluginManager's remote-result interpretation and the spec suite share one pattern list (see CONNECTION_FAILURE_PATTERNS for what deliberately does NOT match).

The connection-options fingerprint a ControlMaster socket (and a resident daemon, which owns one) is keyed on: identity file, ssh extra args, and whether a password is in play. Real ansible-playbook hashes its ssh connection options into its own ControlPath for the same reason: two tasks connecting to the same (host, user, port) with different options must not silently share one authenticated master - a task-level -o BatchMode=yes or a different private key would otherwise be swallowed by a socket established under the previous task's options.

Read-only accessor for diagnostics/specs - the stable control-socket directory itself carries no secrets, and the spec suite pins its pid-independence (the exact property f500.ufw's warm-rerun lockout hinged on).

become_user is interpolated into a shell command line here, same as the one-shot path's own target string - it is expected to have already passed PluginManager.valid_become_user? at the call site, which is where that allow-list is enforced for both paths. Quoting here is defense in depth: if the allow-list is ever relaxed or bypassed, the shell still sees one argument, not injected command text.

sudo_needs_password is the probe result from #spawn_daemon (and deliberately NOT the password itself - a password must never land in this string, which travels as an argv element of the local ssh process AND of the remote shell's own bash -c). With it set, sudo runs sudo -H -S -p '' (read the password from stdin, empty prompt so nothing is written to the daemon channel's stderr) and #spawn_daemon has already put the password line on the stdin pipe ahead of the frames. Public as a spec seam: the -n/-S command construction is pinned directly in the unit tests.

term is the controller's TERM, forwarded only for become daemons. Ansible's ssh connection runs every become task under ssh -tt (use_tty defaults to true), so the remote sudo and module inherit the controller's TERM. A script that colours its output via tput emits ANSI codes there and plain text without TERM (issue #606). Non-become daemons get no pty in ansible, so they get no TERM here.

Sends one request and returns the plugin's own JSON result, unwrapped - unlike exec/exec_script, there is no exit-code-vs- stdout arbitration to do here (that was a one-shot-process concept; interpret_remote_result doesn't apply to a persistent pipe), the daemon's response IS the plugin's real output.

Failure classification is deliberately split by HOW FAR the request got, because the caller's fallback re-EXECUTES the module:

• spawn/write failures mean the frame never went out in full, so nothing dispatched and a fresh run cannot double-execute. The exception propagates raw (this is also the entire reconnect story: a stale daemon - e.g. after ansible.builtin.reboot killed the SSH session mid-play - fails exactly once as a broken pipe on the write, falls back safely for that one task, and a fresh daemon gets lazily spawned next time).
• a lost/unusable RESPONSE on a daemon this call spawned itself is kept in the same class: a never-proven daemon (refused connection, missing remote binary, failed sudo) fails at read time exactly like one that died mid-dispatch, and "never came up" is the common case - the caller's fallback stays safe.
• a lost/unusable response on an ALREADY-CACHED daemon raises DaemonDispatchUnknownError: a successful write proves the remote process was alive and reading at that moment, so "it ran and we can't tell" is real, and re-executing could double-apply the module. The caller fails the task instead.

Perf item 3: send a whole batch of steps as ONE daemon request. Same connection, same framing and the same rescue-and-let-the-caller-fall-back contract as #daemon_send above; only the payload shape differs.

Returns the plugin results keyed by the step's index in steps. An index that is ABSENT never ran - the daemon stopped at an earlier failing step, exactly as BatchScript.parse reports a script that halted. Callers must honour that the same way they already do for the script transport.

Every step here must agree on become_user, because that is what selects which resident daemon (and therefore which user) serves them - see TaskExecutor#run_batch_steps, which does the partitioning.

Whether a daemon for this key is worth attempting at all. Public so PluginManager can skip the attempt before building a request, rather than learning about it from a raised exception.

Download file from remote host via SCP

Execute command on remote host port nil means "let ssh resolve it" (no -p flag; ~/.ssh/config and /etc/ssh/ssh_config apply) - only an explicit port overrides.

Runs script on the remote host via ssh ... bash -s, piped over that single invocation's own stdin - used by TaskExecutor's batch path (batching, on by default; --no-batching disables it) to run several plugin invocations in one SSH round trip. Deliberately separate from exec: exec wraps a command through bash -c <string>, which has no stdin of its own to carry a whole script through; this uses bash -s (reads its script from stdin) specifically so the caller can hand over a multi-step script without needing to escape it into a single command-line string.

ansible-core's ssh connection plugin adds these whenever NO connection password is configured (ssh.py's "ansible_password/ansible_ssh_password not set" block): without them ssh falls through to keyboard-interactive and password prompts on an auth failure, and the askpass noise lines ("ssh_askpass: exec ...", "Permission denied, please try again.") pollute the unreachable stderr that ansible's fatal msg embeds verbatim (issue #887: ansible's rejected-user stderr is just the known-hosts warning plus the user@host: Permission denied line). With a password configured sshpass/askpass handle the prompt the way ansible's password path does, so the options stay off there.

Resolves the connection password for one target. A nil password is the only no-op (never erases a password registered elsewhere for the same triple); a found password is delivered by whichever mechanism the spawn sites pick - sshpass -e when sshpass is available, the askpass env overlay otherwise (see #sshpass_prefix and #sshpass_env).

A registered connection password is delivered to ssh by one of two mechanisms, mirroring ansible-core 2.19's connection plugin (whose default password_mechanism is ssh_askpass): sshpass -e when the sshpass program is installed, and OpenSSH's own SSH_ASKPASS mechanism when it is not. Registration therefore never raises - a password without sshpass is not an error (ansible-core succeeds in exactly this situation), the askpass fallback covers it.

meta: reset_connection - drops this host's persistent connection state so the next task opens fresh connections. Two layers exist here: the resident plugin daemons (one ssh process per host/user/port/become_user) and ssh's own ControlMaster socket underneath them. Both are dropped for every key matching the host; the ssh -O exit is best-effort (no master socket = no-op).

Reset statistics

The -e value both rsync paths in this file build - shared so the nil-port omission (no -p; ssh's own config resolution applies) stays identical in both. With a password and sshpass installed the ssh it names is itself wrapped in sshpass -e (rsync word-splits this string into argv itself, so no shell quoting is involved and the password still only ever travels in SSHPASS - which rsync passes down to the sshpass child from the env this file's Process.run spawned rsync with). Without sshpass the ssh stays unwrapped and inherits the askpass overlay env from rsync, plus the same single-prompt bound the other builders carry. Public as a spec seam.

Upload file using rsync (more efficient for incremental updates) Returns true if successful, false if rsync not available or failed

The argv for a batched plugin upload. --no-o/--no-g countermand -a's -o/-g: when the connecting user is root, rsync would otherwise recreate the CONTROLLER's uid/gid on the target, and that pair usually has no passwd/group entry there (issue #729: guidugli.fspart's find -nouser/-nogroup scans then reported krikri's own staged plugin binaries as unowned/ungrouped files, where the same role run under ansible-playbook reported nothing - ansible's module payloads go over SFTP, which can only ever be owned by the connecting user). Kept separate from the Process.run so command construction is testable without spawning rsync.

Batch upload multiple files using rsync (most efficient) Returns true if successful, false if rsync not available or failed

Runs block (given the just-spawned process) on a separate fiber and bounds it to timeout_seconds wall-clock time - both exec and exec_script accepted a timeout: parameter that was never actually enforced anywhere, leaving every blocking Process call (#wait, and for exec_script, the write of a potentially large script into the process's own stdin pipe) able to hang forever with no escape hatch at all. -o ServerAliveInterval=/ ServerAliveCountMax= only bound an established SSH session going idle - they do nothing for a local pipe write blocking because nothing is reading the other end (the remote host went unreachable mid-handshake, before ssh itself even started forwarding stdin), which is a local blocking syscall the SSH protocol's own keepalive machinery never sees. Found running konstruktoid-hardening's newly-batched (0.9.155) 411-item "Find possible suid binaries" step: the whole krikri-playbook process sat silent, producing no further output and no error, until an external timeout wrapper eventually killed it - a real host/ network hang, not a task bug, and previously nothing inside krikri-playbook itself could ever recover from it.

On timeout, SIGKILLs the process and gives it a further 5s to actually exit and report a status before giving up entirely (a hard bound of its own, in case even the kill signal doesn't unblock the fiber - e.g. if it's wedged in kernel-level uninter- ruptible I/O, vanishingly rare but not impossible).

Fiber lifecycle on the timeout path (audited, not assumed): the SIGKILL guarantees the child dies, which unblocks the spawned fiber's #wait (or errors its stdin pipe write with EPIPE), so the fiber completes and its result lands in the capacity-1 channel - either received within the 5s grace (the killed process's real exit status is then returned, e.g. 137) or, if even that doesn't arrive in time, discarded when this method returns and the channel is garbage. Either way the fiber terminates on its own; nothing leaks permanently.

Public (not private) so the timeout semantics can be spec'd directly against a real local process - see test/unit/ssh_manager_timeout_test.cr.

Same extraction for the scp argv (upload and download share every option; they differ only in the -r flag, the ConnectTimeout option - download has never carried it - and their trailing path pair, hence recursive/connect_timeout/local_args). Public as a spec seam.

Spec seams: install/remove a Process as the cached daemon for a key, so #daemon_send's dispatch-vs-response-lost classification can be tested against a LOCAL fake daemon process with no SSH in the loop (a fresh spawn would dial a real host).

The full argv every ssh invocation in this file spawns (exec, exec_script and spawn_daemon all share the exact same option set) - extracted so the construction is one definition instead of three, and so the unit tests can pin it (including the sshpass prefix) without ever spawning a real ssh. remote_args is what follows user@host: the wrapped command, bash -s, or the daemon command. Public as a spec seam.

Spec seam - the availability probe shells out to which, so tests that need the "sshpass missing" path (or need it NOT to fire on a machine that happens to lack sshpass) pin the result directly.

argv prefix for ssh/scp when a password is set. With sshpass installed the password travels only in SSHPASS (sshpass -e reads it from its own environment) - never on argv, where any local user's ps could read it. Without sshpass the argv stays unwrapped and the askpass overlay in #sshpass_env carries the password instead (also env-only).

Get connection pool statistics

Upload file to remote host via SCP