Drivers¶
reemote.drivers is the execution engine. run is
the entry point used by deploy scripts and examples; the remaining functions
are exposed so the two phases can be driven, inspected and rendered
separately.
The command-execution path is asynchronous:
send_command is a coroutine that runs each
command over asyncssh (or forwards to a SimulatedHost in tests), and
therefore run_apply and
run are coroutines as well. Fact gathering is
asynchronous for the same reason — a fact's process runs over asyncssh — so
materialize and
run_prepare are coroutines too. The
formatters are pure local rendering and stay synchronous.
Fact gathering is per-host by default — FactCache.get
reads one fact for one host. gather_fact is
the driver-level fan-out across the whole inventory: it runs a fact's process
for every host concurrently with asyncio.gather and returns a host-aligned
mapping of host.name to that host's value (mirroring the baseline's get_facts,
which returns {host: value}). Facts stay single-host — only the driver owns
the concurrency — so process, FactCache and the SimulatedHost seams are
untouched.
A request whose doas/sudo/su tri-state resolves to elevation
(Privilege.ON, i.e. sudo=True; see
Deploy scripts) is elevated by the driver, which
collapses the tri-state to a concrete bool before sending.
command_shell renders the shell text for a
command descriptor — prefixing doas -n (or doas -n -u <user>) and sudo
(or sudo -u <user>), and wrapping su <user> -c …, combining them in the
fixed order doas then sudo then su (doas is the outermost escalation, su
the innermost payload). The run-as users come from the
host's inventory privilege section, never from the operation; root, being
the default target of all three, collapses to a bare doas -n / sudo / su -c.
This rendering is what format_log shows and is
deliberately credential-free: when privilege carries a password,
send_command elevates through
Host.run_shell_privileged, which
runs sudo -A against a generated askpass helper (or pipes a su_password to
su's stdin) so the password is supplied without a TTY and never reaches the
log (doas is passwordless, so it carries no credential at all). On the real
transport an applied command that exits non-zero raises
CommandError (fail-fast) instead of being
silently logged as applied — unless doas, sudo or su reports the failure as its
own authentication rejection (a wrong or missing password, or a denied doas
rule), which raises
PrivilegeError so a rejected credential is
reported as a privilege problem rather than a broken command. See
Privilege: sudo, su and doas.
A request whose when guard fails on a host is skipped by the driver in both
phases: prepare records a skipped
PlannedOperation without materializing the
operation at all (no facts are read for it — the guard may exist precisely
because the host cannot provide them), and apply logs a skipped
LogEntry without sending anything. The request
still consumes its lockstep slot, so the shared schedule stays aligned across
hosts; format_plan and
format_log render it as
[skipped: when guard <predicate name>] — the reason names the failing
predicate, not the request — a third outcome beside changed and
no change. See Deploy scripts for the guard's
static-data and stable-predicate rules.
Phase awareness¶
Only the drivers know which phase they are running in. Facts, operations and
command descriptors are phase-unaware: the driver expresses the difference
between prepare and apply entirely through the lifetime of the
FactCache it hands out, and through whether it
forwards a command to send_command.
deploy(host) generator
│ yield OperationRequest
▼
run_prepare ──► materialize() ──► (changed, reason, commands) [plan / dry run]
run_apply ──► materialize() ──► await send_command() per cmd [host mutated]
This invariant also powers the in-deploy guard for actions.
run_prepare and run_apply are wrapped by _deploy_phase, which sets a
ContextVar for their duration (propagating into the per-host tasks
asyncio.gather spawns) and resets it cleanly even when a phase raises. Only
the drivers toggle it, so facts, operations and actions stay phase-unaware;
gather_action merely reads the flag and
raises ActionInDeployError at the call
site if an action is attempted inside a deploy. The guard is reinforced from
the other side: materialize requires an
async-generator operation, so wrapping an action class as op(Shell, ...) is
refused with OperationError rather than being
run as a deploy step.
The same two invocation points enforce the yield requests rule. Both
run_prepare (through _prepare_host) and the
lockstep pull in run_apply type-check every item
the deploy body hands over and tally the requests it builds, so a body that
returns no request stream (an op(...) call with no yield anywhere), yields
something that is not an
OperationRequest, or builds a request and
drops it raises DeployRequestError — in a
normal run always during prepare, before a command is sent, and with the hosts of
an apply step aggregated into one message. The tally lives on each
HostRuntime, so it is per host and per phase, and
compose(..., dedupe=True) charges its own collapses
so a deliberate duplicate is not mistaken for a lost request.
Concurrency across hosts¶
Hosts are independent — each owns its own connection, deploy generator and
FactCache — so the driver issues the per-host work
for a phase or lockstep step concurrently with asyncio.gather, then joins:
run_preparegathers each host's whole prepare (collect its requests, materialize the plan).run_applygathers step 2 (materialize +send_command) for every active host; the request pull and the determinism check stay synchronous and complete before anything is sent.
The gather join is the barrier that preserves cross-host lockstep ordering,
and because asyncio.gather returns results in input order, the plans and the
run log stay deterministic regardless of the order the hosts finish. Commands
within a single host still run sequentially, since some command sequences are
order-dependent. gather_fact applies the same
fan-out to a one-off fact read outside the two-phase flow.
Both gathers also report failures the same way, since concurrency means several
hosts can fail at once. run_prepare collects the per-host
HostConnectionError instances of an
inventory, and run_apply the per-host
PrivilegeError instances of a step, each
merging them into a single aggregate naming every failed host instead of only
the first to lose the race. gather_fact gives
the same treatment to
CommandNotFoundError: every host
missing the command the fact runs is named in one aggregate. Any other
exception propagates unchanged, so a CommandError or a genuine bug still
aborts the run.
Reference¶
Drivers (design doc section 8).
The prepare phase collects the request sequence per host and materializes
each operation to compute (changed, reason) for plan/--dry output.
The apply phase advances one generator per host in lockstep, recreating
each host's FactCache at every step so generator-body fact reads see
the effects of prior operations. Operations and facts never inspect a
phase flag; only the driver changes behavior between phases.
Within each phase the per-host work is independent (each host owns its own
connection, generator and FactCache), so it is issued concurrently across
hosts with asyncio.gather: run_prepare gathers each host's whole
prepare, and run_apply gathers step 2 (materialize + send) for all active
hosts. The gather join is the barrier that preserves cross-host lockstep
ordering, and asyncio.gather returns results in input order so plans and
the run log stay deterministic. Commands within a single host still run
sequentially, since some command sequences are order-dependent.
Command execution is asynchronous: send_command (and therefore
run_apply and run) are coroutines, preparing the apply path for a
real asyncssh-based transport. Fact gathering is asynchronous for the same
reason: a fact's process will run over asyncssh, so FactCache.get,
the operations that await it, and materialize are coroutines too. The
whole prepare/apply path is therefore awaited end to end.
dataclass
¶
PlannedOperation(
request: OperationRequest,
changed: bool,
reason: str,
commands: tuple[Command, ...],
skipped: bool = False,
)
One materialized request in a host's plan.
A skipped entry was not materialized at all: the request's when
guard failed on this host, so no facts were read and no commands were
generated. changed is False and reason names the failing
predicate.
dataclass
¶
LogEntry(
host: str,
name: str,
changed: bool,
reason: str,
commands: tuple[Command, ...] = (),
sudo: bool = False,
sudo_user: str | None = None,
su: bool = False,
su_user: str | None = None,
doas: bool = False,
doas_user: str | None = None,
skipped: bool = False,
)
One executed (or skipped) operation in the apply-phase run log.
skipped marks an entry whose when guard failed on this host: it
was neither materialized nor sent, changed is False and the
privilege fields stay at their inert defaults (nothing ran, so nothing
was elevated). This is a third outcome alongside changed and
"no change": ineligible regardless of state, versus nothing to do.
Raised when the apply-phase request sequence diverges from the plan.
If a generator body branches on a mutable fact, the prepare and apply
phases may produce different request sequences, breaking lockstep
pairing (design doc section 11.1). The driver treats this as a hard
error. The same applies to request values: a when predicate (or
any other argument) constructed fresh on each phase invocation -- e.g.
an inline lambda in a deploy body -- compares unequal across phases
even when semantically identical, so guards must be stable function
objects defined at module or composition level.
async
¶
run(
inventory: list[dict],
deploy_func: Callable,
*,
host_factory: Callable[[dict], Host] = from_inventory,
dry: bool = False,
) -> tuple[
dict[str, list[PlannedOperation]],
ApplyReport | None,
list[Host],
]
Top-level entry point: prepare, then apply (unless dry).
inventory is the user-supplied list of host dictionaries (see
hosts.py). It is validated up front with validate_inventory --
it must be a list of dictionaries, each carrying at least host and
connection -- and raises InventoryError on bad input before any
host is materialized. Each entry is then materialized into a Host
via host_factory. The default factory builds plain library hosts;
the pytest suites inject a factory building SimulatedHost objects (the
test-framework class in tests/simulated_host.py) so the simulation
stays out of the library. The examples/ scripts use the default
factory and connect for real, so they mutate the hosts they target; the
two exceptions are
examples/demonstrations/composition/composed_group_tag_roles.py, which
injects a simulated factory, and
examples/actions/action_in_deploy_guard.py, whose guard fires during
prepare, before any transport is opened.
Returns (plans, report, hosts) so callers can inspect the final
host state. A coroutine: both the prepare phase (which materializes
the plan) and the apply phase (which executes commands via the async
send_command) are awaited.
async
¶
run_prepare(
hosts: Iterable[Host], deploy_func: Callable
) -> tuple[
list[HostRuntime], dict[str, list[PlannedOperation]]
]
Prepare phase: invoke each host's generator and materialize the plan.
Returns (runtimes, plans). Fact reads go through the prepare-phase
cache and reflect pre-deployment state; nothing is executed. Each host is
prepared concurrently via asyncio.gather (hosts are independent), and
the results come back in input order, so runtimes and plans stay
deterministic. A coroutine, because materializing awaits operations and
their facts.
The _deploy_phase wrapper holds the in-deploy guard for the whole
phase: a gather_action call from a deploy generator raises
ActionInDeployError immediately (actions execute at once and would
mutate hosts during prepare).
Because the hosts connect concurrently, several may fail to be reached at
once. Per-host HostConnectionError instances are therefore collected
(via return_exceptions=True) and merged into a single aggregate naming
every unreachable host, so the caller learns about all of them together
rather than only the first to fail. Any other exception is a genuine error
and propagates unchanged.
async
¶
run_apply(
runtimes: Iterable[HostRuntime],
plans: dict[str, list[PlannedOperation]] | None = None,
) -> ApplyReport
Apply phase: advance all host generators in lockstep.
All hosts execute their Nth request before any host pulls request
N+1. Within a step the per-host execution (materialize + send) runs
concurrently across hosts via asyncio.gather; the gather join is the
barrier between steps, and results are collected in host order so the log
stays deterministic. The request pull and the determinism check stay
synchronous and complete before anything is sent. If plans is given,
each pulled request sequence is checked against the plan; a mismatch
raises DeterminismError. The _deploy_phase wrapper holds the
in-deploy guard here too, so an action called from a generator body
raises ActionInDeployError in apply exactly as it does in prepare.
Because a step's hosts execute concurrently, several may have their
elevation rejected at once. Per-host PrivilegeError instances are
therefore collected (via return_exceptions=True) and merged into a
single aggregate naming every host whose credentials failed, so one bad
sudo_password is reported for the whole inventory at once -- the same
treatment run_prepare gives HostConnectionError. Any other
exception is a genuine error and propagates unchanged, so a CommandError
or a bug still aborts the run.
The pull is also where the "yield requests" contract is enforced in apply
(mirroring _prepare_host): each pull runs inside the runtime's
_RequestCounter so op() calls in the body are tallied, every pulled
item is type-checked as an OperationRequest, and a body that finishes
having built more requests than it yielded raises DeployRequestError --
aggregated across the hosts of the step, since a lockstep step pulls from
every active host at once. A runtime the caller assembled itself, with no
deploy accounting to speak of, still gets the type check.
A coroutine, because commands are executed via the async
send_command.
async
¶
materialize(
host: Host, cache: FactCache, request: OperationRequest
)
Fully materialize an operation's async command generator.
Returns (ctx, commands). Materializing computes ctx.changed
(design doc section 6.3, cost of approach C); it is a coroutine because
the operation awaits facts, which gather their values asynchronously.
An operation must be an async generator function. Anything else -- e.g.
an action class passed to op() -- raises OperationError here rather
than failing obscurely in the async for: actions execute immediately
and cannot run inside a deploy (see reemote.actions).
async
¶
gather_fact(
hosts: Iterable[Host],
fact_cls: type,
*args: Any,
**kwargs: Any,
) -> dict[str, Any]
Gather one fact across all hosts concurrently.
The driver-level fan-out that FactCache.get performs for a single
host: it runs fact_cls.process for every host at once via
asyncio.gather and returns a host-aligned mapping of host.name to
that host's gathered value. Hosts are independent -- each owns its own
connection and (in tests) its own simulated state -- so the per-host reads
overlap on the network instead of running one host at a time.
asyncio.gather preserves input order in its result list, so zipping the
hosts with the gathered values keeps the mapping deterministic regardless
of the order in which individual hosts finish. A dict keyed by host name is
returned rather than a positional list so a caller can look a value up
without re-deriving the inventory order (the mapping is keyed by host
name, {host: value}).
fact_cls is a fact class exposing an async def process(host, ...)
classmethod (e.g. DnfRepositories); args/kwargs are forwarded to
it verbatim, exactly as FactCache.get does. A coroutine, because each
fact's process gathers its value asynchronously.
Because the hosts are read concurrently, several may lack the command the
fact runs at once. Per-host CommandNotFoundError instances (exit 127)
are therefore collected (via return_exceptions=True) and merged into
a single aggregate naming every host missing the command -- the same
treatment run_prepare gives HostConnectionError -- rather than
reporting only the first to fail. Any other exception is a genuine error
and propagates unchanged.
command_shell(
cmd: Command,
sudo: bool = False,
sudo_user: str | None = None,
su: bool = False,
su_user: str | None = None,
doas: bool = False,
doas_user: str | None = None,
) -> str
Render the shell string a command descriptor runs as (the loggable form).
Command descriptors stay dumb (design doc section 4.5); turning one into
shell text is the driver's job. With doas, sudo and/or su this
is the elevated form the apply log shows; it is deliberately credential-free
-- when the host's privilege carries a password the real transport runs
it under an askpass helper (see Host.run_shell_privileged), plumbing this
rendering never exposes so the password cannot leak into format_log.
send_command also calls it with the flags off to get the plain
descriptor shell it hands to the transport. A TransferCommand is never
elevated -- it copies over the connection's scp channel as the login user --
so its rendering is returned without any doas/sudo/su prefix
even when the flags are set.
The escalations combine in the fixed order doas then sudo then
su: doas is the outermost prefix and su the innermost, so a
request flagged all three renders as doas -n [-u <doas_user>] sudo
[-u <sudo_user>] su <su_user> -c '<shell>'.
sudo_user/su_user/doas_user are the run-as users taken from the
host's inventory privilege section (never from the operation).
sudo_user is emitted as sudo -u <user> except that root --
sudo's own default target -- collapses to a bare sudo. su_user is
emitted as su <user> -c except that None/root -- su's default
target -- collapses to a bare su -c. doas_user is emitted as
doas -n -u <user> except that None/root collapses to a bare
doas -n (doas is passwordless, so no credential is ever rendered).
async
¶
send_command(
host: Host,
cmd: Command,
*,
sudo: bool = False,
su: bool = False,
doas: bool = False,
) -> None
Execute one command descriptor on a host.
Two paths, chosen by the seams the host provides:
- The test framework's
SimulatedHostexposes anexecute_commandseam; when present it applies the command's simulated effect (so subsequent fact reads observe the change) and nothing is sent anywhere.sudo/su/doasare irrelevant here -- privilege has no in-memory effect -- so they are not forwarded. This is the path pytest always takes. - Any other host -- a plain library
Hostbuilt from a real inventory entry -- dispatches on the descriptor kind. ATransferCommand(e.g.files.put'sPutCommand,files.get'sGetCommand) copies over the connection's own transport via the descriptor'stransfer(asyncssh's scp), never through a shell, so thesudo/su/doasescalation flags do not apply to it and are ignored. AWaitCommand(e.g.server.wait_service'sWaitForFactCommand) likewise runs no shell: the driver awaits the descriptor'swait, which polls the host until its condition holds or raisesWaitTimeoutError; the escalation flags are ignored for it too. Anything else runs the command's shell over asyncssh, using the host'sconnectiondata. An elevated request (doas,sudoand/orsu) goes throughhost.run_shell_privileged, which applies the run-as users and any passwords (via askpass helpers) sourced from the inventory, in the fixed orderdoasthensudothensu; an unprivileged one goes throughhost.run_shell. Either way a non-zero exit raisesCommandError(fail-fast), so a command that fails -- a permission error, a broken package transaction -- is never silently reported as applied. A failure doas, sudo or su reports as its own authentication rejection (a wrong or missing password, or a doas denial) raisesPrivilegeErrorinstead, whichrun_applyaggregates across hosts exactly asrun_prepareaggregatesHostConnectionError.
format_plan(
plans: dict[str, list[PlannedOperation]],
) -> str
Render plan output as shown in design doc section 10.
format_log(report: ApplyReport) -> str
Render the apply-phase run log, including generated commands.