Skip to content

Module examples: server

Scripts under examples/modules/server/ — one per fact or operation of the server module. See the Examples overview for the shared "connects for real" warning and the error-reporting contract every script follows.

The read-only fact-gathering examples (server_arch.py, reading Arch, server_authorized_keys.py, reading AuthorizedKeys for the login user, server_date.py, reading Date, server_env.py, reading Env for PATH and HOME, server_groups.py, reading Groups, server_has_gui.py, reading HasGui, server_home.py, reading Home for the login user and for root, server_hostname.py, reading Hostname, server_kernel.py, reading Kernel, server_kernel_modules.py, reading KernelModules, server_kernel_version.py, reading KernelVersion, server_last.py, reading Last, server_linux_distribution.py, reading LinuxDistribution, server_linux_gui.py, reading LinuxGui, server_linux_name.py, reading LinuxName, server_locales.py, reading Locales, server_lsb_release.py, reading LsbRelease, server_mounts.py, reading Mounts, server_port.py, reading Port for TCP 80, UDP 53 and a closed port, server_processes.py, reading Processes, server_timezone.py, reading Timezone, server_uptime.py, reading Uptime, server_user.py, reading User, server_users.py, reading Users, and server_which.py, reading Which for python3 and httpd, all over gather_fact) live alongside the operation examples in examples/modules/server/. They are not rendered here — each reads a single fact, and the facts themselves are documented on the server module page.

server_lsb_release.py first installs its prerequisite: the CentOS demo hosts do not ship lsb_release in their base repos (EPEL provides it), so a short dnf.packages deploy flagged sudo=True puts the binary in place before the fact is gathered. The install is idempotent, and the read never hits the missing-command path.

Like the dnf examples, every script reads the shared project-root inventory.json and guards on the "centos" group by default (when=is_centos on every request, a host-list filter before each gather_fact fan-out) — the one exception is the locale example below, which guards on is_ubuntu because it is Debian-family only.

Ensuring system groups

server.group in action: the deploy creates the docker group (the baseline's documented example) and a lusers system group with a fixed gid, both under sudo. The operation is idempotent — it compares each name against the Groups fact and yields a GroupAddCommand only where the group is absent, so a second run plans no change. system=True renders groupadd -r and gid=1500 renders --gid 1500; on a FreeBSD host the same deploy would render pw groupadd -n ... instead (the Kernel fact picks the command form, and system is ignored on BSD). Removing a group is the mirror image: present=False yields a GroupDelCommand only when the group is present.

server_group

Example: ensure the docker and lusers system groups exist on real CentOS hosts.

This is a usage example, not part of the library. It shows reemote's server.group operation, baseline server.group parity for adding and removing system groups: the deploy creates the docker group (the baseline's documented example) and a lusers system group with a fixed gid, both under sudo because groupadd needs root.

The operation is idempotent: it reads the Groups fact (the group names in /etc/group) and yields a groupadd only where the group is absent, so a host that already has docker plans no change and a second run is a no-op everywhere. system=True renders groupadd -r and gid=1500 renders --gid 1500; both options only apply at creation -- an existing group is never modified. The Kernel fact picks the command form: on a FreeBSD host the same deploy renders pw groupadd -n ... (and system is ignored, BSD having no system groups), every other kernel gets groupadd/groupdel. Removing a group is the mirror image: present=False yields groupdel only when the group is present.

Every failure mode of the run -- an unreachable host, a rejected elevation, or a groupadd that exits non-zero (a gid already taken, for instance) -- reaches main as a typed error whose message already names the host, so main catches them and prints one line each instead of a traceback.

Unlike the pytest suite -- which injects the test framework's SimulatedHost via run's host_factory and never touches the network -- this example uses the default factory (Host.from_inventory): each plain Host lazily opens an asyncssh connection, reads /etc/group and uname -s for real, and runs groupadd under sudo where needed.

Every request carries when=is_centos, the declarative eligibility guard: the predicate reads only static inventory data -- host.group, set by the "group" key of the inventory entry -- and is a stable module-level function, so the prepare and apply phases reach the same skip decision. The driver skips a rejected request without materializing it, so on hosts outside the "centos" group neither the commands nor the facts behind them run, and plan and log render the step as [skipped: when guard is_centos] in its lockstep slot. The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file to target your machines.

The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file (or edit inventory.json) to target your machines.

Run it with::

uv run python examples/modules/server/server_group.py

Setting the hostname

server.hostname in action: the deploy names each host after its inventory label (web1 becomes web1.example.com), under sudo. The operation is idempotent — it compares the requested name against the Hostname fact and yields a SetHostnameCommand only where they differ, so a second run plans no change. The Which fact picks the command form: hostnamectl set-hostname where hostnamectl is available (it persists the name across reboots), the bare hostname command on older systems. The baseline's hostname_file upload branch is deliberately dropped — reemote's transfers cannot be elevated, so the root-owned /etc/hostname write has no faithful equivalent; on a systemd host hostnamectl already persists the name. The script is named server_op_hostname.py to keep it distinct from the read-only server_hostname.py fact example.

server_op_hostname

Example: set the system hostname on real CentOS hosts.

This is a usage example, not part of the library. It shows reemote's server.hostname operation, baseline server.hostname parity for naming a machine: the deploy gives each host a fully-qualified name derived from its inventory label (web1 becomes web1.example.com), under sudo because setting the hostname needs root.

The operation is idempotent: it reads the Hostname fact (uname -n) and yields a command only where the current name differs from the request, so a host already named correctly plans no change and a second run is a no-op everywhere. The Which fact picks the command form: a host with hostnamectl (systemd) gets hostnamectl set-hostname, which also persists the name across reboots; an older system without it gets the bare hostname command. The baseline's hostname_file upload branch is deliberately dropped -- reemote's transfers cannot be elevated, so the root-owned /etc/hostname write the baseline does through files.put has no faithful equivalent; on a systemd host hostnamectl already persists the name, so nothing is lost there.

Every failure mode of the run -- an unreachable host, a rejected elevation, or a hostnamectl that exits non-zero (an invalid name, for instance) -- reaches main as a typed error whose message already names the host, so main catches them and prints one line each instead of a traceback.

Unlike the pytest suite -- which injects the test framework's SimulatedHost via run's host_factory and never touches the network -- this example uses the default factory (Host.from_inventory): each plain Host lazily opens an asyncssh connection, reads uname -n and probes command -v hostnamectl for real, and sets the name under sudo where needed.

Every request carries when=is_centos, the declarative eligibility guard: the predicate reads only static inventory data -- host.group, set by the "group" key of the inventory entry -- and is a stable module-level function, so the prepare and apply phases reach the same skip decision. The driver skips a rejected request without materializing it, so on hosts outside the "centos" group neither the commands nor the facts behind them run, and plan and log render the step as [skipped: when guard is_centos] in its lockstep slot. The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file to target your machines.

The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file (or edit inventory.json) to target your machines.

Run it with::

uv run python examples/modules/server/server_op_hostname.py

Killing a process by PID

server.kill in action, and the determinism-safe way to "find the PID, then kill it" in ONE deploy. Like the baseline, the operation takes an explicit pid (and an optional signal, default TERM), so the script first takes a Processes snapshot via gather_fact across the guarded hosts, then builds the deploy from a factory closing over that frozen mapping: the generator loops over the snapshot, never live state, so prepare and apply yield the identical op(server.kill, pid=...) sequence. No first deploy is needed — nothing mutates the process table before the read, so this is just the snapshot half of the two-deploy split pattern. The operation is stateless and non-idempotent (every planned kill shows as a change), the signal flag and PID are shell-quoted, and the requests carry sudo=True because the nginx master runs as root. The frozen-snapshot trade-off: a PID that exits before the apply makes kill exit non-zero (CommandError); re-running the script re-snapshots and plans an empty deploy once nginx is gone.

server_kill

Example: find a process's PID, then kill it -- in ONE deterministic deploy.

This is a usage example, not part of the library. It shows reemote's server.kill operation, baseline server.kill parity for sending a signal to a running process. Like the baseline, the operation takes an explicit pid (and an optional signal, default TERM): discovering which PID to kill is the caller's job, and this script demonstrates the determinism-safe way to do it.

The deploy generator cannot read the Processes fact itself: it is synchronous (there is nothing to await inside it), and branching it on live process state would break the determinism guard -- the generator runs twice (prepare against pre-deployment state, apply against live state), so a state-dependent branch can yield a different request sequence per phase and run_apply raises DeterminismError. The sanctioned move is the snapshot half of the two_deploy_split pattern:

  1. A fact SNAPSHOT taken BEFORE the run: gather_fact fans the Processes read out across the guarded hosts and returns a {host.name: {pid: info}} mapping of the real process table.
  2. build_kill(snapshot) -- a factory closing over that frozen mapping, returning the deploy. Its loop consults the snapshot, never live state, so the generator is a pure function of static input: prepare and apply yield the identical op(server.kill, pid=...) sequence and the determinism guard holds.

Unlike two_deploy_split no FIRST deploy is needed here: nothing mutates the process table before the snapshot is read, so one snapshot plus one complete prepare/apply cycle is the whole story. (The full two-deploy split earns its keep when an earlier deploy step changes the fact the second deploy branches on -- install nginx, then configure it.)

The operation itself is stateless and non-idempotent, exactly like the baseline: it reads no fact and always yields one kill -<signal> <pid> command, so every planned kill shows as a change. The signal flag and the PID are shell-quoted, so a signal string carrying metacharacters reaches kill literally. The requests carry sudo=True because the nginx master process runs as root; the run-as user and any password come from the inventory privilege section, never from the operation.

The trade-off: the snapshot is frozen. If nginx exits between the snapshot and the apply, kill targets a dead PID, exits non-zero and the run fails with CommandError (the baseline behaves the same). Re-running the whole script is naturally safe: it re-snapshots, finds no nginx PID and plans an empty deploy.

Every failure mode of the run -- an unreachable host, a rejected elevation, or a kill that exits non-zero -- reaches main as a typed error whose message already names the host, so main catches them and prints one line each instead of a traceback.

Unlike the pytest suite -- which injects the test framework's SimulatedHost via run's host_factory and never touches the network -- this example uses the default factory (Host.from_inventory): each plain Host lazily opens an asyncssh connection, reads the process table for real, and runs kill under sudo where needed.

The hosts come from the shared inventory.json in the project root, read with load_inventory (parse + validate_inventory) and resolved from __file__ so the example works from any working directory. The snapshot gather is guarded by is_centos (host.group == "centos"): gather_fact takes no when parameter, so the predicate filters the host list before the fan-out and hosts outside the "centos" group are named on a == Skipped (when guard is_centos): ... == line and never connected. Every kill request also carries when=is_centos, and a skipped host has no snapshot entry, so its deploy yields nothing.

Run it with::

uv run python examples/modules/server/server_kill.py

Loading and unloading kernel modules

server.modprobe in action: the deploy loads the nbd module and then ensures loop and brd are loaded, both under sudo because modprobe needs root. The operation is idempotent — it compares the names against the KernelModules fact and yields a ModprobeCommand only for the modules that differ from the request, so a host where loop is already loaded re-loads nothing and a second run plans no change. module accepts a single name or a tuple of names, force=True would render modprobe -f (without ever forcing a reload of an in-state module), and unloading is the mirror image: present=False yields modprobe -r -a only for the names that are loaded. A request needs the modprobe binary (the kmod package) and the .ko file of each named module under the running kernel, so the script first drives an idempotent dnf.packages install of kmod and kernel-modules-core — the package shipping nbd, loop and brd — before the load deploy reads its facts. The dependency on the running kernel's tree is why these modules are chosen over anything in kernel-modules-extra: installing that package only populates /lib/modules for the newest installed kernel, so on a host not yet rebooted into it modprobe would report the module as "not found in directory" — a real CommandError, which the script reports as one line (the same path an unknown module or a module still in use on removal takes).

server_modprobe

Example: load the nbd, loop and brd kernel modules on real CentOS hosts.

This is a usage example, not part of the library. It shows reemote's server.modprobe operation, baseline server.modprobe parity for loading and unloading kernel modules: the deploy first ensures the software behind the request is installed, then loads the nbd module and ensures the loop and brd modules are loaded, both under sudo because modprobe needs root.

Prerequisite software. A modprobe request needs two things on the host: the modprobe binary itself (the kmod package) and the .ko file of every named module in the running kernel's /lib/modules tree. The three modules requested here -- nbd, loop and brd -- are all shipped by kernel-modules-core, the package every kernel install depends on, so their module files are guaranteed present for the booting kernel. That matters: a module living in kernel-modules-extra would make the example version-sensitive, because a bare dnf install of that package only populates /lib/modules for the newest installed kernel, leaving modprobe reporting "not found in directory" on a host that has not been rebooted into it. main therefore drives a small install deploy (dnf.packages of kmod and kernel-modules-core, idempotent) before the modprobe deploy -- the same two-run split two_deploy_split.py demonstrates -- so the load step runs against a host whose module database is really in place.

The operation is idempotent: it reads the KernelModules fact (the loaded modules from /proc/modules) and yields a modprobe -a only for the names that are missing, so a host where loop is already loaded re-loads nothing and a second run is a no-op everywhere. module accepts a single name or a tuple of names (tuples keep op() requests hashable), and force=True would render modprobe -f -- without ever forcing a reload of an already-loaded module. Unloading is the mirror image: present=False yields modprobe -r -a only for the names that are loaded.

Every failure mode of the run -- an unreachable host, a rejected elevation, or a modprobe that exits non-zero (an unknown module, or a module in use on removal) -- reaches main as a typed error whose message already names the host, so main catches them and prints one line each instead of a traceback.

Unlike the pytest suite -- which injects the test framework's SimulatedHost via run's host_factory and never touches the network -- this example uses the default factory (Host.from_inventory): each plain Host lazily opens an asyncssh connection, reads uname -s and /proc/modules for real, and runs modprobe under sudo where needed.

Every request carries when=is_centos, the declarative eligibility guard: the predicate reads only static inventory data -- host.group, set by the "group" key of the inventory entry -- and is a stable module-level function, so the prepare and apply phases reach the same skip decision. The driver skips a rejected request without materializing it, so on hosts outside the "centos" group neither the commands nor the facts behind them run, and plan and log render the step as [skipped: when guard is_centos] in its lockstep slot. The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file to target your machines.

The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file (or edit inventory.json) to target your machines.

Run it with::

uv run python examples/modules/server/server_modprobe.py

Mounting and unmounting filesystems

server.mount in action: the deploy mounts /data from /dev/sdb1 as ext4 with the rw and noatime options, then ensures /scratch is unmounted, both under sudo because mount/umount need root. The operation is idempotent — it compares against the Mounts fact and yields a MountCommand only where the mount differs from the request: a fresh mount when the mount point is absent, an umount when mounted=False finds it present, and a mount -o remount,... only when an already-mounted filesystem is missing some of the requested options (a subset check, so extra options already on the mount never trigger a remount). When path is not found directly but device is given, the mount is located by device instead, and that found mount point is what an unmount or remount acts on; on a BSD host the remount takes the mount -uo ... -t ... form (the Kernel fact picks it). Like the baseline, /etc/fstab is never touched — a mount made this way does not survive a reboot unless an fstab entry is managed separately. The mount target directory must already exist and the device must be present; a missing device or directory makes mount exit non-zero and the run fails with a typed CommandError, which the script reports as one line. The demo hosts have no /dev/sdb1 (only /dev/sda), and CentOS staff_t confinement additionally denies creating /data and mounting filesystems — a host-policy and storage prerequisite, not a script defect: it needs an operator-provisioned device, an existing mount point and an unconfined sudo domain.

server_mount

Example: mount a data filesystem and unmount a scratch one on real CentOS hosts.

This is a usage example, not part of the library. It shows reemote's server.mount operation, baseline server.mount parity for managing mounted filesystems: the deploy mounts /data from /dev/sdb1 as ext4 with the rw and noatime options, then ensures the /scratch filesystem is unmounted, both under sudo because mount/umount need root.

The operation is idempotent: it reads the Mounts fact (the mounted filesystems from /proc/self/mountinfo) and yields a mount only when the mount point is absent, an umount only when it is present, and a mount -o remount,... only when an already-mounted filesystem is missing some of the requested options -- so a second run is a no-op everywhere. When path is not found directly but device is given, the mount is located by device instead (the baseline's fallback for paths whose spelling differs from the kernel's listing), and that found mount point is what an unmount or remount acts on. On a BSD host the remount takes the mount -uo ... -t ... form instead (the Kernel fact picks it). Like the baseline, /etc/fstab is never touched: a mount made this way does not survive a reboot unless an fstab entry is managed separately (a files.edit recipe).

The mount target directory must already exist and the device must be present -- this example does not create /data or partition disks. A missing device or directory makes mount exit non-zero and the run fails with a typed CommandError, which main reports as one line; an already-unmounted /scratch simply plans no change.

Host-policy and storage prerequisite. The demo hosts have no /dev/sdb1 (only /dev/sda), and on the CentOS hosts SELinux staff_t confinement additionally denies creating the /data mount point and mounting filesystems. The script is correct (sudo=True); it needs a real block device provisioned by the operator, an existing mount point, and an unconfined sudo domain. Point INVENTORY_PATH at a host with the target device and directory -- or relax SELinux and attach a disk -- to run it for real.

Every failure mode of the run -- an unreachable host, a rejected elevation, or a mount/umount that exits non-zero (a missing device, a busy filesystem on unmount) -- reaches main as a typed error whose message already names the host, so main catches them and prints one line each instead of a traceback.

Unlike the pytest suite -- which injects the test framework's SimulatedHost via run's host_factory and never touches the network -- this example uses the default factory (Host.from_inventory): each plain Host lazily opens an asyncssh connection, reads uname -s and /proc/self/mountinfo for real, and runs mount/umount under sudo where needed.

Every request carries when=is_centos, the declarative eligibility guard: the predicate reads only static inventory data -- host.group, set by the "group" key of the inventory entry -- and is a stable module-level function, so the prepare and apply phases reach the same skip decision. The driver skips a rejected request without materializing it, so on hosts outside the "centos" group neither the commands nor the facts behind them run, and plan and log render the step as [skipped: when guard is_centos] in its lockstep slot. The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file to target your machines.

The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file (or edit inventory.json) to target your machines.

Run it with::

uv run python examples/modules/server/server_mount.py

Setting the timezone

server.timezone in action: the deploy ensures every host keeps Europe/Amsterdam (the baseline's documented example), under sudo. The operation is idempotent — it compares the requested zone against the Timezone fact and yields a SetTimezoneCommand (timedatectl set-timezone) only where they differ, so a second run plans no change. It is supported on systemd systems only: when a change is needed the Which fact probes timedatectl, and a host without it raises OperationError during the prepare phase — before anything is sent — rather than falling back. The baseline's non-systemd fallback (symlinking /etc/localtime and writing /etc/timezone) is deliberately dropped; on a systemd host timedatectl already persists the zone across reboots. The script is named server_op_timezone.py to keep it distinct from the read-only server_timezone.py fact example.

server_op_timezone

Example: set the system timezone on real CentOS hosts (systemd only).

This is a usage example, not part of the library. It shows reemote's server.timezone operation, baseline server.timezone parity for the system timezone: the deploy ensures every host keeps Europe/Amsterdam (the baseline's documented example), under sudo because timedatectl needs root.

The operation is idempotent: it reads the Timezone fact (readlink -f /etc/localtime) and yields a command only where the current zone differs from the request, so a host already on the right zone plans no change and a second run is a no-op everywhere. It is supported on systemd systems only: when a change is needed the Which fact probes timedatectl, and the command it yields -- timedatectl set-timezone -- also persists the zone across reboots. A host without timedatectl raises OperationError during the prepare phase, before anything is sent; the baseline's non-systemd fallback (symlinking /etc/localtime and writing /etc/timezone) is deliberately dropped, so there is no silent edit behind systemd's back. An already-set zone stays a no-op on any host, like the baseline, which also compares the fact before probing for the tool.

Every failure mode of the run -- an unreachable host, a rejected elevation, a timedatectl that exits non-zero (an unknown zone, for instance), or a non-systemd host missing the tool -- reaches main as a typed error whose message already names the host, so main catches them and prints one line each instead of a traceback.

Unlike the pytest suite -- which injects the test framework's SimulatedHost via run's host_factory and never touches the network -- this example uses the default factory (Host.from_inventory): each plain Host lazily opens an asyncssh connection, resolves /etc/localtime and probes command -v timedatectl for real, and sets the zone under sudo where needed.

Every request carries when=is_centos, the declarative eligibility guard: the predicate reads only static inventory data -- host.group, set by the "group" key of the inventory entry -- and is a stable module-level function, so the prepare and apply phases reach the same skip decision. The driver skips a rejected request without materializing it, so on hosts outside the "centos" group neither the commands nor the facts behind them run, and plan and log render the step as [skipped: when guard is_centos] in its lockstep slot. The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file to target your machines.

The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file (or edit inventory.json) to target your machines.

Run it with::

uv run python examples/modules/server/server_op_timezone.py

Creating and removing user accounts

server.user in action: the deploy creates a deploy service account with a login shell and a GECOS comment in the apps group, appends the staff group to it, then removes a legacy account — all under sudo, because useradd, usermod, userdel and the home-directory pair need root. The two groups come from server.group requests first, since the lockstep barrier finishes those steps before any host touches an account. The operation is idempotent — it compares the request against the Users fact and yields useradd for an absent account, a usermod carrying only the differing fields for a present one, and userdel for present=False, so a second run plans no change. groups is a tuple because op() arguments must stay hashable, and append=True adds memberships without removing any. The default ensure_home is the baseline's files.directory delegation, inlined: with create_home=False (the default) useradd builds nothing from the skeleton, so the plan shows the mkdir -p /home/deploy plus chown deploy:deploy /home/deploy pair that actually creates the directory. The append step also makes the per-phase fact cache visible: in the plan both user requests read the same pre-deployment state and both show a change, while in the apply each step re-reads the facts, so the fourth step sees the groups the third wrote and sends only usermod -a -G staff deploy. The baseline's public_keys/delete_keys options are deliberately dropped (key files are managed through the separate server.user_authorized_keys operation), as are its plan-time Which guards, so a missing useradd surfaces at apply as CommandNotFoundError. The script mutates the shared demo inventory for real: the account, its home directory and the two groups stay on the hosts. The script is named server_op_user.py to keep it distinct from the read-only server_user.py fact example.

server_op_user

Example: create, adjust and remove a user account on real CentOS hosts.

This is a usage example, not part of the library. It shows reemote's server.user operation, baseline server.user parity for user accounts: the deploy creates a deploy service account with a login shell and a GECOS comment, puts it in the apps group, then appends the staff group to it, and finally removes a legacy account that is no longer wanted -- all under sudo, because useradd, usermod, userdel and the home-directory mkdir/chown pair all need root.

The operation is idempotent: it reads the Users fact (every /etc/passwd row joined with id's groups, the lastlog row and the shadow hash) and yields a command only where the account differs from the request -- useradd for an absent user, a usermod carrying just the differing fields for a present one, userdel for present=False. A second run therefore plans no change. Two details are worth seeing in the plan:

  • the ensure_home branch (on by default) is the baseline's delegation to files.directory, inlined as mkdir -p /home/deploy plus chown deploy:deploy /home/deploy when the home is missing, a plain chown when only the owner differs, and no command at all when the home is a symlink. create_home=False here means useradd builds nothing from the skeleton, so that pair is what actually creates the directory;
  • the append step shows the per-phase fact cache. In the plan both user requests read the same pre-deployment state, so both show a change; in the apply each step re-reads the facts, so step 4 sees the groups step 3 wrote and sends only usermod -a -G staff.

groups is passed as a tuple: every op() argument must stay hashable for compose(dedupe=True), so a list would be rejected with UnhashableArgumentError. The two secondary groups are created first by server.group requests, which the lockstep barrier completes before the account is touched. The baseline's public_keys/delete_keys options are deliberately dropped -- that branch rides files.line appends and an elevated files.put, neither of which reemote has -- so the script manages no authorized_keys file; the baseline's plan-time Which guards are dropped too, so a missing useradd surfaces at apply as CommandNotFoundError from the exit status. password is also left unset here: it takes a pre-encrypted shadow hash, which a shared demo inventory should not carry.

Unlike the pytest suite -- which injects the test framework's SimulatedHost via run's host_factory and never touches the network -- this example uses the default factory (Host.from_inventory): each plain Host lazily opens an asyncssh connection, reads /etc/passwd, /etc/group, /etc/shadow, uname -s and the home directory status for real, and runs the account commands under sudo where needed.

Every failure mode of the run -- an unreachable host, a rejected elevation, an OperationError for a home path that exists as something other than a directory, or a useradd/usermod that exits non-zero (a group that turns out to be missing, for instance) -- reaches main as a typed error whose message already names the host, so main catches them and prints one line each instead of a traceback.

Note that this script mutates the shared demo inventory: the deploy account and its home directory stay on the hosts (and the apps/ staff groups with them), while legacy is removed wherever it existed. Re-running it is a no-op.

Every request carries when=is_centos, the declarative eligibility guard: the predicate reads only static inventory data -- host.group, set by the "group" key of the inventory entry -- and is a stable module-level function, so the prepare and apply phases reach the same skip decision. The driver skips a rejected request without materializing it, so on hosts outside the "centos" group neither the commands nor the facts behind them run, and plan and log render the step as [skipped: when guard is_centos] in its lockstep slot. The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file to target your machines.

The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file (or edit inventory.json) to target your machines.

Run it with::

uv run python examples/modules/server/server_op_user.py

Authorising SSH public keys

server.user_authorized_keys in action: the deploy ensures the login user's ~/.ssh directory (mode 700) and authorized_keys file (mode 600) exist and are owned correctly, then writes the public keys not already listed — the baseline's documented flow with its controller-side key-file paths dropped (every public_keys entry here is a literal key string, and it is a tuple because op() arguments must stay hashable). The operation is idempotent — it reads the Home fact to resolve the directory, a private _KeysPathStat probe for the directory/file owner and mode, and the AuthorizedKeys fact (read with the explicit key-file path) for which keys are present, yielding a command only where they differ, so a second run plans no change. Creating a key file from nothing shows the whole setup (mkdir -p + chmod 700 + chown for ~/.ssh, then touch + chmod 600 + chown for the file) before the write. The write itself is a spool-and-install pair: the operation reads the file's current text through the AuthorizedKeysFile raw-content fact, appends the missing keys controller-side (comments and ordering preserved), scps the result to a /tmp spool named after the key file, then installs it with sh -c 'mkdir -p … && cp … && chmod 600 … && rm -f …'. The scp half stays unprivileged even under sudo=True — transfers never elevate — so the log shows two privilege levels in one request. Here the keys are managed for the login user themselves, so no sudo is needed; managing another user's keys would flag sudo=True. The baseline's delete_keys option is supported too (a whole-file rewrite to exactly the requested keys, riding the same spool-and-install pair), though this example only appends. The script is named server_user_authorized_keys.py to keep it distinct from the read-only server_authorized_keys.py fact example.

server_user_authorized_keys

Example: authorise SSH public keys in a user's authorized_keys file.

This is a usage example, not part of the library. It shows reemote's server.user_authorized_keys operation, baseline server.user_authorized_keys parity for SSH keys: the deploy makes sure a user's ~/.ssh directory (mode 700) and authorized_keys file (mode 600) exist and are owned correctly, then appends the public keys that are not already listed -- the baseline's documented flow, minus its controller-side key-file paths (every entry here is a literal key string).

The operation is idempotent and reads four facts. The Home fact resolves the directory when no authorized_key_directory is passed, a private _KeysPathStat probe compares the directory and the file against the shared files.Directory/files.File state, the AuthorizedKeys fact (read with the explicit key-file path) lists the keys already present, and the AuthorizedKeysFile raw-content fact reads the file's current text so the missing keys can be appended controller-side -- only the missing ones ride a write. Two details are worth seeing in the plan:

  • creating a key file from nothing yields the whole setup -- mkdir -p + chmod 700 + chown for ~/.ssh, then touch + chmod 600 + chown for authorized_keys -- before the write, while a run where everything already matches yields nothing at all;
  • the write is a spool-and-install pair: the transformed text rides an unprivileged scp to a /tmp spool named after the key file, then an sh -c 'mkdir -p ... && cp ... && chmod 600 ... && rm -rf ...' install overlays the key file and removes the spool directory. The scp half stays unprivileged even under sudo=True -- transfers never elevate -- so a request for another user's keys shows two privilege levels in one log.

public_keys is a tuple: every op() argument must stay hashable for compose(dedupe=True), so a list would be rejected with UnhashableArgumentError. Here the keys are managed for the login user themselves, so no elevation is needed -- writing into ~/.ssh under your own home is unprivileged. Managing keys for another user is the mirror case: flag the request sudo=True, since the directory and file sit in that user's home. The baseline's delete_keys option is supported as well -- a whole-file rewrite to exactly the requested keys, riding the same spool-and-install pair -- though this example only appends.

Unlike the pytest suite -- which injects the test framework's SimulatedHost via run's host_factory and never touches the network -- this example uses the default factory (Host.from_inventory): each plain Host lazily opens an asyncssh connection, reads the home directory, the .ssh/authorized_keys stat, the existing keys and the key file's text for real, and runs the setup and spool-and-install commands where they differ.

Every failure mode of the run -- an unreachable host, a rejected elevation, an OperationError for a path that exists as the wrong type (an authorized_keys that is a directory, say), or a command that exits non-zero -- reaches main as a typed error whose message already names the host, so main catches them and prints one line each instead of a traceback.

Note that this script mutates the target user's ~/.ssh/authorized_keys for real: the appended keys stay in place. Re-running it is a no-op, because the AuthorizedKeys fact already reports them.

Every request carries when=is_centos, the declarative eligibility guard: the predicate reads only static inventory data -- host.group, set by the "group" key of the inventory entry -- and is a stable module-level function, so the prepare and apply phases reach the same skip decision. The driver skips a rejected request without materializing it, so on hosts outside the "centos" group neither the commands nor the facts behind them run, and plan and log render the step as [skipped: when guard is_centos] in its lockstep slot. The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file to target your machines.

The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file (or edit inventory.json) to target your machines.

Run it with::

uv run python examples/modules/server/server_user_authorized_keys.py

Managing /etc/hosts entries

server.etc_hosts in action: the deploy ensures the line keyed by 10.0.0.5 holds exactly db.internal db, under sudo because /etc/hosts is root-owned. It replaces the earlier two-run files.edit → files.move recipe that used to live in the files examples — the fact read, the pure hosts-file transform and the install now live in one operation, so a single deploy suffices. The operation is idempotent: it compares the parsed EtcHosts fact first (the baseline's no-op reasons verbatim, order-sensitive on the hostnames), and only a differing entry triggers the write — the AuthorizedKeysFile raw-content fact reads the file's current text, the transform rewrites just the IP's line (every other line, comment and blank line preserved), and the result rides a spool-and-install pair: an unprivileged scp to a /tmp spool named after the managed path, then an elevated sh -c 'mkdir -p … && cp … && chmod 0644 … && rm -f …' install. The scp half stays unprivileged even under sudo=True — transfers never elevate — so the log shows two privilege levels in one request. hostnames is a tuple because op() arguments must stay hashable; present=False with names drops only those, without names removes every line for the IP. The deploy yields the same request twice, so the apply log makes the per-phase fact cache visible: the second step sees the entry the first wrote and reports no change.

server_etc_hosts

Example: manage an /etc/hosts entry with server.etc_hosts.

This is a usage example, not part of the library. It shows reemote's server.etc_hosts operation, baseline server.etc_hosts parity: the deploy ensures the line keyed by an IP address holds exactly the requested hostnames, rewriting it in place when the names differ and appending it when the IP is unknown. It replaces the earlier two-run files.edit -> files.move example recipe: the fact read, the pure hosts-file transform and the install now live in one operation, so a single deploy suffices -- no snapshot, no transform factory, no barrier between "spool exists" and "move validates it".

The operation is idempotent and reads two facts. The parsed EtcHosts fact maps each IP to its hostnames; when the entry already matches, the request is a no-op with the baseline's own reason ("already present in", "already absent from", "does not reference any of:"). When a change is due, the AuthorizedKeysFile raw-content fact -- the server namespace's path-generic content reader -- supplies the file's current text and a pure controller-side transform computes the new whole content, preserving every untouched line, comment and blank line (comments on the edited line are dropped, baseline parity).

The write is a spool-and-install pair: the transformed text rides an unprivileged scp to a /tmp spool whose name is derived from the managed path, then an sh -c 'mkdir -p ... && cp ... && chmod 0644 ... && rm -rf ...' install overlays the hosts file and removes the spool directory. The scp half stays unprivileged even under sudo=True -- transfers never elevate -- so the log shows two privilege levels in one request: a $ scp <hosts file> -> ... line followed by a $ sudo sh -c '...' line. /etc/hosts is root-owned, so both requests here carry sudo=True.

hostnames is a tuple: every op() argument must stay hashable for compose(dedupe=True), so a list would be rejected with UnhashableArgumentError. A single string works too (it is split on whitespace); present=False with hostnames drops only those names, and without hostnames removes every line for the IP.

Unlike the pytest suite -- which injects the test framework's SimulatedHost via run's host_factory and never touches the network -- this example uses the default factory (Host.from_inventory): each plain Host lazily opens an asyncssh connection, reads the real /etc/hosts for the fact comparison and the content transform, and runs the spool-and-install pair only where the entry differs.

Every failure mode of the run -- an unreachable host, a rejected elevation, a ValueError for present=True without hostnames, or a command that exits non-zero -- reaches main as a typed error whose message already names the host (or the argument), so main catches them and prints one line each instead of a traceback.

Note that this script mutates the target hosts' /etc/hosts for real: the entry stays in place. Re-running it is a no-op, because the EtcHosts fact already reports the requested names.

Every request carries when=is_centos, the declarative eligibility guard: the predicate reads only static inventory data -- host.group, set by the "group" key of the inventory entry -- and is a stable module-level function, so the prepare and apply phases reach the same skip decision. The driver skips a rejected request without materializing it, so on hosts outside the "centos" group neither the commands nor the facts behind them run, and plan and log render the step as [skipped: when guard is_centos] in its lockstep slot.

The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file (or edit inventory.json) to target your machines.

Run it with::

uv run python examples/modules/server/server_etc_hosts.py

Managing locales (baseline server.locale parity)

server.locale in action: the deploy ensures fr_FR.UTF-8 is enabled under sudo, the baseline parity for /etc/locale.gen. It is a Debian-family operation: /etc/locale.gen and locale-gen exist only on Debian/Ubuntu (the locales package), so it targets the ubuntu host, not the CentOS web1/web2 servers — RHEL-family hosts install locales through glibc-langpack-* and localedef and have no /etc/locale.gen, and a request there raises OperationError in the prepare phase. The operation is spool-and-install: in one request it reads the Locales fact (locale -a) and the raw /etc/locale.gen text, applies the baseline's change condition (enable only when the locale is absent from locale -a, disable only when present) and definition-line lookup — a Python regex instead of a FindInFile grep, OperationError with the baseline's messages on zero or multiple matches — then scp's the transformed text to a deterministic /tmp spool as the login user, installs it over /etc/locale.gen with an elevated sh -c (chmod 0644, spool removed) and finishes with one elevated locale-gen. The log shows the family's two privilege levels in one request: an unprivileged $ scp <locale.gen> -> ... line followed by $ sudo sh -c '...' and $ sudo locale-gen. Re-running the script is a no-op once locale -a reports the requested state, and a definition line already in the desired state skips only the write, never the regeneration. The script replaces the earlier two-run files.edit -> files.move recipe: the fact reads, edit, install and regeneration now live inside the operation, so a single deploy suffices. It is named server_op_locale.py to keep it distinct from the read-only server_locales.py fact example.

server_op_locale

Example: enable a locale on real Ubuntu hosts (Debian-family only).

This is a usage example, not part of the library. It shows reemote's server.locale operation, baseline server.locale parity for /etc/locale.gen: the deploy ensures every host has fr_FR.UTF-8 enabled, under sudo because the definitions file is root-owned and locale-gen writes the system locale archive.

server.locale is a spool-and-install operation, and it replaces the earlier two-run files.edit -> files.move example recipe that used to live in this file: the fact reads, the line edit, the install and the regeneration now happen inside one operation, so a single deploy suffices. In one request it reads the Locales fact (locale -a) and the raw /etc/locale.gen text, uncomments the locale's definition line on the controller, scp's the transformed text to a deterministic /tmp spool (the transfer rides the login user -- transfers never elevate), installs it over /etc/locale.gen with an elevated sh -c (chmod 0644, spool removed), and finishes with one elevated locale-gen. The log shows the family's two privilege levels: an unprivileged $ scp <locale.gen> -> ... line followed by $ sudo sh -c '...' and $ sudo locale-gen.

The operation is idempotent: it plans a change only where the Locales listing disagrees with the request, so a host with the locale already generated plans no change and a second run is a no-op everywhere. When the definition line is already uncommented but the archive has not been generated yet, the write is skipped and only the locale-gen step is planned. It is supported on Debian-family systems only: a host with no /etc/locale.gen (RHEL-family systems install locales through glibc-langpack-*) or no matching definition line raises OperationError during the prepare phase, before anything is sent -- the baseline's own behaviour.

Every failure mode of the run -- an unreachable host, a rejected elevation, a missing or ambiguous definition line, or a locale-gen that exits non-zero -- reaches main as a typed error whose message already names the host, so main catches them and prints one line each instead of a traceback.

Unlike the pytest suite -- which injects the test framework's SimulatedHost via run's host_factory and never touches the network -- this example uses the default factory (Host.from_inventory): each plain Host lazily opens an asyncssh connection, reads locale -a and /etc/locale.gen for real, and enables the locale under sudo where needed.

Every request carries when=is_ubuntu, the declarative eligibility guard: the predicate reads only static inventory data -- host.group, set by the "group" key of the inventory entry -- and is a stable module-level function, so the prepare and apply phases reach the same skip decision. The driver skips a rejected request without materializing it, so on hosts outside the "ubuntu" group neither the commands nor the facts behind them run, and plan and log render the step as [skipped: when guard is_ubuntu] in its lockstep slot.

The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file (or edit inventory.json) to target your machines.

Run it with::

uv run python examples/modules/server/server_op_locale.py

Ensuring a sysctl setting (baseline server.sysctl parity)

server.sysctl is superseded by the native sysctl module (sysctl.setting / sysctl.conf) — this recipe stays as the strict-parity demonstration, built on the generic files.edit primitive (the Sysctl fact it compares against relocated to that namespace as sysctl.Sysctl; the top-level reemote.Sysctl binding still resolves). The script snapshots Sysctl for fs.file-max and FileContents("/etc/sysctl.conf") from every guarded host, then applies the baseline's two independent decisions to the frozen snapshots: the runtime gate is the baseline's fact comparison on the normalized value (lists join with spaces for the shell, digits become ints so "2000" and 2000 compare equal), and the persist gate compares the desired content — computed by the pure sysctl_persist_edit function with the baseline's files.line semantics (match key = value with flexible spacing, commented lines included, normalize every match, append when absent, stale key = other lines survive) — against the file's current text. Run 1 applies the live value through the demo-side stateless sysctl_set operation (a shell-quoted sysctl key=value StringCommand, sudo=True); run 2 spools the content to /tmp/sysctl.conf through files.edit with a closure-factory transform frozen between the snapshot and the run; run 3 installs it into /etc with files.move flagged sudo=True and overwrite=True. The spool and the move cannot share one deploy — files.move validates its source in the prepare phase too, and prepare never sends. Re-running the script is a no-op once the live value and the persist line both match. The script is named server_op_sysctl.py; the read-only fact example relocated to sysctl_sysctl.py, and everyday use should prefer the native sysctl.setting operation.

server_op_sysctl

Example: ensure a sysctl setting (baseline server.sysctl parity).

This is a usage example, not part of the library. It is reemote's strict-parity demonstration of the baseline's server.sysctl operation, built from generic primitives: the persist half is a file edit fully expressible with the files.edit content-transform primitive plus the elevated files.move -- the two-run recipe shape server_op_locale.py used before server.locale was ported as a native spool-and-install operation -- and the runtime half is a single sysctl key=value shell line this example defines itself over the public raw StringCommand descriptor. The native sysctl module (sysctl.setting / sysctl.conf) now supersedes this recipe for everyday use; the recipe stays as the demonstration that the baseline's semantics are expressible without a bespoke operation.

the baseline's server.sysctl(key, value, persist=False, persist_file="/etc/sysctl.conf") normalizes value (a list joins with spaces for the shell; digits become ints so the fact comparison matches), reads the Sysctl fact for the key, runs sysctl 'key'='value' only when the live value differs, and -- independently of that comparison -- ensures the line key = value in the persist file through files.line (match key[[:space:]]*=[[:space:]]*value, replace with key = value, append when absent). The recipe mirrors all of that:

  1. Two fact SNAPSHOTS: gather_fact reads Sysctl for the one key and FileContents of /etc/sysctl.conf from every guarded host (read-only SSH). The pure sysctl_persist_edit function performs the baseline's files.line lookup with a Python regex over the content -- reemote ships no grep-style FindInFile fact, and FileContents plus local parsing is its content-read convention.
  2. Two independent skip gates on the frozen snapshots, matching the baseline's two independent decisions: the runtime gate (sysctl_needed) is the baseline's existing_value != value fact comparison on the normalized value; the persist gate compares the transformed content against the file's current text (the baseline's files.line no-ops when the exact key = value line is already present). A host can need either half, both or neither.
  3. Run 1 applies the runtime value: the demo-side sysctl_set operation yields sysctl key=value (each side shell-quoted, the baseline's QuoteString parity), flagged sudo=True. reemote ships no generic "run a shell command" operation, so this example defines the stateless one-liner itself; the snapshot gate upstream decides whether it runs at all.
  4. Run 2 spools: files.edit writes the desired sysctl.conf content to /tmp/sysctl.conf (login-user-writable). Each host's transform is built ONCE by the make_transform factory closing over the frozen desired content -- a stable callable across both phases, the determinism-safe split shown in examples/demonstrations/composition/two_deploy_split.py.
  5. Run 3 installs: files.move moves the spool file into /etc with overwrite=True and sudo=True. The edit and the move cannot share one deploy: files.move validates src in the prepare phase too, and prepare never sends, so the spool file must really exist first.

Deliberate deviations from the baseline, decided at migration time:

  • The persist edit (files.line's sed/echo) is replaced by the files.edit transform: the generic primitive expresses the edit without a bespoke operation, and reemote's transfers are unprivileged, so root-owned /etc/sysctl.conf is written through the spool-and-install vehicle instead of an elevated sed/echo. The baseline's timestamped sed backup dance goes with it.
  • The line regex escapes the key and the value literally (re.escape); the baseline interpolates them raw into a grep pattern, where the . in fs.file-max would match any character. The POSIX [[:space:]]* class is translated verbatim (minus the newline -- matching is per line).
  • the baseline's files.line search semantics are kept: the pattern is unanchored, so a commented # key = value line matches and is normalized (uncommented) in place, and a longer value sharing the prefix (key = 1000000) matches too. A stale key = other line does NOT match and survives beside the new line, exactly like the baseline -- the last assignment wins when sysctl.conf is replayed at boot.
  • The desired content always ends in one trailing newline (the recipe's "\n".join(lines) + "\n" house style), so a file missing its final newline is rewritten once even when the line itself is already correct.
  • The runtime sysctl step is stateless like the baseline's yielded command; the baseline's noop message ("sysctl {key} is set to {value}") becomes the controller-side skip, printed from the frozen snapshot.

Re-running the whole script is a no-op once the live value and the persist line both match: the snapshot gates skip every run. Reading the settings back without changing anything is the companion examples/modules/sysctl/sysctl_sysctl.py example's job.

The hosts come from the shared inventory.json in the project root, read with load_inventory (parse + validate_inventory) and resolved from __file__ so the example works from any working directory. The snapshot gathers are guarded by is_centos (host.group == "centos"): gather_fact takes no when parameter, so the predicate filters the host list before the fan-out and hosts outside the "centos" group are named on a == Skipped (when guard is_centos): ... == line and never connected. Every deploy request also carries when=is_centos, and only hosts with a snapshot entry can become pending.

Run it with::

uv run python examples/modules/server/server_op_sysctl.py

Ensuring a resource limit (baseline server.security_limit parity)

server.security_limit in action: the deploy ensures * soft nofile 1024 is set under sudo, the baseline parity for /etc/security/limits.conf. It is a spool-and-install operation, and it replaces the earlier files.edit + demo-side install recipe that used to live in this file: the fact read, the line edit and the install now live inside one operation, so a single deploy suffices. In one request it reads the raw /etc/security/limits.conf text through the AuthorizedKeysFile content fact and applies the baseline's anchored files.line lookup — a Python regex instead of a FindInFile grep, the value not part of the pattern, so a stale * soft nofile 65535 line is corrected in place and every match becomes the tab-formatted replacement; with no match the line appends — the fields escaped literally where the baseline interpolates them raw into a BRE. It then scp's the transformed text to a deterministic /tmp spool as the login user and installs it over /etc/security/limits.conf with an elevated sh -c (chmod 0644, spool removed). The log shows the family's two privilege levels in one request: an unprivileged $ scp <limits.conf> -> ... line followed by $ sudo sh -c '...'. The chmod 0644 reset matters — pam_limits reads the file as every user, so a login-user mode would silently stop the limits applying. Re-running the script is a no-op once the file carries the line (the content-level fixpoint), and value accepts both an int and a symbolic string such as "unlimited". Every request carries when=is_centos, the static-inventory guard, so hosts outside the "centos" group are skipped without connecting.

server_security_limit

Example: ensure a resource limit (baseline server.security_limit parity).

This is a usage example, not part of the library. It shows reemote's server.security_limit operation, baseline server.security_limit parity for /etc/security/limits.conf: the deploy ensures every guarded host carries * soft nofile 1024, under sudo because the limits file is root-owned and pam_limits reads it as every user.

server.security_limit is a spool-and-install operation, and it replaces the earlier files.edit + demo-side limits_install example recipe that used to live in this file: the fact read, the line edit and the install now happen inside one operation, so a single deploy suffices. In one request it reads the raw /etc/security/limits.conf text through the AuthorizedKeysFile content fact, applies the baseline's anchored files.line lookup on the controller (the value is NOT part of the pattern, so a stale value is normalized in place), scp's the transformed text to a deterministic /tmp spool (the transfer rides the login user -- transfers never elevate), and installs it over /etc/security/limits.conf with an elevated sh -c (chmod 0644, spool removed). The log shows the family's two privilege levels: an unprivileged $ scp <limits.conf> -> ... line followed by $ sudo sh -c '...'.

The operation is idempotent at the content level -- the baseline's files.line no-op when a matching line already reads exactly the replacement: a host whose limits.conf already carries the line plans no change, and a second run is a no-op everywhere. One documented deviation: the desired content always ends in one trailing newline, so a file missing its final newline is rewritten once even when the line itself is correct. The match regex escapes domain, type and item literally (re.escape), the hardening over the baseline's raw BRE interpolation; an item prefix like nofilex matches (the baseline's .*$) while a commented-out line does not (the anchor). The value accepts both an int and a symbolic string such as "unlimited" (the baseline types it int but its own fixture passes "unlimited").

Every failure mode of the run -- an unreachable host, a rejected elevation, or an install step that exits non-zero -- reaches main as a typed error whose message already names the host, so main catches them and prints one line each instead of a traceback.

Unlike the pytest suite -- which injects the test framework's SimulatedHost via run's host_factory and never touches the network -- this example uses the default factory (Host.from_inventory): each plain Host lazily opens an asyncssh connection, reads /etc/security/limits.conf for real, and installs the limit under sudo where needed.

Every request carries when=is_centos, the declarative eligibility guard: the predicate reads only static inventory data -- host.group, set by the "group" key of the inventory entry -- and is a stable module-level function, so the prepare and apply phases reach the same skip decision. The driver skips a rejected request without materializing it, so on hosts outside the "centos" group neither the commands nor the facts behind them run, and plan and log render the step as [skipped: when guard is_centos] in its lockstep slot.

The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file (or edit inventory.json) to target your machines.

Run it with::

uv run python examples/modules/server/server_security_limit.py

Starting a service and waiting for it

The waiting design in action: server.service starts and enables httpd under sudo (idempotently — it reads the unit's ActiveState/UnitFileState and yields systemctl commands only where the unit differs), then server.wait_service yields a WaitForFactCommand that the driver polls over SSH every two seconds until SystemdServiceState reports active, or raises WaitTimeoutError after sixty. The wait is always planned as a change — like the baseline's server.wait it reads no facts, so the request sequence is identical in both phases; whether the unit is already active is discovered by the first poll. Both hosts finish the start step before either polls, and the two poll loops then run concurrently.

wait_for_httpd

Example: start httpd on real CentOS hosts, then wait until systemd reports it active.

This is a usage example, not part of the library. It shows reemote's second kind of non-shell operation: server.wait_service does not run a command string, it yields a WaitForFactCommand that the driver polls -- re-reading the SystemdServiceState fact over each host's asyncssh connection every interval seconds until the is_active predicate holds or timeout expires (raising WaitTimeoutError and aborting the run).

The deploy is the two-step shape from the baseline's systemd.service + server.wait idiom. server.service is the idempotent half: it reads the unit's ActiveState and UnitFileState and yields systemctl start / systemctl enable only when the unit differs from the request (so the second run plans no change). It needs root, hence sudo=True. server.wait_service is the non-idempotent half: like the baseline's server.wait it reads no facts and is always planned as a change -- whether the unit is already active is discovered at apply time by the driver's poll loop, never at plan time. Deciding at plan time would read pre-deployment state in the prepare phase and post-start state in the apply phase, and branching on that would trip the determinism guard. The wait itself is unprivileged (systemctl show works as the login user), so it carries no sudo flag.

Every failure mode of the run -- an unreachable host, a rejected elevation, a systemctl that exits non-zero, or a wait that times out -- reaches main as a typed error whose message already names the host, so main catches them and prints one line each instead of a traceback.

Because the apply phase runs each step on every host concurrently and joins before the next one, both hosts have finished systemctl start before either begins polling, and the two poll loops then run in parallel; a slow host only delays the step barrier, not the other host's polling.

The example assumes httpd is installed on the hosts (the dnf base example examples/modules/dnf/dnf_packages.py installs it).

Unlike the pytest suite -- which injects the test framework's SimulatedHost via run's host_factory and never touches the network -- this example uses the default factory (Host.from_inventory): each plain Host lazily opens an asyncssh connection, reads the unit state for real, runs systemctl under sudo, and polls until httpd is active.

Every request carries when=is_centos, the declarative eligibility guard: the predicate reads only static inventory data -- host.group, set by the "group" key of the inventory entry -- and is a stable module-level function, so the prepare and apply phases reach the same skip decision. The driver skips a rejected request without materializing it, so on hosts outside the "centos" group neither the commands nor the facts behind them run, and plan and log render the step as [skipped: when guard is_centos] in its lockstep slot. The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file to target your machines.

The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file (or edit inventory.json) to target your machines.

Run it with::

uv run python examples/modules/server/wait_for_httpd.py

Waiting for a port to listen

The same waiting design, asked of the kernel instead of systemd: server.wait yields the very same WaitForFactCommand — its port argument is the baseline's only option — but polls the Port fact until its listening flag turns true. The deploy starts httpd under sudo, then waits up to forty-five seconds for TCP port 80 at a one second interval — a port on its way up usually appears within a second, where a unit's start takes longer, which is why the wait's default interval is 1.0 rather than wait_service's 5.0. Reading the socket table is the stronger claim of the two: a unit can report active while its daemon is still binding, and a service that systemd does not manage opens its port all the same. Like every wait the request reads no facts and is always planned as a change, so both phases yield it identically. The poll runs unprivileged (ss -ltn needs no root), and Port therefore keeps presence in its own listening flag instead of inferring it from the owning process — an unprivileged listing shows the socket but not process/pid, and is_listening reads the flag alone. protocol="udp" waits the other family; a port outside 1-65535 or an unknown protocol raises at plan time, before anything is sent. The script is named server_op_wait.py to keep it distinct from the read-only server_port.py fact example.

server_op_wait

Example: start httpd on real CentOS hosts, then wait until port 80 is listening.

This is a usage example, not part of the library. It shows the port half of reemote's second kind of non-shell operation: server.wait runs no command string, it yields a WaitForFactCommand that the driver polls -- re-gathering the Port fact over each host's asyncssh connection every interval seconds until the is_listening predicate holds or timeout expires (raising WaitTimeoutError and aborting the run).

server.wait mirrors the baseline's server.wait(port=...), the only option upstream has. The baseline polls remotely: a while true shell loop over netstat with a fixed one second sleep and no timeout at all, so a port that never opens hangs the run forever. Here the wait is described, not scripted: the driver polls the fact and timeout bounds it, and interval is adjustable. The default interval is 1s rather than wait_service's 5s -- a port on its way up usually appears within a second, where a unit's start takes longer.

The deploy is the two-step shape from the baseline's systemd.service + server.wait idiom, and it pairs with examples/modules/server/wait_for_httpd.py: same start, two different readiness questions. server.wait_service there asks systemd whether the unit reports active; this asks the kernel's socket table whether anything is listening -- which is the stronger claim, since a unit can be active while its service is still binding, and a non-systemd service opens its port all the same. server.service is the idempotent half: it reads the unit's state and yields systemctl start only when the unit differs from the request (so the second run plans no change). It needs root, hence sudo=True. server.wait is the non-idempotent half: like every wait it reads no facts and is always planned as a change -- whether the port is already open is discovered at apply time by the driver's poll loop, never at plan time. Deciding at plan time would read pre-deployment state in the prepare phase and post-start state in the apply phase, and branching on that would trip the determinism guard.

The wait itself is unprivileged (ss -ltn works as the login user), so it carries no sudo flag -- and that is deliberate, because it also decides what the fact reports. An unprivileged listing shows the socket but not the process owning it, so Port keeps presence in its own listening flag instead of inferring it from an owner, and is_listening reads that flag alone. A port owned by a daemon the login user cannot inspect still satisfies the wait. A bad port or protocol (outside 1-65535, or not tcp/udp) is rejected by ValueError/TypeError at plan time, before anything is sent.

Every failure mode of the run -- an unreachable host, a rejected elevation, a systemctl that exits non-zero, or a wait that times out -- reaches main as a typed error whose message already names the host, so main catches them and prints one line each instead of a traceback.

Because the apply phase runs each step on every host concurrently and joins before the next one, both hosts have finished systemctl start before either begins polling, and the two poll loops then run in parallel; a slow host only delays the step barrier, not the other host's polling.

The example assumes httpd is installed on the hosts (the dnf base example examples/modules/dnf/dnf_packages.py installs it).

Unlike the pytest suite -- which injects the test framework's SimulatedHost via run's host_factory, seeds its listening_ports seam and never touches the network -- this example uses the default factory (Host.from_inventory): each plain Host lazily opens an asyncssh connection, reads the unit state for real, runs systemctl under sudo, and polls the socket table until port 80 answers.

Every request carries when=is_centos, the declarative eligibility guard: the predicate reads only static inventory data -- host.group, set by the "group" key of the inventory entry -- and is a stable module-level function, so the prepare and apply phases reach the same skip decision. The driver skips a rejected request without materializing it, so on hosts outside the "centos" group neither the commands nor the facts behind them run, and plan and log render the step as [skipped: when guard is_centos] in its lockstep slot. The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file to target your machines.

The path is resolved from __file__, so the example works from any working directory; point INVENTORY_PATH at your own file (or edit inventory.json) to target your machines.

Run it with::

uv run python examples/modules/server/server_op_wait.py