Lessons learned · Contribute

What went wrong while building Apelora, why, and the rule we follow now. Every coding agent reads this before starting a task (AGENTS.md says so) and adds an entry when something fails for a reason that was not obvious.

How to use this file

Before a task: read the sections that touch your area. The rules here come from real failures.
Add an entry when a check fails for a non-obvious reason, when a real system behaves differently from the mock or the documentation, when the owner corrects how we work, or when a fix needed more than one attempt.
Entry format: one bullet, **What happened** → cause → rule now, with the date (YYYY-MM-DD) and, where useful, the file or test that now guards it. Keep it to three lines.
Keep it useful: merge duplicates, and when a rule moves into code, a test or a skill, shorten the entry to a pointer. Delete entries that no longer apply.
Review it at the end of each phase: look for repeated causes, which mean a process or tooling gap. Fix the gap, not just the instance.

Patterns so far

Read these first. Most entries below come from one of four causes:

The mock or stub was kinder than the real thing. It answered instantly, had no limits and never failed. Model latency and limits in stubs, and test against a real local system before calling something done.
A failure was silent. Nothing in the logs or on screen said why. Every refusal and swallowed error gets a log line with identifiers, and the UI shows the server's reason.
Check the result, not the step. Wait for the observable result (an instance's location changed, the page shows data), not for "the job finished".
Environment differences between macOS and Linux, development and production builds, and mouse and touch input.

Front end and browser

The provider's sign-in code was exchanged twice in development and the sign-in failed (2026-10-21) → React StrictMode runs mount effects twice. The second run found the pending sign-in already used, failed, and triggered a request with no token. → Do one-shot side effects (code exchange, one-time tokens, POSTs on mount) once per page load through a shared promise (finishOidcSignIn in apps/fabric-console/src/App.tsx). Guard: the e2e test delays the token endpoint by 400 ms and expects exactly one exchange.
Stubbed tests passed while the real flow failed (2026-10-21) → the stub answered at once, so the race never happened. → Give stubs of external services realistic latency, and finish UI work with one run against a real local service (infra/dev, profile oidc).
"Your session ended" when the real cause was a refused token (2026-10-21) → the UI picked a generic reason. → Show the server's message when the cause is not the obvious one (fromProvider in App.tsx).
A Playwright test's second goto to another hash did not reload the page → hash-only navigation keeps the old route stubs and state. → Use separate tests for separate stub setups, or page.reload().
A stubbed identity provider failed in the browser → no CORS headers. Real providers send them for registered origins. → Stubs of cross-origin services send access-control-allow-origin.
Row actions pushed Delete off screen at 390 px; 24 px buttons in measurements → no wrapping, and the measurement ran without touch emulation, while the console makes controls 44 px on coarse pointers. → Use flex-wrap justify-end for action groups, and measure phones with isMobile and hasTouch on.
Screenshots caught dialogs mid-animation; Ctrl+click did nothing on macOS → use emulateMedia({ reducedMotion: "reduce" }) and ControlOrMeta.
The Studio showed the generic "saved sign-in was not accepted" after a refused provider token (2026-10-21) → two places ran the same detection, and the one that did not do the exchange reported last. → Keep facts about the page load (which token the provider issued) at module level, not in one call's local variable (providerToken in apps/studio/src/features/applications/backend.ts).
A docs agent captured screenshots while the same page was being redesigned in the main checkout (2026-10-21) → parallel work on one surface. → Do not run screenshot work for a page while it is being changed; regenerate shots after the design settles, in the same change as the design.
A strict content security policy reported eval in both applications although nothing called it (2026-10-21) → zod 4 probes for Function("") inside a try; the browser reports the attempt even though it is caught. → z.config({ jitless: true }) in packages/assembly-model/src/schema.ts; prove a policy by loading the production build under it (tests/e2e/web-serving.spec.ts), never by reading the code. The terminal emulator writes <style> elements at run time, so both sites' policies allow inline style elements (the Console opens terminals too since 2026-10-21).
A Playwright test counted three loads where it expected two (2026-10-21) → the dev server runs React StrictMode, which mounts views twice. → Count requests from a baseline taken just before the action (image mirrors show the mode… in tests/e2e/fabric-console.spec.ts).
A shared terminal component spent its single-use ticket on a socket React closed at once (2026-10-21) → moving the Studio's terminal into packages/terminal made it a component that mounts with the ticket, and React StrictMode in development runs a fresh mount's effect twice; the stub saw two sockets; against the broker the first would spend the ticket and the second would get 401. In the Studio's old code the effect ran on a prop change of an already mounted component, so it never double-ran. → Effects that spend something single-use start it on the next task and cancel it in cleanup (TerminalScreen); the Console spec counts exactly one socket per session.
Tests broke when a fixture list grew (permission catalogue counts) → expected when you add to a shared fixture: update the counts in the same change.
"One primary button" tests failed after adding dialogs (2026-10-21) → FormDialog renders its children while closed, so a closed dialog's primary button counts. → Render a dialog's form only while it is open ({open && <form>…}), which also makes autoFocus work on opening.
A new field in an API response broke stubbed pages (2026-10-21) → the stubs in tests/e2e/identity-fixtures.ts (and screenshots.spec.ts) return the old shape, so the page read undefined. → When a response gains a field the UI reads, add it to every stub of that route in the same change.
A new header chip pushed the platform badge under it at 390 px (2026-10-21) → the Studio header's left group was shrink-0 whitespace-nowrap, so it could only grow sideways. → Header groups wrap (flex-wrap); a Playwright test opens the chip's panel at 390 px (studio-capacity.spec.ts).
ConfirmDialog never focused its Cancel button although the code called focus() (2026-10-21) → daisyUI's modal is visibility: hidden during its opening transition, and a hidden element cannot take focus; prefers-reduced-motion did not change that. → Focus dialog controls after the modal is visible (the dialog retries on the next frames, components/ui.tsx); assert focus with toBeFocused() (fabric-console-operations.spec.ts).
A form dialog's autoFocus field was not focused when the dialog opened (2026-10-21) → the same cause as ConfirmDialog: the field mounts while daisyUI's modal is still visibility: hidden. → Focus the first field with a retry on the next frames (useFocusWhenShown, views/Alerts.tsx); tests assert toBeFocused().
A spacing measurement reported a 492 px gap between sections (2026-10-21) → the capture had been trimmed by dropping middle sections, which leaves their space behind. → Trim only a contiguous run of subtrees from the top, or keep the whole page; re-read any gap finding against the trim.
TypeScript refused views/account/index.tsx next to views/Account.tsx (2026-10-21) → macOS file systems ignore case, so ./views/account resolved to the file and tsc reported "differs only in casing". → Never name a folder like a sibling file; the Console's credential pages live in views/credentials/.
Every accessible-name check failed for "Sign out" buttons labelled "Sign rita out of this sign-in" (2026-10-21) → axe's label-content-name-mismatch wants the visible text inside the accessible name (voice control users say what they see). → Start an aria-label with the visible words: "Sign out rita, signed in …".
A ticking "checked 12 s ago" inside aria-live would be read out every few seconds (2026-10-21) → live regions announce every text change. → Announce state changes (errors, stale data), never clocks; see the fabric-console skill.
Grouping the navigation broke a phone test on another page (2026-10-21) → visible group headings and a taller header added 40 px above every page on phones, pushing the directory rows' actions out of the first screen that toBeInViewport checks. → On phones the navigation stays one wrapping row with the group names for screen readers only (App.tsx); run every console spec after changing the shell, not just the new page's.
The sectors test failed axe colour contrast (1.62:1) on a dialog heading, every run, with nothing changed in the view (2026-10-28) → axe ran right after Save, while the closing modal was still fading out, and measured the half-transparent text. → Wait for the dialog to be hidden (await expect(dialog).toBeHidden()) before an accessibility scan; scan a dialog only once it is fully open or gone.
Servers tabs rendered as plain text, then failed axe (2026-10-21) → daisyUI styles .tab only as a direct child of .tabs (an li wrapper broke it), and its inactive tab colour is below 4.5:1. → Tabs are <a class="tab"> directly inside the nav.tabs, inactive ones text-base-content/75 (views/servers/Servers.tsx; axe in fabric-console-servers.spec.ts).
A deployed application's canvas said "Mock · Design only" with no live state (2026-10-25, found through the screens on the lab) → only the deploy dialog started watching the status, and the stubbed spec deployed and opened the canvas in the same page. → The designer starts watching a deployed application itself (ApplicationRoute.tsx). Every stubbed flow that matters also gets a step in tests/lab/ui.spec.ts, which reaches the screen the way people do (a link, a reload, the next day).
A worktree's Playwright run would have tested the main checkout's code (2026-10-25) → reuseExistingServer accepts any server on 5173/5174, and a dev server from the main checkout was still running. → Check which directory the server runs from before trusting a run, and give separate runs separate ports (ui-e2e.sh does).
A lab test sent "volumes": null to the worker (2026-10-25) → spec["volumes"].as_array_mut() on a serde_json::Value inserts the missing key as null before reading it (mutable indexing creates the entry). → Read an optional key with get_mut(…), never value[key] on a mutable value.
A test timed out clicking a "Try again" that had gone (2026-10-25, only in the full suite) → one switch made both stubbed endpoints succeed, and the servers panel re-read on its own before the test clicked its retry, so the error panel and its button vanished. → Give each stub its own failure switch and flip it just before the retry that should succeed; a panel that polls can recover between two steps (fabric-console-health.spec.ts).
"That token was not accepted" vanished now and then (2026-10-29, 1 failure in the full suite, 2 in 15 under load) → the sign-in check and the overview's own request were both refused for the wrong token; whichever came second ran onAuth again, found the token already cleared and set the reason to a bare sign-in page. → A refusal keeps the reason already shown (setAuthNeeded((shown) => shown ?? reason), App.tsx). To catch an ordering race, delay one stub and wait for its answer before asserting (fabric-console.spec.ts, token mode); passing on a single rerun proves nothing.

Control plane and identity

The control plane refused a provider's token and logged nothing (2026-10-21) → refusals returned 401 without a log line. → Each refused credential is logged at warn as credential refused with method, path, credential kind, issuer and reason, never the token. It is not audited, since one row per bad request would let anyone fill the audit log.
A real provider issues 24-hour tokens; Apelora accepts at most one hour (2026-10-21) → provider defaults differ from our limits. → Document the limit next to the setting, configure the dev provider for one hour, and make the refusal reason visible.
A provider that is down at startup stayed refused until a restart (2026-10-21) → discovery ran only at startup and on save. → Retry unreachable providers every minute (auth_config::retry_unreachable).
An LDAP bind with an empty password succeeds (2026-10-21) → it is an anonymous ("unauthenticated") bind, not a failed one. → Refuse an empty password before connecting (ldap::sign_in; guarded by an_empty_password_is_refused_before_anything_is_sent).
ldaps://LDAP.Example kept its capitals after URL parsing (2026-10-21) → the url crate lowercases hosts only for special schemes (http, https, …), not ldap/ldaps. → Lowercase the host ourselves (directories::validate): the directory URL is its people's issuer, so its spelling must be stable.
The development directory's memberOf was empty (2026-10-21) → the memberof overlay does not run for offline slapadd. → The seed lists memberOf explicitly (infra/dev/directory/seed.ldif).
Any change to sign-in settings can lock everyone out → the API enforces the guards, not just the UI (lockout_guard, set_api_tokens, guard_leaving_local_accounts). Every new way in (local accounts, directories) must be counted by all of them.
extract(epoch …) came back as NUMERIC and failed to decode as f64 (2026-10-21) → PostgreSQL 14+ returns numeric. → Cast to ::float8 in queries that sqlx reads into floats.
A real control plane on the shared development database would have broken other checkouts (2026-10-21) → sqlx refuses to start when the database has a migration the binary does not know, and parallel sessions share apelora on port 15432. → For real end-to-end runs of a branch with a new migration, create a separate database on the same server (docker exec apelora-dev-db psql -U apelora -c "CREATE DATABASE …"), point DATABASE_URL at it, and drop it afterwards.
A new HealthState schema silently replaced the instance HealthState in the OpenAPI document (2026-10-21) → utoipa names schemas by the Rust type name only, so two types with one name collide without an error. → After npm run api:generate, check git diff packages/api-client/src/generated.ts for removed lines; give new schemas a distinct prefix (Health*, Capacity*).
Two people disabling the last two super admins at once both succeeded (2026-10-21) → the "last one" guard counted first and changed afterwards, outside a transaction; it also counted assignment rows, not enabled people. → Guards that keep "at least one" run the change and the count in one transaction under an advisory lock (identity::change_keeping_a_super_admin; containment_tests.rs races two requests).
A token or role could reach further than the person who made it (2026-10-21) → the escalation guard covered assignments only; tokens, revocation and legacy tokens (which skipped the route gate) had their own, weaker checks. → Every path that mints a credential or removes access applies the same containment (beyond_caller, check_revocation), and every caller passes the route gate.
A security review found guards that covered one path to an effect but not its neighbours (2026-10-25) → the escalation guard covered assignments but not joining a group or mapping one; group names matched without their source; the sign-in limit read the socket peer while the rate limiter believed trusted proxies; a terminal re-check skipped the sign-in that opened it. → When adding or reviewing a guard, list every path to the same effect (granting a role, naming the client, ending access) and give each the same check and its own test (containment_tests.rs, ADR 0016).
A persistence function generic over impl sqlx::Acquire broke tokio::spawn of the supervisor with "implementation of Send is not general enough" (2026-10-21) → the borrowed-lifetime bound cannot be proven for every lifetime inside a spawned future. → Take &PgPool, or &mut PgConnection for work inside a transaction (enqueue_job_in(&mut tx, …)); impl PgExecutor for single statements is fine.
A provider save audited reachable, which is only known after the settings reload (2026-10-21) → once the audit row moved into the change's transaction, the reload and the probes could only run after the commit. → In-transaction audit rows carry only what is known before the commit; probes run afterwards and their result goes in the response (auth_config.rs, directories.rs).
Worker test run_stops_on_shutdown failed now and then (2026-10-21) → a fixed 100 ms shutdown raced the heartbeat task when the machine was busy. → Wait for the observable condition (the worker's row) before shutting down (services/deployment-worker/src/lib.rs).
Parallel sqlx::tests asserting on a global metrics gauge would race each other (2026-10-21) → gauges are process-wide statics and tests run at once. → Assert on what the function returns (outbox::Pass); check the family's presence in the listener test (deployment-worker/src/metrics.rs).

npm run api:generate failed again for the first file download (GET /support-bundle, a tar) (2026-10-21) → the generator knew JSON and text only. → application/x-tar, application/zip and application/octet-stream map to Blob, and the client returns res.blob() (client.test.ts).
Widening the secret scan to every Apelora token prefix flagged an existing test's all-zero aps_ token (2026-10-21) → the scan greps source text. → Tests build token- and key-shaped values at run time (format!("aps_{}", "0".repeat(64)), a PEM label joined from words), so the scan stays strict.
Two handlers made #[derive(OpenApi)] fail with "storage_pools_config is defined multiple times" (2026-10-21) → utoipa names a path's generated config after the module path joined with _, so storage::pools collided with the existing storage_pools handler. → Name new handlers so module_fn cannot equal an existing handler's name (storage::pool_usage).
A generic schema name (Comparison) went into the OpenAPI document (2026-10-21) → utoipa names schemas by the Rust type only; a generic name would later collide silently. → Domain enums exposed through the API carry their area's prefix (AlertComparison, AlertSeverity); check generated.ts for generic new names, not only removed lines.

npm run api:generate failed on the first route that answers text (GET /metrics) (2026-10-21) → the client generator accepted only JSON responses. → The generator maps a text/plain success to string and the client returns the text (packages/api-client/src/client.test.ts guards it). A new non-JSON route needs the same treatment.
Failed terminal opens were never in the audit trail (2026-10-21) → the audit table accepts the outcomes success, denied and failed; the code wrote failure, the database refused the row, and a let _ = swallowed the error. A mutation test and a test of the real database found it, not the compiler. → Write audit outcomes through one set of constants or an enum, never free text; never discard an audit result with let _ = (log it, or refuse the action); a test of every outcome against the real table (sessions_tests).
A limit that refused a request had already spent the single-use approval it came with (2026-10-21) → the approval was consumed before the per-person cap was checked, so a person had to ask another person again. → Reserve the slot first, spend the approval last, and give it back (discard the reservation) if a later step fails (a_refusal_by_a_limit_does_not_burn_the_approval).
A session's timers and its terminate button stopped working while a client was stalled (2026-10-21; found by the security review, not a test) → the loop awaited a send to a browser that had stopped reading, and nothing else in it ran. → Every send in a long-running loop has a timeout, limits are polled first (biased), and the slot is released by a drop guard however the task ends.
The first review of the terminal feature found six holes the tests did not (2026-10-21) → the tests proved what the author thought of. → A read-only security review by a separate agent runs before a security feature is committed; give it the contract (secure-remote-access) and ask it to find what the code does not do.

Incus (REAL INCUS TESTED behaviour that the documentation did not make obvious)

apelora-dns exited at once inside Debian and systemctl is-active returned 3 (2026-10-23) → the stub listener holds 127.0.0.53:53, and the kernel then refuses 0.0.0.0:53 (Address in use). The binary was fine; a direct run before the stub started was fine. → Turn the stub listener off in that instance (DNSStubListener=no), restart systemd-resolved when it is active, then start the unit. Run reset-failed first and ignore its status: a unit that has never failed makes it exit 1, and a crash loop otherwise hits the start limit so a later restart is refused even after the port is free (install_dns_resolver).
A later network.sync could not update the resolver: text file busy (2026-10-23) → the first attempt had already started the binary, then waited on another instance, and the retry opened that path with truncate. → Write apelora-dns.new and mv -f it into place; the running process keeps the old inode until systemctl restart.
The service name resolved and the instance name stopped after the client restarted (2026-10-23) → Debian's resolv.conf is the stub 127.0.0.53, so the resolver forwarded to itself, and dns.domain replaced the incus search domain the instance name lives in. → Drop loopback forwarders, read /run/systemd/resolve/resolv.conf for the real servers, and set dns.search to <application>.internal,incus.

Waiting on a copy operation was refused for the restricted identity ("permission for project default") → the operation is created in another project. → Fallback: poll the project's running operations for the names, then check that the target exists (copy_volume, project_operation_running).
Copied volumes lost their size and user.* labels → set them explicitly on the copy.
A refresh copy deletes the target volume's own snapshots and is accepted while the instance runs. Plan restores around that.
Moving an instance was refused → the project needs restricted.cluster.target=allow (now in scripts/test-cluster/up.sh).
Attached custom volumes do not follow a live-migrating VM unless marked dependent, which ties their lifetime to the instance. Design moves with that in mind (docs/design/mobility-and-ha.md).
A module's IPv6 traffic was refused although its connection was drawn (2026-10-22) → Incus gives an OVN network an IPv6 subnet even when the uplink has none, and an instance gets its IPv6 address about a second after IPv4. network.sync filled the address set as soon as any address existed, so the set held only IPv4 and nothing synced again. → When the network has ipv6.address, the sync waits for each reachable instance's IPv6 address (a_sync_waits_for_an_ipv6_address_that_comes_after_ipv4; scripts/test-cluster/ipv6.sh). After 60 s with IPv4 and no IPv6 the set keeps IPv4 and the sync succeeds (a_sync_stops_waiting_for_ipv6_that_never_arrives); otherwise a guest that never configures IPv6 fails forever and the job-failures alert stays on. Check every address family a network hands out, not only the one the uplink uses. Check every reachable instance in a replica slot, not the slot as a whole.
An IPv6 /64 cannot be a service range (2026-10-22) → enumerating 2^64 addresses is impossible, and IPv6 has no broadcast so the last address is usable. → APELORA_SERVICE_ADDRESSES accepts IPv4 /16–/30 (skip network and broadcast) or IPv6 /112–/126 (skip the subnet address), including comma-separated ranges of either family; each module still gets one address. Backends must match the listen family (/32 vs /128). Guard: service_ranges_skip_network_and_broadcast_and_reject_odd_input, a_sync_puts_ipv6_backends_behind_an_ipv6_service_address.
Giving the lab uplink (incusbr0) an IPv6 address wedged new OVN network creates (2026-10-22) → incus network set incusbr0 ipv6.address=fd42:198:18::1/64 on the live four-member cluster made incus network create … --type=ovn hang with no operation; systemctl restart incus is the same trap as before (SIGKILL incusd then start recovered). IPv6 service addresses only need ipv6.routes and restricted.networks.subnets. → Keep the lab bridge at ipv6.address=none; route fd42:198:100::/112 (ovn.sh). Guard: scripts/test-cluster/ipv6.sh.
mapfile is not on the macOS bash that runs lab scripts (2026-10-22) → macOS ships bash 3.2; mapfile is bash 4. Scripts that run on the workstation (not inside the VM) use a while read loop (scripts/test-cluster/ovn-ha.sh).
PEM files cannot be arguments to limactl shell (2026-10-22) → newlines split the remote command. Copy the files onto the VM and incus config set …="$(cat …)" there (ovn-ha.sh).
Debian's ovn-central.service is a dummy oneshot (ExecStart=/bin/true, 2026-10-22) → the databases are ovn-ovsdb-server-nb, ovn-ovsdb-server-sb and ovn-northd. systemctl enable --now ovn-central does not restart them. Package install starts empty local DBs before join options are written. → Write /etc/default/ovn-central first, restart those three units, and wipe the joiner DBs (ovn-ha.sh).
macOS LibreSSL openssl x509 -req signs with SHA-1 (2026-10-22) → Debian OpenSSL 3 refuses the leaf (ca md too weak) and OVN never loads TLS. The CA was SHA-256 because req -new -x509 defaults differently. → Always pass -sha256 when signing leaves (ovn-ha.sh issue).
A single-node OVN Connection of ptcp:<ip>:6641 is replicated to every raft member (2026-10-22) → joiners try to bind the first member's address and never listen on 6641/6642. → After the cluster forms, ovn-nbctl set-connection pssl:6641 and ovn-sbctl set-connection pssl:6642 (no IP: each member listens on all interfaces).
gh attestation verify printed the command list and every archive looked unverified (2026-10-26, Ubuntu's gh 2.45) → the subcommand arrived in 2.49. → Check the gh version first; with an old one, query the attestations API per digest (RELEASING.md), and say that the signature was not checked.
The mock copy ignored fail_create; the mock move did not carry used_by → when the real adapter gains a behaviour, give the mock the same behaviour, failures included, in the same change.
A real-move test passed too early → it waited for the job, not for the result. → Wait until the instance's location has changed.
Two "memory used" figures disagree by gigabytes (2026-10-21, REAL INCUS TESTED) → /1.0/cluster/members/<m>/state gives total and free RAM (caches count as used), /1.0/resources?target=<m> gives used without caches. → Show the /1.0/resources figure (HealthServerResources.memory).
A shared pool (clustered LVM, Ceph) appears in every member's state with the same space (2026-10-21) → it is one pool. → Count it once (capacity::pool_is_shared); summing per member multiplied the test cluster's free space by four.
The restricted deploy identity sees only its own projects' instances → per-server instance lists from it are partial. Say so on screen (Health → Instances) instead of implying the server is empty.
Cluster settings came back empty for the deploy identity (2026-10-21, REAL INCUS TESTED read) → Incus answers GET /1.0 with config: {} to a project-restricted certificate; the admin identity sees it, together with secrets and the placement scriptlet. → Read settings with the admin identity and return only allow-listed keys (server_config(keys), CLUSTER_SETTINGS); defaults come from /1.0/metadata/configuration, which the deploy identity can read.
A virtual machine listed Running is not ready for incus exec (2026-10-23, VM recovery drill) → Incus reports Running when the VM process is up, before the guest agent answers. The first drill aborted on set -e at incus exec … cat (the helper discards stderr), and the exit trap deleted the volume before the marker could be read. → Wait until incus exec … true succeeds, and capture a failed read (|| true) so the script can say FAIL before the trap runs.
Incus refuses an instance over a project limit with HTTP 500, not 403 (2026-10-21, REAL INCUS TESTED) → the limit check runs inside the create and is reported as an internal error ("Reached maximum number of instances", "Reached maximum aggregate value … for limits.cpu"). → Never map a limit refusal by status code; Apelora checks quotas itself before queueing (tenancy::check_deployment), so the Incus refusal is only the backstop.
A project's state comes back as Limit/Usage, capitalised (2026-10-21) → Incus's ProjectStateResource has no JSON tags, unlike the rest of the API. → Decode both spellings (WireResource in crates/incus-adapter/src/tenancy.rs); read wire shapes from the pinned source, not from the YAML example.
A stand-by reports database: true (2026-10-21, REAL INCUS TESTED read) → the flag means "holds a copy", not "votes". → Voters are members with the database role (clustering::database_role).

"The SAN's lock stops a second writer" is only half true on clustered LVM (2026-10-21, read in the Incus 7.0.2 source) → lvmcluster activates container file systems exclusively but VM disks and block volumes shared (so they can move live); a shared lock does not exclude another host. → Fence on the failed host's sanlock lease being DEAD (its watchdog has fired), never on time or on a lock refusal (ADR 0013, apelora_domain::fencing).
The restart failed with "LV locked by other host" right after a member reported the failed host DEAD (2026-10-21, REAL INCUS TESTED) → each member's sanlock judges a lease dead from its own readings, up to about 10 s apart, and the target member had not yet. → The instance.recover job waits (Outcome::Waiting, no attempt used) for up to 120 s after fencing; the lock refusal is the storage protecting the data, not an error to escalate.
Incus reported all four members Offline for 10 s while only one VM was stopped (2026-10-21, REAL INCUS TESTED) → the heartbeat round was held up by the dead member. → Nothing on shared storage moves on Incus's status alone; the lease check kept every healthy member unfenced.
An offline member's instances cannot be read and cannot be moved to a group (2026-10-21) → Incus lists them without config, and with the source offline the move request is forwarded to the target, which must be a member. → Recovery policy comes from Apelora's deployment records, and Apelora names the target member (recovery::choose_target); the placement scriptlet still validates it.
Rewriting every ACL on every network.sync dominated a 120-container run (2026-10-22) → PUT /1.0/network-acls/{id} took a mean 3–6 s once the cluster held about a hundred objects, and one worker never finished. Four workers got the instances Running, then every sync still failed its first attempts (20 s GET timeout, or POST "already exists" after GET 404). → Skip ACL writes whose rules already match (a_sync_writes_only_the_acls_whose_rules_changed). Treat POST "already exists" as success when the object carries our marker (ensure_treats_already_exists_as_ours_when_the_object_carries_our_marker). Measure with scripts/test-cluster/measure.sh; figures in docs/design/capacity-lab-sizing.md.
A mock-verified resize failed on the lab's first attempt (2026-10-21) → PATCH /1.0/instances/{name} on a running container is answered synchronously, but the adapter expected an asynchronous operation for every write. The job's retry hid it: the second attempt saw nothing to change. → A real run shows retries: read the test's output for will be retried lines, not just the result. Calls that may finish at once return None for the operation (start_operation_or_done).
The substitute replica ran, yet half the requests to its service address failed for the whole outage (2026-10-25, REAL INCUS TESTED) → Incus refuses every load balancer create, update and delete while any member is unreachable: it notifies all members with NotifyAll and reverts on failure (driver_ovn.go, cluster/notify.go, 7.0.1 and 7.0.2). Retrying cannot get through, and OVN's health check kept the dead backend. → When a retry never succeeds, read the upstream code path before tuning retries. The runbook tells the operator to bring the member back or force-remove it, and the upstream report is a maintainer's to file. A passing drill (2026-10-20) had hidden this.

The supervisor "restored" a replica seconds after it was deployed (2026-10-25, found by a lab run) → a new instance is Stopped in Incus until its instance.create job has started it, and the supervisor checked only for an instance.start job in flight. → A stopped replica with a create job in flight counts as still being created (a_replica_still_being_created_is_not_started_by_the_supervisor). When a state can be reached two ways, check every job that leads to it.

Lab, shell and tooling

A new member of a sized loop pool needs the size and an empty source (2026-10-24) → the join that brought the second, third and fourth members online answered both: size (12GiB) and source empty, so Incus created the loop file. A preseed with only the address and the token does not answer those questions. → scripts/enrol/enrol-member.sh --pool-size 20GiB writes both for the pool named local. A disk pool still uses --member-config. This switch has not been run against a cluster; the keys are the ones that join used.
A moved replica never rejoined its service address (2026-10-26, evacuation drain on the lab) → after an evacuation the container ran on its new member with its IPv6 address but no IPv4 lease for a few seconds; the sync that ran then skipped it (no address of the service's family) and still succeeded, and nothing changed afterwards to sync again. → A running replica without its address makes the sync retry (services in network.rs, test a_running_replica_without_its_address_yet_is_waited_for_not_left_out). When a sync skips something silently, ask what will make it look again.
The evacuation check counted failures the step before had caused (2026-10-26) → after the member-failure step the load balancer still pointed at the retired substitute until a refused sync went through, while Apelora's own port check already said both replicas were healthy. → Start a measurement from what the data path reports (OVN backend health from the load balancer's state), not from the application status.
grep with no match ended a set -euo pipefail script (2026-10-26, demo.sh) → failed=$(grep -o FAIL file | wc -l) fails the pipeline when nothing matches, which is the success case. → Wrap a grep that may find nothing: { grep … || true; } | ….
The first install showed an old sign-in page (2026-10-24) → nothing newer than draft v0.1.0-alpha.1 (20 October) had been tagged, so the installer unpacked that package. Its screens predate the sign-in page, and its control plane has no GET /api/v1/sign-in. → Install v0.1.0-alpha.2 (the default in scripts/test-cluster/install-units.sh). That tag is main, so the control plane, the worker and both screens are one tree. Leave v0.1.0-alpha.1 unpublished.
Incus installed on the first new guest and failed on the other three (2026-10-24) → those guests were still running their first-boot package setup, which held /var/lib/apt/lists/lock. The installer called apt-get once and exited. → scripts/test-cluster/install-incus.sh waits for cloud-init, then retries apt while the lock is held.
Copying a file to several new hosts at once did not record their host keys (2026-10-24) → OpenSSH rewrites known_hosts through a temporary name, and the parallel copies collided (known_hosts.old: File exists). → Add new hosts one at a time, or give each copy its own UserKnownHostsFile.
A storage helper said creating a volume timed out, and the volume was empty (2026-10-24) → the helper returned after the volume existed and before the image was copied (a 3 GiB volume holding 56 KiB). Copying the image into that volume worked. → After that error, check how much of the volume is used. If it is empty, copy the image into it. Do not run the helper again; it tries to create the same volume.
npm run lint failed with "multiple candidate TSConfigRootDirs" (2026-10-21) → agent worktrees live under .claude/worktrees/ inside the repository, and ESLint scanned their copies. → .claude/** is in the ESLint ignores; any new repo-wide tool must ignore it too.
"cargo run gives development mode" was wrong inside the repository (2026-10-21) → .cargo/config.toml sets DATABASE_URL for every cargo command. → Development mode is DATABASE_URL= cargo run -p apelora-control-plane; check tool-level config before documenting a default.

A worktree with a new migration must not use the shared development database (2026-10-21) → once a worktree applies a migration the main checkout does not have, the main checkout's control plane refuses to start (sqlx: an applied migration is missing from its sources). → For real-stack runs in a worktree, create a separate database on the dev server (CREATE DATABASE apelora_<topic>) and drop it afterwards; #[sqlx::test] already uses fresh databases.
Dev servers on 5173/5174 belong to whichever checkout started them (2026-10-21) → Playwright reuses them, so a worktree's run would test the main checkout's code. → In a worktree, run Playwright with a temporary config on other ports (e.g. --port 5274) and remove it afterwards.
Lab scripts broke on macOS → /usr/bin/env bash is bash 3.2, which has different quoting and breaks on empty arrays. → Lab scripts use /opt/homebrew/bin/bash.
The ops self-test failed on macOS while the scripts passed by hand (2026-10-21) → it ran them through their #!/usr/bin/env bash shebang, which is bash 3.2. → Operations scripts (scripts/ops/) must also run on Linux servers, so they keep /usr/bin/env bash, refuse bash < 4 with a message, and tests call them as "$BASH" script.
A restore drill that passed on the host failed as soon as the client tools ran in a container (2026-10-23) → psql -f /path/on/the/host names a file the container cannot see, and the error is only "No such file or directory". → Stream the SQL on standard input (psql -f - <file). APELORA_PG_CLIENT_IMAGE never receives a host path.
A base backup waited 60 seconds for a write-ahead segment that recovery later found (2026-10-23) → pg_walfile_name(pg_current_wal_lsn()) names the segment still being written. pg_basebackup had already archived the segments the backup needs and said so. → Read the file name from backup_label, and check that file is in the archive. Do not wait for the segment that is still open.
The archive proof passed and the CI step still failed (2026-10-23) → pg_basebackup inside the container leaves the backup directory mode 0700, owned by uid 999, and rm -rf by the runner exits non-zero. A trap that ends on that rm turns a passing proof into a failed job. → Open the mode from a root container (chmod -R a+rwX) before removing the directory, and end the trap with an explicit exit.
postgresql-client-18 was installed and pg_dump --version still was not 18 (2026-10-23) → /usr/bin/pg_dump is the pg_wrapper, and with no local cluster it does not select the client just installed. The binaries are /usr/lib/postgresql/18/bin/pg_dump. → Point the service at those files (a link in /usr/local/bin, which systemd searches first). Do not trust pg_dump --version until you have run that path.
verify.sh printed VERIFIED and exited 1 (2026-10-21) → its EXIT trap ended with [[ -n $TMP ]] && rm …, which returns 1 when there is nothing to remove, and a trap's status becomes the script's. → End traps and functions with if …; then …; fi, and check the exit status in script tests, not just the output.
nftables rules that looked right were refused (2026-10-21) → fwd is an nft keyword (not a chain name) and comment must end a rule. → Lab scripts that generate rules validate them with nft --check in plan (scripts/test-cluster/egress.sh), so the dry run fails instead of block.
A plain sleep was blocked in the agent shell → wait with until <condition>; do sleep 1; done.
An edit piped as python3 /dev/stdin <<EOF changed nothing and said nothing (2026-10-21) → in the agent shell the heredoc did not reach the interpreter; the file stayed as it was and a later test failed for a reason that looked unrelated. → Put edit scripts in a file in your scratchpad folder, run that, and check the change with grep before moving on.
In an agent worktree, source ~/.config/apelora/test-cluster/env and multi-step shell lines were refused (2026-10-21) → the worktree guard cannot prove such commands stay inside the worktree. → Put lab runs in a script in your scratchpad subfolder that exports the variables itself and run it with /opt/homebrew/bin/bash <script>; do edits with the editor or a script file, one plain command per call.
The Incus CLI attach failed → it needs a device name.
A design-review tool refused SVGs with text → review illustrations by rendering them and looking at screenshots instead.
Another agent overwrote a helper script in the shared scratchpad mid-task (2026-10-21) → parallel sessions share the scratchpad directory. → Keep your helpers in your own subfolder of the scratchpad, named after your worktree.
The fence probe stopped working on a member after its VM was restarted (2026-10-21) → Lima gives a VM new SSH host keys at every boot, and the probe pins host keys (as it must). → After restarting a lab VM run scripts/test-cluster/fence-probe.sh apply again; ha-drill.sh does it after the drill and its plan refuses a lab where a probe fails.
Commands with a cd, a loop or a heredoc were refused in an isolated worktree (2026-10-21) → the worktree guard cannot prove such commands stay inside the worktree. → Put multi-step shell work in a script file in your scratchpad subfolder (absolute paths) and run that one file; keep direct commands plain.
A spacing measurement reported a 92 px gap that did not exist (2026-10-21) → the snapshot had been trimmed to fit the tool call and lost a sibling. → Trim whole subtrees (e.g. keep <main>), never single siblings, and re-check any finding against the full capture.
IPv6 loopback was warned as a remote OTLP receiver (2026-10-23) → comparing host_str() to "::1" missed the form the URL crate actually returns. → Classify loopback with IpAddr::is_loopback plus localhost (any case). Guard: traces_endpoint_accepts_loopback_http_and_rejects_other_schemes.
Stryker's Vitest plugin reported every mutant as surviving (2026-10-21) → with Vitest 5 the plugin's per-file activation hook never ran, so mutants were never switched on, and a copied sandbox imported the original workspace packages through node_modules symlinks anyway. → stryker.config.mjs uses the command runner and mutates in place; check a new mutation setup by breaking one line by hand and seeing it killed before trusting any score.
A killed Stryker run left 240 source files rewritten (2026-10-21) → in-place mode injects // @ts-nocheck and restores from a backup only when it exits cleanly; the session ended mid-run. → Start from a clean git status, never edit apps/ or packages/ while it runs, and after an interrupted run git checkout -- <the mutated paths> after checking git diff shows only injected lines.
cargo mutants reported timeouts on mutants that were fine (2026-10-21) → the first build of a fresh copy takes minutes, so the per-mutant timeout (150 s) counted build time. → Give it --build-timeout and a test timeout well above one test run, run one scope at a time (they compete for CPU), and re-run only the missed functions with -F after adding tests.
A clippy lint appeared on code that passed an hour earlier (2026-10-21) → the toolchain moved (ok().expect(), sort_by). → Run cargo clippy --workspace --all-targets -- -D warnings again right before committing, not only after the last edit to that file.
Test code was flagged by clippy::unwrap_used and by code scanning (2026-10-21) → the workspace denies unwrap outside tests only by convention, and CodeQL reads inline mod tests with fixed keys as hard-coded secrets. → Test fixtures live in tests.rs/*_tests.rs (left out of scanning) with #![allow(clippy::unwrap_used)] and a reason; an inline test module with a fixed secret moves out.
A CLI that prints a one-time token cannot satisfy the cleartext-logging query (2026-10-21) → writing the secret straight to stdout instead of through println! only moved the finding. → Do not rewrite code to hide a deliberate flow from a scanner; decide to dismiss with a written reason, or change the design (for example write to an owner-only file).

clippy::large_enum_variant flipped between failing and being an unfulfilled expect as the mock gained fields (2026-10-21; Provider in the adapter) → which variant is larger changes whenever either side gains a field, and an unmet #[expect] is itself an error. → Use #[allow(..., reason = ...)] for a lint whose trigger depends on sizes.
Every sqlx::test failed with PoolTimedOut (2026-10-21) → the dev PostgreSQL container had stopped after a restart, and the tests give no hint. → Run docker compose -f infra/dev/compose.yaml up -d and wait a few seconds before concluding a Rust test is broken.
A timing test (concurrent_callers_share_one_probe) failed in a full workspace run and passed alone, three times (2026-10-21) → it depends on scheduling and the full run loads the machine. → Re-run a failing test alone before debugging it; if it only fails under load, make it wait for a condition instead of time (not done yet for this one).
The adapter could not reach a lab MAAS that curl reached (2026-10-21) → the lab certificate came from openssl req -x509, which sets CA:TRUE; OpenSSL and curl accept that as a server certificate, the Rust TLS stack (rustls/webpki) refuses it ("CA used as end entity"), and the error said only "cannot connect". → Generate lab server certificates as leaves: -addext basicConstraints=critical,CA:FALSE -addext extendedKeyUsage=serverAuth; and test a TLS client against the real thing with the client you ship, not only curl.
cargo test --exact missing_name exits 0 (2026-10-22) → no test matches, nothing runs, the result is 0 passed; 0 failed; N filtered out. A CI step that lists names and greps only for a green job would hide a rename. → After the run, require test result: ok. N passed for the N names listed (real-incus.yml).
A new workflow never ran, and the pull request's checks looked green (2026-10-22) → it used ${{ runner.temp }} in job-level env, where the runner context does not exist; GitHub refused the file on push with a zero-second "failure" named after the path, and the pull request's check list simply lacked the job. → ci.yml runs actionlint on every change; locally, docker run --rm -v "$PWD:/repo" --workdir /repo rhysd/actionlint:1.7.7 before pushing a workflow, and look for the new job in gh pr checks, not only for red.
The first estimate for BMC fencing (40–60 s) was wrong (2026-10-21) → it ignored that clustered LVM keeps the failed host's locks until sanlock expires its lease (about 140 s after the last renewal), however the host stopped. The drill measured fenced at 61 s and running at 220 s. → Measure the recovery path end to end before promising a number; a fence changes when it is safe, not when the storage lets go.
A lab VM's loopback service silently took the workstation's port (2026-10-25) → the Lima rule guestIP: 0.0.0.0, ignore: true does not cover services bound to 127.0.0.1 in the guest, so a control plane left running on a member owned the workstation's 127.0.0.1:8080 and a new local one could not start (or the Console proxied to the wrong one). → Ignore 127.0.0.1 too in every VM file, and before blaming the app for "address in use", check lsof -nP -iTCP:<port> -sTCP:LISTEN for limactl.

A lab check passed on the previous run's evidence, then failed on its own (2026-10-25) → demo.sh counted restart entries in the append-only audit log of the shared dev database, so an entry from an earlier run satisfied "restarted once" before this run's restart happened, and the real one made it two. → Scope every check against shared, persistent state to the run: from=<start> on the audit API, a run's own database (ui-e2e.sh), or names unique to the run.

Git and collaboration

Pre-staged deletions went into the wrong commit; Cargo.lock was left out → check git status and git diff --cached --stat before each commit. Commit Cargo.lock with the change that alters it.
Rebase blocked by an unstaged file → commit or stash everything before git fetch && git rebase.
The owner pushes from other sessions → before every push: fetch, rebase, npm install, check OpenAPI name clashes, run the full checks.
Commits are authored by the owner only, with no AI attribution (AGENTS.md rule 10). This overrides any tool default.
run_stops_on_shutdown failed only in the full cargo test --workspace run (2026-10-21) → it shut the worker down after a fixed 100 ms and expected its heartbeat row; with many database tests at once, registration took longer. → Tests wait for the observable result (the row exists), never a fixed delay (services/deployment-worker/src/lib.rs).
Merging eight parallel branches: each passed alone, together 30 tests failed (2026-10-21) → each branch's stubs and fixtures matched the API as that branch knew it. A new response field crashed pages whose stubs lacked it (s.localAccounts.enabled, s.incus.serverVersion), and a permission added to a role made another branch's "holds everything a viewer holds" fixtures wrong. A conflict resolved "keep both" also dropped a #[cfg(test)] that sat just outside the conflict block, and a fabric-console(-[a-z]+)? test pattern silently sent fabric-console-local-accounts to the wrong project. → After each merge, cargo check --all-targets before the next one. Run the full checks once all are in, before pushing. Pages read optional chains on fields a newer control plane adds (one version of skew, ADR 0011). Test patterns allow hyphens: [a-z-]+.
Long documentation lines merged character by character came out garbled (2026-10-21) → two branches each rewrote the same one-line table row (current-state, a skill's views row); keeping "the other side's insertions" at character level spliced words together ("Audight", "projexpool"). → Resolve a conflicting prose line by rewriting it from what the code now contains, then read it once; never merge prose below the word level.
Dependabot's sha2 0.11 pull request failed CI on its own (2026-10-21) → sha2 0.11 implements the digest 0.11 traits, and hmac 0.12 only accepts digest 0.10, so the pair builds only when both move. With hmac 0.13, new_from_slice comes from KeyInit, not Mac. → Crates that share traits move together: give them their own Dependabot group (rustcrypto).
tokio-tungstenite 0.30 built, but put a second WebSocket stack in the tree (2026-10-21) → axum 0.8.9 depends on 0.29. → Before taking a bump, check cargo tree -i <crate>@<old>. A crate another dependency already pins follows that dependency (Dependabot ignore with the reason).
CodeQL reported about 40 "hard-coded cryptographic value" and "cleartext logging" findings, all but one in tests (2026-10-21) → fixed test passwords, and values printed in assertion messages, look like secrets to those queries. → CodeQL runs from .github/codeql/codeql-config.yml, which leaves test files out by name. A reported inline mod tests moves to <module>/tests.rs. The one finding in shipping code (token create prints the new token once) was dismissed with its reason.
CodeQL cleartext-transmission flagged Redfish Basic Auth (2026-10-22) → shipping code rewrote https:// to http:// for loopback tests (Mapping.plain), so the scanner treated the BMC password as going over HTTP. → Keep TLS on the shipping path (https_only(true), refuse a non-https URL before basic_auth); drive tests through a local HTTPS BMC with a CA:FALSE leaf and a CA file. Do not add an HTTP downgrade in shipping code to make tests easier.
CodeQL uncontrolled-allocation-size flagged Vec::with_capacity(rows.len()) on the jobs list (2026-10-22) → the store already pages, but the allocation took the Vec's length, which the query treats as user-controlled. rows.len().min(JOB_LIST_LIMIT) stayed tainted (alert #59). → Size with_capacity from the named constant alone (JOB_LIST_LIMIT), never from rows.len().
curl --cacert of Incus's cluster certificate failed on the members (OpenSSL 3) (2026-10-23) → the cluster cert is a self-signed leaf (CA:FALSE), and pinning server.crt missed: :8444 presents the cluster certificate, the same on every member. → Pin /var/lib/incus/cluster.crt's public key (--pinnedpubkey); --insecure without that pin is a bypass. Guard: scripts/test-cluster/metrics.sh.
apt-get install nginx started the default site on :80 before the example config was linked (2026-10-23) → systemctl enable --now nginx then left :443 closed even though nginx -t was clean. → Reload nginx after linking apelora.conf (scripts/test-cluster/install-units.sh).
README screenshots still showed the old module stripe after the look had been fixed (2026-10-21) → the screenshots were regenerated before the CSS change and nobody re-ran the script. → After any visual change to a shared component, regenerate docs/screenshots in the same change and look at the README image.
My local checkout was 57 commits behind origin/main and I nearly redid work that was already pushed (2026-10-21) → other sessions push to the same repository. → git fetch and read git log HEAD..origin/main at the start of a task and before saying something is not done; rebase before pushing.
A new environment variable failed an unrelated-looking test (2026-10-21; every_setting_the_services_read_is_listed) → every variable the services read must be listed in settings::SETTINGS for the Administration page. → Add a new setting to that list, .env.example and the runbook in the same change.

Model and catalog

A design note read as a list of remaining work (2026-10-24) → the page opened with a status line, slice numbers, and decision numbers, and the body was a backlog. → Say what the product does, in the present tense. Put decisions in one "See also". Put what has not been tried in a short limits section. Update the page when the behavior changes; do not leave the old plan in place.
CodeQL rejected the deploy-time environment key (2026-10-24) → the token was built with /^_+|_+$/, and a property id is catalog input, so a long run of underscores makes that expression backtrack. → Turn the id into a token with a loop (setupToken in packages/assembly-model/src/runtime.ts). Do not trim a repeated character with a regular expression on catalog text.
Every mon cable compiled to "module may reach the collector on 9100" (2026-10-21) → the compiler treated every output as the consumer and used one default port per protocol, but metrics are read (the collector connects) and almost no software serves them on 9100. → Metrics outputs reverse the direction and use the class's own port (ModuleClass.metrics, metricsEndpoint); a class offers mon only when a collector can read metrics text on a port (packages/module-catalog/src/metrics.test.ts). Check a protocol's direction and real port before adding it to DEFAULT_PORTS.
Grafana could not be cabled to Prometheus, and broker "consumer" outputs connected only to another broker (2026-10-25) → the manifests modelled data flow (Prometheus query out → Grafana ui in; Kafka out → consumers), but a cable means "the output side may reach the input side", so the compiled rules let Prometheus reach Grafana and never the reverse. A tcp peer output (CockroachDB) could not connect to a sibling at all, because tcp outputs feed only tcp inputs. → Model who opens the connection: clients on the output side (Grafana query → Prometheus api; producers and consumers → broker in), and make peer/repl/primary/bridge outputs match a service input of the same class (packages/module-catalog/src/model.test.ts, "connections between catalog classes").
Simple Icons marks are not all CC0 (2026-10-25) → its metadata gives some marks Apache-2.0, CC-BY(-SA) or custom trademark terms, and our attribution file said CC0 for all. → Read the license field for each mark, record it in assets/logos/ATTRIBUTION.md, and use only CC0 or permissive marks for new classes (a test checks every bundled mark is listed with a licence).
A deployable NGINX showed twelve settings and applied one (2026-10-25) → only the image choice reached Incus; the rest were saved in the design, and the deploy-time fields arrived as SETUP_*/SECRET_* variables that the image never reads. Nothing on screen said so. → A runtime maps settings explicitly (command, environment), appliedSettings says which, and the Studio marks the others. Before making a class deployable, list what its image really takes and prove it inside the instance (CHECKS in scripts/runtime-requests.ts).
A new storage key nearly overwrote the custom modules (2026-10-25) → the catalog menu's open state was first given apelora.studio.catalog.v1, which already holds the custom module classes. → Grep for a localStorage key before choosing one; the menu state lives in apelora.studio.catalog-menu.v1 and an e2e test checks the custom-class key is untouched.

Renames and compatibility

"an module" left behind after renaming "appliance" → the article sat before a line break. → Do renames with a regex that crosses line breaks, then grep .env.example, CITATION.cff, scripts and docs. Keep serde aliases and a migration for stored data.
A private struct still had the old field name → serde aliases go on every deserialised type, including private worker types.

Secrets and evidence

Tokens must never reach the transcript, logs or screenshots → serve test tokens to the browser from a git-ignored file, revoke them when done, and delete any screenshot that shows one.
The audit log is append-only → when cleaning up test tokens, delete the token rows in their own statement. Never touch audit rows.
Label every result: MOCK VERIFIED, CONTRACT VERIFIED, REAL INCUS TESTED, or REAL-BROWSER TESTED against a named local service. Never round up.
A count written in prose goes stale (2026-10-21) → "fifteen built-in alert rules" stayed in three pages while the code had sixteen, and a first attempt to count them by matching the rule constructors missed one written as a struct literal. → Take a count from something the code already asserts (the test's RULES.len() literal), or generate the page (as for openapi.json and SETTINGS.md); don't copy numbers into setup text.
Example systemd units with a Documentation=file://… that is not on the runner fail systemd-analyze verify (2026-10-22) → the check copies the units and drops that line (scripts/ci/verify-install-examples.sh). Parsing is not the same as running them under a dedicated account on a production host.
nginx -t as the runner could not load the example TLS keys (2026-10-22) → the dummy keys are installed 0600 root:root, matching a host, and the GitHub runner is not root. On a host the NGINX master reads them as root. → sudo nginx -t (scripts/ci/verify-install-examples.sh).