Apelora

Current state

Updated: 2026-10-20 (after Phase 09 slice 1, Phase 07 slice 5, Phase 10 slice 3) · Repository: github.com/Apelora/apelora-fabric, branch main · Related: ADR 0001, implementation phases, runbooks. The Phase 01 audit this file started as is summarised under Earlier design below.

Evidence labels: REAL INCUS TESTED (on the disposable test cluster, docs/development/TEST-CLUSTER.md), MOCK VERIFIED (against the mock provider), CONTRACT VERIFIED (against recorded Incus responses).

What exists and works

The parts of an application are called modules (formerly appliances; ADR 0006). Designs, API requests, stored records and Incus labels written before the rename are still read. Illustrations: platform layers, from a design to running instances, storage layers.

AreaLocationState
Apelora Studio (designer)apps/studioCatalog, canvas with typed terminals, cables and link tags, properties (resources, availability and replicas, internet access, configuration, ports), components, zones, undo/redo, import/export, validation; deploy dialog (plan preview, network rules, published ports, jobs) and a live overlay of every replica and service address; a deployed module's properties list each replica's server and addresses, read only while they are open.
Fabric Consoleapps/fabric-consoleNavigation grouped (2026-10-21). Overview with an alerts row. Health: each component's state with reason and next step; every server's CPU, memory and storage with warnings (runbook). Servers: Members (database role, leader, quorum, failure domain, group, Incus release, capacity; evacuate, restore and remove with a dry run), Machines and join tokens, Cluster settings (read-only), Certificates, Upgrades (read-only version skew against the pin), Preflight. Workloads: applications (instances, port checks, service addresses, volumes); instances (deploy, start, stop, delete, move with a dry run, terminal) with an instance page (restart, resources, addresses, disks, snapshots, recent jobs, logs); operations with a job page (retry, cancel, attempts, Incus operations, linked jobs). Networks: per application, the rules compiled from the design next to what Incus holds, with drift; service and published addresses; uplink and address ranges. Storage: pool use per server, application volumes with schedules and backups, restore from a backup, volumes no application uses. Alerts: built-in rules, acknowledge and silence, webhook channels. Identity & Access: organisations, projects and environments with quotas and their restricted Incus projects, users, groups, roles, assignments, service identities, policies, approvals, emergency access, sign-in settings, sessions, audit history. Audit: filters in the address, export. Administration: API tokens, versions, Incus projects, image mirrors, monitoring, effective settings (secrets never shown), licence notices (public /licenses), support bundle. From the user menu: Your tokens (personal apt_ tokens with rotation) and Your sign-ins. Old addresses (#/membership, #/preflight, #/infrastructure) lead to the new places. Evidence per feature in the rows below.
Control planeservices/control-planeAxum API (OpenAPI 3 document, generated TypeScript client), API tokens and roles, inventory, preflight, instances, applications (plan, deploy, revisions, status with services), cluster administration (machine enrolment, join tokens, evacuation) through a separate admin identity, audit log.
Deployment workerservices/deployment-workerJob queue with leases and fencing, woken by LISTEN/NOTIFY when a job is queued (2 s check kept as the fallback); the replica supervisor runs only in the holder of a leader lease in PostgreSQL, with fenced writes, so several workers can run (MOCK VERIFIED with a real PostgreSQL, 2026-10-21); instance lifecycle; network.sync (OVN networks, ACLs, address sets, load balancers with health-check repair, publishing, internet access); replica supervisor (recreate, restart, substitute on failed members, retire, sync retry, health with hysteresis and restart of failing replicas); failed members: fencing through the storage lease and restart of shared-storage modules elsewhere (instance.recover, ADR 0013).
Domain, persistence, Incus adaptercrates/Pure planning (replica slots, evacuation dry run, network rules), PostgreSQL 18 with migrations 0001–0009, Incus REST client with TLS pinning and a mock provider.
Catalog and modelpackages/143 original module classes in nine categories with subcategories (packages/module-catalog/src/taxonomy.ts), seven deployable: NGINX, Valkey, Uptime Kuma and Blackbox exporter (OCI containers), Debian and the service-name resolver (system containers), Linux base (VM, for live migration). The other classes stay design-only until their images have run on the lab. An OCI runtime can turn number, choice and on/off settings into its command and any setting but a secret reference into its environment (Valkey's memory limit, eviction policy and append-only file); the Studio marks every other setting of a deployable module as kept in the design, and a deploy-time field its image does not read as such (REAL INCUS TESTED 2026-10-25: Valkey reported the settings it was created with, Uptime Kuma answered on 3001, Blackbox exporter ran a probe; catalog_runtimes_run). Deploy-time setup fields (a name, a public address, a secret reference) are asked when an application is deployed. An application can set a minimum and maximum for vCPU and memory. Custom classes export and import as a catalog file. Zod schemas and generated JSON Schema, network intent compiler.
TestsRust, packages/*/src/*.test.ts, apps/**/*.test.ts, tests/e2e, tests/lab, services/deployment-worker/tests/real_incus.rs791 Rust tests, 219 TypeScript unit tests (packages and apps) and 392 Playwright tests against a stubbed API (Studio 150, Console 242; 34 opt-in screenshot and visual specs skipped) pass (2026-10-25). 9 real Incus tests pass on the four-member test cluster. Through the screens on that cluster (scripts/test-cluster/ui-e2e.sh, tests/lab), REAL INCUS TESTED 2026-10-25, 10 of 10: token and password sign-in, a local account given a role, three applications built by dragging from the catalog, deployed and operated, with placement, addresses, traffic through the service address, enforced connections, a volume's data across Stop and Start, a service name through the resolver and a virtual machine's address checked on the cluster itself.
CI.github/workflows/Frontend, Rust, generated-file drift, Playwright (pull requests run only the areas they change), dependencies and secrets (cargo deny, npm licences and audit, secret scan), dependency review, weekly upstream check; release workflow (draft releases with checksums).
Test cluster and drillsscripts/test-cluster/, scripts/enrol/Four Lima VMs standing in for physical servers, with Incus 7.0.1 (three database voters, one standby), plus a Lima VM apelora-san standing in for a SAN (iSCSI target); OVN, placement scriptlet; pools local (btrfs), backup (dir) and shared (clustered LVM on the SAN, shared-storage.sh, 2026-10-21, not used by Apelora yet); demo.sh runs the end-to-end demonstration; measure.sh records lab sizing (issue #16, 2026-10-22); enrol-member.sh enrols a server by hand.
Metricscrates/metrics, crates/telemetry, both services, Console → Administration → Monitoring, infra/observeGET /metrics in the text exposition format (OpenMetrics on request): API requests by route template, auth outcomes, authorisation denials, jobs and queue age, worker attempts and lease losses, supervisor actions and passes, scheduled snapshots and backups, Incus calls by endpoint template with reachability, database pool, inventory and application status counts, certificate expiry, process. The control plane's endpoint needs fabric.metrics.read (built-in role Metrics Reader); the worker serves metrics and a liveness probe on 127.0.0.1:9471; optional unauthenticated listener APELORA_METRICS_ADDR. JSON logs (APELORA_LOG_FORMAT=json); optional OTLP traces (APELORA_OTLP_ENDPOINT) for request and job spans. Example collector: Prometheus, Loki, Tempo, Pyroscope, Grafana (infra/observe, observe.sh). Incus call metrics CONTRACT VERIFIED; the rest MOCK VERIFIED against a real PostgreSQL. Apelora's /metrics scraped into the example collector while talking to the four-member lab is REAL INCUS TESTED 2026-10-23 (observe.sh: Prometheus apelora_incus_up=1 on both processes, four members Online, Loki ingested JSON logs, Tempo held request traces). Collecting Incus's own /1.0/metrics with a metrics-only certificate is REAL INCUS TESTED 2026-10-23 (scripts/test-cluster/metrics.sh: four members, 38 families each, /1.0/instances 403 with that certificate). Monitoring.
Logs and troubleshootingboth servicesRUST_LOG levels per crate, APELORA_LOG_FORMAT=json, an x-request-id on every API response that is on every log line of the request and in its audit detail, a job span on every worker line with the request_id that queued the job (stored on the job with every Incus operation it started, shown in the Console's Operations view), refusals (403, spent approvals) audited with their code, audit write failures logged as errors (MOCK VERIFIED); a change whose audit row cannot be written is rolled back (500 store_error, 2026-10-21). Troubleshooting runbook.
Job control and instance pagescontrol plane, worker, ConsoleRetry queues a new job linked to the failed one (retry_of; the failed job is kept unchanged; refused when not failed, retried already, superseded by a newer job for the same target, or an application operation); cancel ends a waiting job at once or asks the worker to stop at the kind's next safe point (between Incus changes; a create only before the instance exists; a move only while on its original member; an application operation between phases); both need the permission that queues that kind of job on its target and are audited (job.retry, job.cancel, refusals as denied). Every attempt is recorded (job_attempts, migration 0023: worker, timings, outcome, error, Incus operations; lost when a worker's lease ran out); jobs say who queued them. GET /api/v1/instances/{project}/{name} (detail), …/logs, …/logs/{file} and …/console-log (last 1–1024 KiB; application.logs.read or instance.manage). MOCK VERIFIED with a real PostgreSQL and CONTRACT VERIFIED (adapter). REAL INCUS TESTED 2026-10-23 (a_real_instance_has_a_detail_log_files_and_a_bounded_console_log on the four-member lab: a running Debian container listed lxc.log, a bounded tail, and a 612-byte console log; ../../etc/passwd was refused). Troubleshooting.
Alertsworker, control plane, Console #/alertsSixteen built-in rules (jobs, workers, leader leases, supervisor, replica health, application status, Incus and members, certificates, volume schedules, outbox, restore drill) evaluated every 30 s by the worker holding the alert-evaluator lease, with hysteresis and one open alert per rule and subject (migration 0025); thresholds, durations and severities editable (alert.configure); acknowledge and silence (alert.respond), audited in the change's transaction; webhooks hold a secret reference (APELORA_ALERT_SECRET_* on the workers) and receive HMAC-signed bodies through the outbox, with a test send and delivery state; metrics apelora_alerts_firing, apelora_alert_evaluation_duration_seconds. MOCK VERIFIED with a real PostgreSQL and a local HTTPS receiver; not REAL INCUS TESTED; no real receiver used. Design, runbook.
Public surfacecontrol plane, infra/serve/Security headers on every API response; public /health coarse and cached (Incus asked at most every 15 s; versions and Incus errors only on the signed-in /system); per-address and per-caller rate limits with 429 + Retry-After (per process); TLS of its own or a declared proxy before a non-loopback address; an example proxy and static serving configuration whose content security policy is proven by loading both production builds in Chromium (no violation, no request outside the origin). MOCK VERIFIED (API tests with a real TLS handshake) and REAL-BROWSER TESTED (policy); the example proxy was started on a lab member from draft v0.1.0-alpha.1 (install-units.sh, 2026-10-23: Studio and Console 200).

Identity and application management (2026-10-20)

AreaState
Identity, tenancy, authorizationOrganisations, projects, environments, users, groups, eleven predefined (Metrics Reader added 2026-10-21) and custom roles, scoped assignments with an escalation guard, service identities, approvals, emergency access, append-only audit, OIDC token validation, identity providers configured and edited in the Fabric Console with lockout guards, a designed sign-in page with PKCE (tested in a real browser against a local development provider), a default-deny route gate with exact-resource checks (MOCK VERIFIED; 46 auth, 10 identity, 29 persistence and 22 API tests). Credential containment (2026-10-21): an API token never carries more than its creator holds, a fabric-admin token needs another person's approval, every token expires within a year (API and command line), legacy tokens pass the route gate, revoking a role needs the power to grant it, and the last enabled super admin cannot be disabled, revoked or removed from their group, race-safely (MOCK VERIFIED against PostgreSQL; 8 API, 2 persistence and 1 auth tests). Audit integrity (S4, 2026-10-21): every change and its audit row in one transaction with listed exceptions, TRUNCATE refused, a test over every state-changing route, provider sign-ins audited hourly, audit filters and CSV/JSON-lines export (MOCK VERIFIED against PostgreSQL). Personal tokens (apt_, migration 0026, 2026-10-21): bound to a person and evaluated live, 1–365 days, rotation with an overlap, revoked by the person or with token.manage, revoked when the person is disabled, never for terminals, new credentials, approvals or emergency access; a person's sign-ins listed and ended (MOCK VERIFIED against PostgreSQL; 5 API, 3 persistence tests, audit-coverage steps). Tenancy isolation and quotas (ADR 0012, migration 0027, 2026-10-21): each environment's restricted Incus project is created (or adopted when restricted) by the control plane with the admin identity, given its own default profile and added to the deploy identity's project list; quotas on organisations, projects and environments (quota.manage, or tenants dividing their own) are checked when a deployment is planned (409 quota_exceeded before anything is saved) and mirrored as the tightest limit onto each environment's Incus project; Identity & Access → Tenancy & quotas; the Studio's deploy review shows quota use (MOCK VERIFIED: 9 API, 3 persistence, 12 domain, 4 contract tests, audit-coverage steps; REAL INCUS TESTED on the four-member lab: creation with the baseline, profile, grant, read-back by the deploy identity, limits mirrored and enforced by Incus, deletion; design). Live identity provider and MAAS 3.8 mapping are NOT VERIFIED. Details.
Fabric Console → Identity & AccessEight pages plus assignment preview with effective-permission origins, emergency access dialog and copy-once service tokens; Playwright with a stubbed API and axe (no serious violations).
Application managementDrafts with conflict detection, immutable published versions, server-derived status with per-caller permitted actions, Start/Restart/Shut down/Stop/Migrate as operations carried out by the worker (MOCK VERIFIED; ADR 0015: Shut down is graceful, Stop powers off at once, Migrate moves every instance to one chosen server after a dry run and a confirmation, through the same machinery as moving one instance; operations REAL INCUS TESTED at the worker on the lab, 2026-10-21; the dry run and the Studio dialog only against the mock). Deploying is done from the Applications list, not the canvas; the canvas saves by itself; the list imports and exports designs (2026-10-21). Studio: Applications page (cards and list, search, filters, sorting, pagination), per-application workspace (Overview, Designer, Deployments, Monitoring, Versions, Settings), icon picker using the editor's logos, publish and deploy-version flows. Run against the real control plane and worker with the mock provider: sign-in, draft save, publish, deploy through the worker, and a refused Start that showed the server's real error ("none of the application's instances exist in Incus"). The mock inventory is static, so Start/Stop on running instances is not yet REAL INCUS TESTED.
Interactive access to modulesTerminal sessions (and PowerShell for Windows modules) brokered by the control plane: permissions instance.console/instance.desktop, the Instance Access role, production approval, single-use tickets, limits, re-checked access, audit without content; Studio right-click and Monitoring entries with xterm; the same sessions from the Fabric Console (Terminal… on Instances and Applications, a Sessions page to see and end them, history from the audit trail; shared browser code in packages/terminal). MOCK VERIFIED and CONTRACT VERIFIED; not run against a real Incus, a Windows guest or a live identity provider. SSH daemons are deliberately not used. Remote desktop (built 2026-10-26, off unless APELORA_RDP_GATEWAY=enabled): the same broker relays the person's own RDP client through apelora-desktop-tunnel to a fixed relay inside the Windows guest, so 3389 is never published and Apelora relays the encrypted connection without decrypting it (the person checks the Windows certificate); MOCK and CONTRACT VERIFIED, never run against a Windows guest (design). Details.
Mutation testingcargo mutants on the authorisation crate (230 caught, 0 missed), the access and lifecycle rules, logs and optional traces (crates/telemetry), and Stryker on the model rules plus Studio and Console loggers. How and results.

Earlier design

An earlier hypervisor-based design (a Rust node agent and a VM per module) was never built and was replaced by ADR 0001. Its documents and skill were removed on 2026-10-20; skills, agents, README, CLAUDE.md, SECURITY.md and the commercial licence drafts describe the Incus-first platform.

What carries over unchanged

  • The whole Studio app and both packages. The Incus runtime is an addition to a module class (Phase 06), not a change to existing documents.
  • Port/terminal model, catalog rules, port placement convention, visual design contract.
  • Licence: AGPL-3.0-only for our code. Incus (Apache-2.0) and MAAS (AGPL-3.0, if adopted per ADR 0002) are used over their APIs.

Gaps against the target architecture

TargetGapPhase
Cargo workspace, control-plane and worker servicesSkeleton created in Phase 01 (compiles, no behaviour)01 → 03
Upstream tracking (upstream.lock.yaml, update checks, fork Apelora/incus)Done in Phase 02 (docs/development/UPSTREAM-INCUS.md)02
Incus adapter (read-only + mock) and inventory APIDone; REAL INCUS TESTED on the disposable test cluster (Incus 7.0.1)03
PostgreSQL persistence and worker job loop (leases, fencing)Done, tested on real PostgreSQL 1803
Instance lifecycle (create/start/stop/delete), ownership marker, project allow-listDone; REAL INCUS TESTED for LXC, OCI and VM (one member)04
Read-only preflight (/api/v1/preflight)Done; REAL INCUS TESTED; clock and latency not measurable via the API04
Disposable test cluster (scripts/test-cluster, Lima VMs)Done; four members (Lima VMs standing in for physical servers) and a SAN VM, OVN, placement scriptlet, see docs/development/TEST-CLUSTER.md04
Image pinning (fingerprint / OCI digest) and architecture checksDone; REAL INCUS TESTED on one member (mixed-architecture and offline-member paths unit-tested only)04
Catalog runtime fields (runtime, instanceRequestFor); NGINX and Debian deployableDone; REAL INCUS TESTED04
Class metrics descriptions (ModuleClass.metrics, all 83 classes; mon only where metrics text is served on a port; a mon connection opens the class's own port for the collector)Done; unit-tested (MOCK VERIFIED); facts researched from project documentation, unverified ones marked; no runtime turns metrics on yet07
Cluster join tokens with approval (ADR 0003), second test memberDone; REAL INCUS TESTED (two members)04
Auth (API tokens) and rolesDone, with OIDC sign-in configured in the Console03
OpenAPI document + generated TS client (packages/api-client)Done; console uses it03
Fabric Console views (instances and deploy, operations, preflight, networks and storage, membership, administration, audit)Done (slices 1–2); verified in a real browser against the test cluster05
Application deployment backend (plan, deploy, revisions, status by module)Done; REAL INCUS TESTED (two members)06
Studio deploy panel and live status overlayDone; verified in a real browser against the test cluster06
Network intent compiler (rules, service names, segments)Done (TypeScript, tested); shown in the Studio, not enforced07
Enforcing connections (per-application OVN network, ACLs and address sets)Done; REAL INCUS TESTED across two members07
Cleanup of network objects with the modules (owner-labelled)Done; REAL INCUS TESTED07
Publishing (gateway connections → service address + ingress rule)Done; REAL INCUS TESTED (outside route required)07
Internet access per module (@external egress)Done; REAL INCUS TESTED07
Service discovery: APELORA_SERVICE_<PROVIDER> in every consumer (ADR 0004)Done; REAL INCUS TESTED07
Stateless replicas, anti-affinity (placement scriptlet), supervisor restoring lost replicasDone; REAL INCUS TESTED (two members)08
Service addresses (OVN load balancers with health checks) and restart of crashed replicasDone; REAL INCUS TESTED end to end (two members) for IPv4. IPv6 listen addresses (APELORA_SERVICE_ADDRESSES /112–/126, IPv6 backends, health checks) REAL INCUS TESTED 2026-10-22 (scripts/test-cluster/ipv6.sh: fd42:198:100::1, connected client reached it, unconnected module blocked). The lab uplink routes that range (ipv6.routes) and keeps ipv6.address=none. DNS AAAA records stay MOCK VERIFIED07/08
Evacuation with dry run (API, console), audited; supervisor stays out of the wayDone; REAL INCUS TESTED (two members) and verified in a real browser08
Recovery from a failed member (substitute, then retire), health-check repair, sync retryDone; REAL INCUS TESTED (member VM force-stopped)08
Health with hysteresis from the load balancer's port check; restart of failing stateless replicas (paced, audited); health in status and StudioDone; REAL INCUS TESTED (four members, demo.sh)08
Application-level readiness: a module can declare a command run inside each instance (exit 0 = ready), combined with the port check, with the same hysteresis and restart rules; set per module in the Studio (Properties → Application check) or by a class (design)Built; MOCK and CONTRACT VERIFIED; the command run itself REAL INCUS TESTED on a lab container (2026-10-21: exit 0 and 1 read, a hanging command stopped at its limit, a missing program not an exit status, an unprivileged user refused a root file); the supervisor's decisions on a real failing application, and VMs, are not; no catalog class declares one yet08
Service names (DNS)Built 2026-10-22. When a module has a service address, every consumer also gets APELORA_SERVICE_NAME_<PROVIDER> = <module>.<application>.internal, and those records are written on the application network as user.apelora.dns with dns.domain set to <application>.internal (merged into the network config). An optional catalog module (resolver, ADR 0004 step 4) runs apelora-dns inside the application, is advertised as dns.nameservers, answers those names and forwards every other lookup. It is not added unless the design includes it. MOCK VERIFIED (domain tests and the mock provider). REAL INCUS TESTED 2026-10-23 (scripts/test-cluster/dns.sh: consumer env, dns.domain, user.apelora.dns, client reached the service address). REAL INCUS TESTED 2026-10-23 (dns.sh): after the client restarted, the service name resolved to the service address and the instance name still resolved. Without the resolver module, OVN DNS still answers instance names only08
RebalancingBuilt 2026-10-22. The members report lists one move for a stateless module whose replicas share an Online member (dry run). The supervisor queues instance.move (application.replica.rebalance) only when APELORA_REBALANCE=1, at most one per pass, skipped while a move is in flight or failed within five minutes. VMs are planned live; containers stop briefly. REAL INCUS TESTED for containers (four members, 2026-10-22, three runs): two replicas on one member, one pass moved -r2 to another member, Running 3.5–3.8 s later, and the next pass queued nothing. The VM path of the spreader is MOCK VERIFIED. Incus cluster.rebalance.interval stays a warned setting and is not this spreader08
Resources per module as Incus limits (CPU, memory, fractional CPU quota)Done; REAL INCUS TESTED (catalog runtimes)06
Volumes per module (backend): data-local class, pool from APELORA_VOLUME_POOL, one volume per instance created on the member Incus placed it on, attached before start, kept when the instance is deleted, never adoptedDone; REAL INCUS TESTED (volumes_follow_the_instance_and_outlive_it)06
Volume records (volumes, migration 0012), listing with live state (attached, kept, missing), explicit delete with the name typed (application.delete, audited); Fabric Console Volumes panelDone; REAL INCUS TESTED (four members) and verified in a real browser06
Volumes in designs: class defaults (runtime.volumes), per-module volumes in the Studio Properties window (checked against the schema, refused for stateless classes), listed in the deploy review and sent with the planDone; MOCK VERIFIED in the browser (stubbed control plane); the request shape is the one REAL INCUS TESTED through the API06
Volume grow (online, grow-only: a shrink is refused before it reaches Incus), API and Fabric ConsoleDone; REAL INCUS TESTED (adapter, running instance)06
Volume snapshots: take (named by time, says whether crash-consistent), list, restore with the name typed and only while every user is stopped, delete; audited; Fabric Console Snapshots dialogDone; REAL INCUS TESTED (adapter and in a real browser against the cluster)06
Scheduled snapshots (kept to what the schedule says; manual ones never pruned) and backups copied into a backup pool on another member, scheduled or on request (worker)Done; REAL INCUS TESTED (backup copy across pools and members)06
Backup API and console, restore from a backup: backups per volume, back up now, delete a backup with its name typed (DELETE …/backups/{backup}, the volume unchanged), schedules (PUT …/volumes/{volume}/schedules), restore as a volume.restore job (dry run, volume name typed, safety backup first, refresh copy, refused while an instance using it runs; migration 0024), Storage and Networks pages in the ConsoleDone 2026-10-21; MOCK VERIFIED with a real PostgreSQL and stubbed browser tests; the restore job REAL INCUS TESTED on the lab (2026-10-21: backup job, data changed, refused while the instance ran, stop, restore with a safety backup, start, the backup's data read back from inside the instance; the dry run, the typed name and the Console dialog were not part of that run; deleting a backup is MOCK VERIFIED, not run on the lab) (read-only GETs against the lab confirmed the volume, ACL, address-set, load-balancer and project shapes)06
Backup of Apelora's own database (ADR 0011 decision 6): scripts/ops/db-backup.sh (custom-format dump to a directory outside the managed cluster, SHA-256, manifest, retention, optional age/openssl cms encryption) and scripts/ops/db-restore-drill.sh (restore into a scratch database, verify migrations, row counts and the audit append-only trigger, drop it, write restore-drill-last*.json); scripts/ops/wal-archive.sh and db-basebackup.sh (continuous archiving); scripts/ops/db-restore-new-host.sh (logical restore onto another PostgreSQL, which is kept); self-tests in CI; runbookTested against a real PostgreSQL 18.6 on 2026-10-23. Logical dump and scratch drill, including age and openssl cms. Continuous archiving on a throwaway server: the archiver recorded no failure, and recovery into a new data directory returned the row written after the base backup. Logical restore onto a second PostgreSQL (another port on the same workstation): 29 migrations and 196 audit rows, and the source was unchanged. The example backup and drill units are started under systemd by scripts/ops/test-db-units.sh in CI, not on a production host and not by waiting for 02:15. Not a second physical machine. On 2026-10-24 scripts/ops/test-db-plane.sh started the control plane and the worker on a restored throwaway database (mock provider): the pending job from the backup was claimed and failed once (instance apelora/from-backup does not exist), and a job inserted after the backup was absent. The Console view of the last verified restore is built (Administration → Apelora's database backups)10
Moving an instance to another member (instance.move job; dry run, typed name): VMs with live migration move running; containers and VMs with data-local volumes stop, their volumes are copied and the old copies deleted once it runs there; Studio Keep running when moved for VM modules; Fabric Console Move…Done; REAL INCUS TESTED (container with a volume; VM live, 0.26 s longest pause, no reboot, from the console in a real browser)08
HA slice B (ADR 0013): data-replicated volumes and their module's root disk on APELORA_SHARED_POOL; failed members tracked (member_failures, migration 0028), fenced only by a DEAD sanlock lease read on a healthy member through the SSH fence probe (APELORA_FENCE_PROBE) or a person's confirmation (POST /cluster/members/{member}/fencing, approval); one decision per failure, instance.recover jobs (relink, start, wait while the target's lock view lags); GET /api/v1/cluster/recovery, Servers → Members Failed servers, Health note, alert member-unfenced, metricsDone 2026-10-21. REAL INCUS TESTED (four members, Incus 7.0.1, lvmcluster on the lab SAN): ha-drill.sh runs restarted a container module on another member 144–155 s after its member's VM was force-stopped (three runs), only after the lease was DEAD, data intact; real_incus shared-pool create and cold move. API, supervisor and Console MOCK VERIFIED. Studio choice of the class and BMC fencing are built (see the rows below). Ceph fencing is not. A VM whose disks are all on the shared pool uses the same Restart plan. REAL INCUS TESTED 2026-10-23 (HA_DRILL_VM=1, Incus 7.0.2, nested virtual machines): three runs, migration.stateful=true, Running on another member 141–144 s after the force-stop, the marker read back 158–164 s after the stop. Not a physical server08
Cluster settings changed from the Console: a dry-run review with warnings (for example Incus healing competing with Apelora's recovery), cluster.sensitive_change and another person's approval of exactly those settings, applied through the admin identity, audited with old and new values; only the nine listed settings, each value checkedBuilt; MOCK and CONTRACT VERIFIED; the change itself through the administrator identity REAL INCUS TESTED on the lab (2026-10-21: one setting changed, read back and put back as it was; the deploy identity refused; a key outside the list never sent); the review, approval and Console dialog not part of that run05
Database backup check: Administration → Apelora's database backups reads the backup and restore-drill scripts' records from APELORA_DB_BACKUP_DIR (newest backup and its age, size, encryption; last verified restore; a failed drill) and says what needs attention; read-onlyBuilt; MOCK VERIFIED (records in a temporary directory), not checked against a scheduled run10
Studio chooses where a module's volumes live: Properties → Volumes → Where the data lives (its server's own disks, or the shared pool so the module restarts on another server after a failure); all or none (schema and API refuse mixing); only the shared class is written to the file and the planBuilt; MOCK VERIFIED (unit and browser tests against a stubbed control plane); the API path is REAL INCUS TESTED for containers (HA slice B)06/08
Rebalancing (slice C)Built 2026-10-22 as Apelora's own plan, not Incus's rebalance interval (see the rebalancing row). REAL INCUS TESTED for containers on the four-member lab, 2026-10-2208
VM modules restarted after a server dies (shared pool): VM drill, partition drills, data-replicated in the StudioThe VM drill is REAL INCUS TESTED 2026-10-23 (three runs, marker read back; design). Studio data-replicated is built. Partition drills have not been run08
Fencing through a BMC (ADR 0014): a failed server in the worker's mapping (APELORA_BMC_MAP) is switched off after a grace time and its power state read back; guards (half the cluster Offline, one at a time, cooldown, hardware UUID must match), audited as cluster.member.power_off; providers redfish, ipmi, maas and the lab-only limaBuilt 2026-10-21. REAL INCUS TESTED with the lima provider (HA_DRILL_BMC=1 ha-drill.sh: a server cut off but running, fenced at 61 s, instance running again at 220 s: the BMC does not shorten the restart on clustered LVM). Redfish CONTRACT VERIFIED against a Redfish-shaped test server, NOT tested on any real BMC. IPMI (built 2026-10-22: system GUID, chassis power state and hard power-off only; cipher suite 17, else 3, never 0; operator role) CONTRACT VERIFIED against known answers, a loopback test BMC and an independent open-source IPMI implementation acting as the BMC on loopback; NOT tested on any real BMC (tests/real_ipmi.rs is ready for one). The mapping is still never shown. Servers → Members lists recent cluster.member.power_off audit rows (member, time, outcome, a one-line summary; no controller address) when any exist. MOCK VERIFIED08
Resizing existing instances: the plan compares an instance's own limits.cpu, limits.cpu.allowance and limits.memory with the design and shows a resize; the instance.resize job changes them in place without stopping the instance (level-triggered, repeat-safe, only Apelora's instances); the increase (not the whole instance) is checked against quotas; Studio review shows the changeBuilt; MOCK and CONTRACT VERIFIED (planner, adapter request bodies, worker job, API with quota, browser against a stubbed API). REAL INCUS TESTED on a lab container (2026-10-21): CPU and memory changed while it ran, it stayed Running, the guest then saw the new CPU count, repeating changed nothing, a missing instance failed without retry (the test found that Incus answers this change at once, not as an operation: fixed). Not tested: VMs, and a limit above the project's quota being refused by Incus. Disk (volume) resize is not part of this06
Machine enrolment (manual provider): register, approve, enrolment script, held until admitted after checks; consoleDone; REAL INCUS TESTED (fourth member) and verified in a real browser09
Member removal: check (evacuated, nothing left on it, recorded volumes, cluster keeps running), typed confirmation, admin identity, audited; Fabric Console Remove…Done; REAL INCUS TESTED (member 4 removed and enrolled again)09
Servers area: GET /api/v1/cluster/members (database role, leader, quorum with voters needed and spare, Incus release per member, version skew against the pin and upgrade blockers), /cluster/groups, /cluster/settings (allow-listed cluster keys with Incus's defaults, read with the admin identity); Console Servers tabs; rolling-upgrade runbookRead-only API REAL INCUS TESTED (read-only GETs, four members, Incus 7.0.1, 2026-10-21) and seen in a real browser against the test cluster; Console MOCK VERIFIED (Playwright, stubbed API, axe). Cluster settings can be changed with a review, another person's approval and an audit row (the Incus change REAL INCUS TESTED 2026-10-21; the Console dialog MOCK VERIFIED). The upgrade assistant (POST /api/v1/cluster/upgrade, Servers → Upgrades) walks members spare and stand-by first and the database leader last: evacuate, record that the package was upgraded on the server, restore. It does not install Incus. MOCK VERIFIED. The package procedure in the runbook remains NOT YET VERIFIED09
MAAS adapter (design): crates/maas-adapter (OAuth PLAINTEXT, https only, machines, power state, hard power-off), the Console's Machines MAAS manages list with Register… (records the MAAS system id), registration validated against MAAS, and MAAS as a fence provider (the BMC credentials stay in MAAS; the hardware UUID must match)Slice 1 built 2026-10-21. CONTRACT VERIFIED against a MAAS-shaped test server; REAL MAAS TESTED on a throwaway MAAS 3.7.4 over TLS in a Lima VM (maas.sh, 2026-10-21): machines listed and read with the OAuth signature, a wrong key refused (401), an unknown machine (404), an untrusted certificate refused, and for a manual-power machine MAAS answers the power query with 400, so the fence provider reads it as unreadable and fences nothing; the Console list and registration were not run against that MAAS. Commission (POST …/commission) and deploy (POST …/deploy, series name and cloud-init, the cloud-init not stored or audited) are built 2026-10-22. CONTRACT VERIFIED against a MAAS-shaped server; the Console dialogs are MOCK VERIFIED. Not REAL MAAS TESTED for these calls, and nothing has run against real hardware. Release (POST …/release) returns the machine to the pool; disks are erased only when asked, and that also needs hardware.delete. A secure wipe is not offered. CONTRACT VERIFIED. Burn-in (POST …/test, built 2026-10-22) runs MAAS's hardware tests on a machine that is Ready, Broken or Failed testing, never on an allocated or deployed one; admission of a MAAS machine is blocked by a failed or unfinished run and warns when it was never tested. CONTRACT VERIFIED; the Console's Burn-in… dialog and test status are MOCK VERIFIED in Playwright and were seen in a real browser against the lab MAAS (status Unknown, no Burn-in offered for its Failed commissioning records). A test run has not been REAL MAAS TESTED09
CI: frontend, Rust, generated-file drift, Playwright, dependency licences/advisories (cargo deny, npm), upstream checkDone10
Runbooks (join, maintenance, failure, deploy identity, revocation, outage); test cluster grown to three members (four since Phase 09)Done (partly verified, labelled per step)10
Secret scanning in CI; release workflow (draft releases; v0.1.0-alpha.1 built); end-to-end demonstration (demo.sh); certificate rotation; path-aware, tiered CI (CI.md: @smoke browser tests on every change, the whole suite nightly, SHA-pinned actions, CodeQL on shipping code, Dependabot auto-merge for passing patch and minor updates)Done; demonstration and rotation REAL INCUS TESTED (three members, three runs)10
Staged upgrades with rollback, remaining runbooksThe file half is built (scripts/ops/upgrade.sh): dry-run order stops the worker before the control plane, APPLY=1 installs only a bundle verify.sh accepts, keeps the previous bin/ and web/, and ROLLBACK=1 puts them back. Refused without a database dump. Tested by scripts/ops/test-upgrade.sh. Stopping the units, migrating, and restoring PostgreSQL are NOT YET VERIFIED on an installation (offline update). Runbooks for storage incidents and security incidents exist10
Transactional outbox (ADR 0011 §2.5, migration 0022): one message per audit row in the same statement; a relay in the worker under the outbox-relay lease, at least once with backoff, a log sink (APELORA_OUTBOX_SINK) and backlog metricsDone; MOCK VERIFIED with a real PostgreSQL (2026-10-21); no external sink yet10
Installations without internet access (ADR 0010): image mirrors and mirrors-only mode where every instance is created; preflight checks for mirrors and GET /api/v1/image-mirrors (catalog servers from the generated packages/module-catalog/catalog-images.json, mirrors probed at most once a minute, upstream servers never); Fabric Console Administration → Image mirrors; signed offline release bundle (scripts/release/bundle.sh, verify.sh) and offline-update runbook; blocked-egress lab profile (scripts/test-cluster/egress.sh)Mirrors and the preflight/API CONTRACT VERIFIED (unit and API tests, probes against local test servers); Console MOCK VERIFIED (Playwright, stubbed API); bundle built, signed and verified locally (tampering refused); egress rules accepted by nft --check on all four members, block not run; nothing REAL INCUS TESTED with egress blocked or with a mirror10
Several sites (ADR 0008, ADR 0009): sectors under one control plane, then sites and regionsSlice 1 built (2026-10-26): other Incus clusters are registered as sectors (GET/POST /api/v1/sectors, DELETE /api/v1/sectors/{name}, cluster.view / cluster.manage, audited, migration 0035) with their site, address and the name of their credentials (APELORA_SECTOR_<NAME>_CLIENT_CERT, _CLIENT_KEY, _SERVER_CERT on the control plane; nothing secret stored); each is probed through its own Incus connection (connected, unreachable, identity missing; members, release, a warning at 50 members); a connected sector in another site lets the members report name it for a second deployment (still not started); Console Servers → Sectors. MOCK VERIFIED (every registered sector is a mock of its own under the mock provider). Not built: deploying an application into a sector, sectors in the access model, networks and moves between sites. One lab cluster cannot show a second site09

Known limitations

  • Incus 7.0.2 is what the lab runs. The LTS package 1:7.0.2-debian13-202610211947 was installed on all four members on 2026-10-23 (stand-by, then the other voters, database leader last). The first upgrade blocked the other three until they ran the same release; afterwards every member was Online on 7.0.2, with three database voters and one stand-by. The cluster was empty, so nothing was evacuated. Earlier evidence was produced on 7.0.1.
  • OVN HA and TLS (issue #13): scripts/test-cluster/ovn-ha.sh converts the lab's one database into three clustered northbound/southbound databases with TLS (APPLY=1, 2026-10-22, four members, OVN 25.03). REAL INCUS TESTED: raft had three members; stopping the leader's ovn-ovsdb-server-nb/sb elected another (term 3); 15 OVN ACLs on apelora-default were unchanged; Incus still listed 30 leftover module ACLs over ssl: remotes; a Debian container on that network pinged the OVN router (10.13.53.1) 5/5 with 0% loss during the outage. ovn.sh still starts the single-node plaintext shortcut.
  • One architecture (aarch64) on the test cluster: no mixed-architecture evidence.
  • IPv6: application-network rules and IPv6 service addresses (fd42:198:100::/112 on ipv6.routes) REAL INCUS TESTED 2026-10-22 (ipv6.sh). The servers' own cluster addresses stay IPv4. Do not set ipv6.address on the lab's NAT incusbr0 (it wedged network create until incusd was SIGKILL-started).
  • Member failure: OVN health checks cannot see a dead member, so a service address keeps sending a share of requests to its replicas there until Apelora substitutes them (APELORA_RECOVER_AFTER_SECS, default 120 s). Single-instance and stateful modules are recovered automatically only when all their disks are on the shared pool, and only after their member is fenced (about 140 s after its last lease renewal; ADR 0013); on member-local pools they wait for their member.
  • Publishing needs the network outside to route service addresses to the application network's router; the OVN router does not answer ARP for them.
  • Service names (DNS) are delivered with the service address (APELORA_SERVICE_NAME_<PROVIDER> = <module>.<application>.internal) and published on the application network. REAL INCUS TESTED 2026-10-23 (dns.sh) for the publication. The optional resolver module answers the names inside the application and forwards the rest. REAL INCUS TESTED 2026-10-23 (dns.sh): the restarted client resolved the service name to the service address, and the instance name still resolved. Without the module, OVN's own DNS still answers instance names only. An instance created earlier keeps the environment, and the DNS servers, it was given at boot (ADR 0004).
  • Changed images and settings (design, issue #48): each new instance records what it was built from (user.apelora.spec, since 2026-10-25). Deploying a version that changes a stateless module's image, settings or storage replaces its replicas one at a time (plan action replace, one application operation of kind update): each new replica (<replica>-u<n>) must pass its service's health check for 20 s before the old one leaves the service address and is deleted; one that is not healthy within 300 s stops the update, is deleted, and the rest keep serving (APELORA_UPDATE_*_SECS). Deploying an earlier version is the same path. REAL INCUS TESTED 2026-10-28 (demo.sh: two NGINX replicas replaced in 151 s, 0 of 6,864 requests failed); the stopped-update path is MOCK VERIFIED. Singleton and stateful modules are rebuilt in place (plan action rebuild, slice 3): stopped, their root disk rebuilt from the new image, given the new settings and started again, keeping name, member, address and volumes with their data; deploying needs confirmDisruption because the module is down meanwhile. The rebuild job is REAL INCUS TESTED (a container's volume data kept across Debian 12 → 13); the whole path through the API is MOCK VERIFIED. A change to a module's volumes is still not applied (kept, marked outdated). Instances created before 2026-10-25 have no record and are not compared.
  • Resources apply when an instance is created, and a later deployment resizes CPU and memory in place (instance.resize, REAL INCUS TESTED on a running container). Disk size is a separate volume grow. VMs, and a limit above the project's quota being refused by Incus, are not tested.
  • Health is Incus status plus the load balancer's TCP port check, with hysteresis; an application that hangs behind an open port is detected only for a module that declares a readiness command, and instances without a service address are not checked. Health trackers live in the worker's memory: after a worker restart they start over.
  • Load balancers while a member is offline (Incus 7.0): Incus refuses every load balancer create, update and delete in the cluster while any member is unreachable (peer node … is down; it notifies all members with a policy that needs every one of them, then reverts). A substitute replica then gets no traffic until the member returns or is removed (runbooks/member-failure.md). The worker recognises the refusal and records why, naming the member; the application's status carries it as trafficBlocked, shown in the Console and the Studio (MOCK VERIFIED; the refusal's text is the one seen on the lab). The upstream report is drafted for a maintainer to file. Updates may also drop the OVN health check (detected through the state endpoint and re-applied).
  • Evacuation and restore first take the replicas Incus will move out of their service addresses (instance_drains, network.resync, waiting at most APELORA_DRAIN_WAIT_SECS), and a moved replica rejoins once it runs and has its address. Before this, a share of a service's requests failed for up to about ten seconds (2026-10-25: 3 of 8 right after the member reported Evacuated). REAL INCUS TESTED 2026-10-26 (demo.sh, four members, requests to the service address throughout an evacuation and a restore, five runs): 0 of 5,560, 0 of 5,384, 0 of about 4,800 and 1 of 11,800 requests failed; the single one was lost while OVN applied a change. In the fifth run the replica that moved back to the restored member was unreachable from the other members for at least 11 s while its health check passed (360 of 4,840 failed; issue #51). The supervisor no longer starts a replica Incus is still moving (issue #52): a stopped replica that carries volatile.evacuate.origin is left to Incus for 60 s first (REAL INCUS TESTED 2026-10-28: demo.sh fails a run in which the supervisor starts one, and none was started; the 60 s is a margin, not measured). For issue #51, demo.sh also requests the returning replica at its own address from the client for DEMO_SETTLE_SECS (default 30) after the restore: on 2026-10-28 with 300 s, 0 of 13,336 such requests failed and 0 of 16,328 to the service address; #51 was not reproduced in that run and stays open.
  • Offline detection: in one drill Incus briefly reported a member that was not stopped as Offline; keep the recovery delay well above that (default 120 s).
  • Bare-metal provisioning (Phase 09): servers installed by hand are enrolled, held and admitted through Apelora; the MAAS adapter lists machines, fences through MAAS, and can commission and install an OS (cloud-init is sent and not stored). Commission, install and release are CONTRACT VERIFIED only, not REAL MAAS TESTED. Disk erasure on release is off unless asked. A secure wipe is not offered. Burn-in runs MAAS's hardware tests and feeds admission (CONTRACT VERIFIED; no test run on real MAAS or hardware). An unreachable member is not removed through Apelora (Incus can only force it; see the runbook). Incus cannot hold a member before it joins, so the enrolment script holds it right after the join; a member that joins another way is flagged, not blocked.
  • Health and capacity (2026-10-21): the page shows the present only (no history or alerts yet); identity providers are reported from the control plane's last check, not probed per read; the per-server instance list covers only the projects Apelora's Incus identity may see; the Studio's capacity fit is advice (Incus reserves nothing for limits; root disks are not counted).
  • Releases: v0.1.0-alpha.11 (29 October 2026: the API breaking pass of issue #49 before the beta, with items lists, after/next paging, ?confirm= on deletions, renamed client functions, /storage-pools removed and the conventions stated in the API document; a wrong API token keeps its message in the Console) is the newest pre-release, after v0.1.0-alpha.10 (28 October: singleton and stateful modules rebuilt in place, issue #48 slice 3; every operation documents its error statuses and codes, and each code has one status; a loop pool's size answered on join; utoipa 6), v0.1.0-alpha.9 (the same day: rolling updates of stateless modules, issue #48 slice 2; migration 0036), v0.1.0-alpha.8 (26 October: remote desktop through the session broker and apelora-desktop-tunnel, both off by default; sectors slice 1; the upgrade assistant checks the release before recording an upgrade), v0.1.0-alpha.7 (the same day, issue #52), v0.1.0-alpha.6 and v0.1.0-alpha.5 (the same day), v0.1.0-alpha.4 and v0.1.0-alpha.3 (25 October) and v0.1.0-alpha.2 (24 October): control plane, worker, Studio and Console, both architectures, with checksums and build-provenance attestations. The v0.1.0-alpha.1 draft was deleted (its screens predate the sign-in page); its tag stays. The staged file upgrade and its file rollback are tested (scripts/ops/upgrade.sh); stopping the units and restoring the database are NOT YET VERIFIED. Example systemd units and the NGINX configuration (infra/systemd/, infra/serve/) are parsed in CI and were started on one lab member from the October draft (install-units.sh, 2026-10-23: both units active under apelora, health connected to Incus 7.0.1, Studio and Console 200). That start is not a run of v0.1.0-alpha.2. Issue #17 stays open until a published release on a production host.
  • Several sites: other Incus clusters can be registered as sectors and are read (Servers → Sectors); applications still deploy only to the home sector. A second deployment in another site is named once a sector of another site answers, and is not started. No network between sites, no move between sites.

Evidence behind the guides

The installation guides (docs/guides/) and the runbooks (docs/runbooks/) are written for the people who install and operate Apelora: steps, expected results, and at most one plain Before you rely on it list per page. What was checked behind them, how and when, is kept here, grouped by page (moved from those pages on 2026-10-25). Labels as at the top of this file; lab procedures are in TEST-CLUSTER.md. Runbook evidence levels: REAL INCUS TESTED means carried out on the disposable test cluster (Lima VMs standing in for physical servers), with the date and cluster size; NOT YET VERIFIED means written from the Incus documentation and our design.

Guides

Servers and network (guides/servers-and-network.md)

  • The layouts are what the four-server lab runs: virtual machines with one network card each, a private network between them, and a NAT bridge that Incus manages as the application uplink. Physical servers, separate management and storage networks, physical uplinks, and a bridge on a single physical card that carries the server's own address have not been tried. The two-card layout is recommended without having been proven. Statements that rest on Incus's documentation or on our recommendation say so.
  • IPv4 is what has been tried for the servers' own addresses; IPv6 and DNS names for servers are Incus features Apelora has not been tried with. IPv6 has been tried inside application networks and for service addresses (ipv6.routes).
  • The lab runs Debian 13 and is ARM64 only: nothing is proven for a cluster that mixes architectures.
  • Sizing (Lima VMs, idle Debian system containers, 2026-10-22): about 40 MiB RAM and 15 MiB disk per container as the kernel counts it, plus the OS and Incus. Not hardware, not busy modules, not VMs (lab sizing).
  • Shared storage: tried with an iSCSI target in the lab. BMC fencing with Redfish: tried against a test server only.
  • OVN databases: the default lab (ovn.sh) uses one database without TLS; ovn-ha.sh APPLY=1 is the three-member TLS cluster, REAL INCUS TESTED 2026-10-22 including leader failover.
  • Service addresses: the lab takes its IPv4 range from the range reserved for network testing (198.18.100.0/28) and an IPv6 ULA /112 (ovn.sh; APELORA_TEST_SERVICES_V6= skips IPv6). IPv4 and IPv6 listen addresses REAL INCUS TESTED. Service names (APELORA_SERVICE_NAME_*, dns.domain, user.apelora.dns) REAL INCUS TESTED (dns.sh, 2026-10-23); the resolver module answering them REAL INCUS TESTED 2026-10-23 (dns.sh: the client resolved the service name and the instance name still resolved).
  • Application networks: connections enforced over IPv4 and IPv6, REAL INCUS TESTED (scripts/test-cluster/ipv6.sh, 2026-10-22: a connected module reached its provider over both, an unconnected one was refused over both).

Install Incus (guides/install-incus.md)

  • The steps are those the four-server lab was built with (scripts/test-cluster/), written with variables. REAL INCUS TESTED on that lab, virtual machines with one network card each. Physical servers and a physical uplink have not been tried; those parts follow Incus's own documentation.
  • The package-repository route is what the lab does; its version, with the key check in code, is scripts/test-cluster/incus-vm.yaml, which refuses a key whose fingerprint differs. The lab uses a file-backed pool, for convenience only. The preseed shown is the lab's, with variables. scripts/test-cluster/up.sh makes both client certificates (ECDSA P-384, client authentication only) with the exact openssl options.
  • OVN: scripts/test-cluster/ovn.sh starts one database without TLS (a shortcut so application-network tests can run) on a managed NAT bridge uplink; it is the worked example for a single database. scripts/test-cluster/ovn-ha.sh (APPLY=1) converts that layout to three clustered databases with TLS. REAL INCUS TESTED on the four-member lab, 2026-10-22, OVN 25.03: three raft members, TLS client remotes (pssl:6641 / pssl:6642), Incus network.ovn.northbound_connection on ssl: with a client certificate. Stopping the leader's database processes elected another member; existing OVN ACLs were unchanged; Incus still listed the project's leftover module ACLs; a container on apelora-default kept pinging the OVN router (0% loss). Debian's ovn-central.service is a dummy oneshot (kept on the page as a tip).

Install Apelora Fabric (guides/installing-apelora.md)

  • The services have run together against the four-server lab (REAL INCUS TESTED).
  • The example systemd units and NGINX configuration are parsed in CI on ubuntu-24.04 (scripts/ci/verify-install-examples.sh: systemd-analyze verify for the units, nginx -t with dummy certificates for the proxy). The page's content security policy is checked by a browser test.
  • They were started on one lab member from the draft release v0.1.0-alpha.1 (scripts/test-cluster/install-units.sh, 2026-10-23): User=apelora, ProtectHome=true, both units active, /api/v1/health connected to the cluster, Studio and Console 200 over TLS through the example proxy, /api/v1/health through the proxy. That is not a published release and not a production host (issue #17); they have not been run as a production proxy or on a production host from a published release. install-units.sh now installs v0.1.0-alpha.2.
  • Releases: every version tag (v*) triggers the release workflow, which builds each service for x86-64 and ARM64 and each screen as its own archive, with SHA-256 checksums, and attaches them to a draft GitHub release that a maintainer reviews and publishes.
  • The step 4 and step 5 commands added to the page (account, /usr/local/bin, /srv/apelora/*, /etc/apelora/serve/) follow infra/systemd/*.service, infra/serve/apelora.conf and install-units.sh.

Signing in (guides/signing-in.md)

  • The screenshots were taken against a stubbed control plane (tests/e2e/screenshots.spec.ts).
  • Directories are configured and checked in the Console, and the directory bind is REAL DIRECTORY TESTED against the development directory. The sign-in form that offers each enabled directory is checked against a stubbed control plane.
  • The user-principal-name directory example is written from documentation; not tested against such a directory.
  • Identity-provider sign-in with PKCE was tested in a real browser against the local development provider only; a live production provider is NOT VERIFIED (see the identity row above).

Terminal sessions (guides/terminal-sessions.md)

  • MOCK VERIFIED and CONTRACT VERIFIED. No session has been opened into a real Incus instance yet (nor a Windows guest, nor with a live identity provider).
  • tests/e2e/web-serving.spec.ts opens a terminal in each production build under exactly the content security policy of infra/serve/apelora.conf.
  • The agents' rules for this area are the secure-remote-access skill.

Testing without hardware (guides/testing-without-hardware.md)

  • Level 1 (mock provider) is MOCK VERIFIED: it proves the logic, the permissions and the screens, and nothing about how Incus, storage or the network behave. Level 2 (Lima lab) is REAL INCUS TESTED.
  • BMC: the lab drives fencing with a provider that force-stops the VM (HA_DRILL_BMC=1 bash scripts/test-cluster/ha-drill.sh plan), REAL INCUS TESTED: a server cut off but still running was fenced at 61 s. The Redfish and IPMI providers are CONTRACT VERIFIED (Redfish against a Redfish-shaped test server; IPMI against a test BMC on loopback UDP and an independent IPMI implementation acting as one) until a real BMC runs them.
  • MAAS: listing and reading machines was run against MAAS 3.7.4 in the lab; those records use manual power and do not boot. Commission, install, release, and a test that boots a machine were checked against a server that answers in MAAS's shape, not against hardware.
  • Shared storage: the failover drills used the lab SAN's iSCSI LUN.
  • Until real servers arrive, statements about the BMC fence provider, the uplink and VLANs, storage speed and hardware faults stay CONTRACT VERIFIED or NOT YET VERIFIED.

Runbooks

Revoke access (runbooks/access-revocation.md): revoking an API token REAL INCUS TESTED (control plane against the test cluster), 2026-10-20; revoking an unused join token REAL INCUS TESTED, 2026-10-20; rotating Apelora's Incus identities REAL INCUS TESTED, 2026-10-20; a server that left the cluster NOT YET VERIFIED.

Alerts (runbooks/alerts.md): NOT YET VERIFIED on the test cluster. MOCK VERIFIED: evaluator, API, webhook delivery to a local HTTPS receiver, Console.

The control plane or the worker is down (runbooks/api-outage.md): the job queue's lease and fencing behaviour is covered by tests (MOCK VERIFIED). REAL INCUS TESTED, 2026-10-20, four members, scripts/test-cluster/drill-worker-restart.sh: a worker killed (kill -9) while Incus was creating an instance; a second worker took the jobs over when the 30 s lease expired, waited on the operation the first had started, and finished with exactly the three instances asked for; the same deployment request sent twice queued its jobs once. A longer outage under load NOT YET VERIFIED.

Rotate Apelora's Incus certificates (runbooks/certificates.md): REAL INCUS TESTED, 2026-10-20: both identities rotated on the three-member test cluster with rotate-identity.sh; the old certificates were gone from the trust store, the deploy identity kept its project restriction, and the real Incus tests passed with the new ones. The test cluster issues the certificates for 90 days.

The deploy identity cannot see its project (runbooks/deploy-identity.md): REAL INCUS TESTED, 2026-10-20. The cause was seen on the test cluster (the project deleted and recreated; access disappeared when a member went offline).

Back up and restore Apelora's own database (runbooks/apelora-database-backup.md), 2026-10-23, PostgreSQL 18.6. Self-tests: scripts/ops/test-db-backup.sh, test-db-archive.sh, test-db-restore-host.sh, test-db-units.sh.

  • Logical backup, scratch restore, retention, a damaged file refused, openssl cms, and age 1.3.2: the dev database on a workstation, client tools from postgres:18-alpine. The age drill decrypted, restored and verified. CI runs the same self-test and requires age to be installed.
  • Continuous archiving, on a throwaway PostgreSQL 18.6 removed afterwards: archive_command stored a completed segment, pg_stat_archiver.failed_count stayed 0, a base backup used --wal-method=none, and recovery into a new data directory returned a row written only after that backup. The source server was stopped before the new data directory started.
  • Logical restore onto another PostgreSQL (a second server on 127.0.0.1:15442, not the dev database's port): 29 migrations and 196 audit rows came back, and the dev database still had 196 audit rows. Restoring into the live server, or into a database that already has tables, was refused.
  • The example backup and drill units are started by test-db-units.sh where systemd runs; it is part of CI (ubuntu-24.04), parses every example unit, starts both services and enables both timers. It was not run on the workstation (no systemd), and it does not wait for 02:15 or Sunday 04:00.
  • The Console's backup view is MOCK VERIFIED (records written in a temporary directory); not checked against a real scheduled run.
  • Not done: a production host, a second physical machine (the new-host proof was a second PostgreSQL on the same workstation), leaving the timers to fire on their own calendars, and starting the control plane and worker on the restored database plus reconciling what the backup missed as one drill. Issue #17 (the install page's units from a published release) is a separate gap.

Reading the Health page (runbooks/health-page.md): the page and its API are MOCK VERIFIED (Rust API tests, Playwright with stubbed data). The server figures (CPU threads, load, memory, storage pool space, members by status, networks and pools) were read from the four-member test cluster with Apelora's restricted Incus identity and compared with direct Incus reads: REAL INCUS TESTED, read-only, 2026-10-21. Worker heartbeats and job counts are tested against PostgreSQL, not yet in a real incident.

Reading quorum, and a rolling Incus upgrade (runbooks/incus-rolling-upgrade.md): the readings (Servers → Members and Upgrades, GET /api/v1/cluster/members) are REAL INCUS TESTED read-only, 2026-10-21 (Incus 7.0.1, four members: three voters and a stand-by). The package upgrade was run on the empty four-member lab on 2026-10-23 (7.0.1 → 7.0.2): the first member blocked the other three until they were upgraded; all four then reported Online. Evacuation was not exercised (no instances), and Apelora's Upgrades tab was not the tool that installed the packages.

A server fails (runbooks/member-failure.md)

  • Stateless replicas: REAL INCUS TESTED, 2026-10-20: a member's VM force-stopped three times with two members and three times with three members (demo.sh, recovery delay 30 s); replicas were substituted and later retired automatically; the service was whole again 84–206 s after the failure. In the one run where Incus refused every load balancer update for the whole outage, the service still became whole after about 3.5 minutes.
  • Stateless replicas, 2026-10-25 (Incus 7.0.2, OVN 25.03, four members, recovery delay 30 s, apelora-test-2 force-stopped for 5.5 minutes): not recovered. The substitute was created at t+53 s and running by t+78 s, but every load balancer update was refused (three syncs of five attempts each, peer node … is down), so the service address kept its two original backends. About half the requests failed for the whole outage. The dead replica's health check stayed online until t+78 s, then had no status, and OVN kept sending it traffic. Incus's code refuses these updates in 7.0.1 as well. A second run the same day (demo.sh, the same member force-stopped) explains the 2026-10-20 drills: every update was refused again, yet the service was whole after 60 s, because OVN's own health check took the dead replica out. Whether it does varies between runs; when it does not, the service stays degraded until the member returns, and the status says why (trafficBlocked, REAL INCUS TESTED the same day: named apelora-test-2 76 s after the failure and cleared once it returned). The member returned and the cleanup through the API succeeded.
  • Shared storage: REAL INCUS TESTED, 2026-10-21 (four members, Incus 7.0.1, clustered LVM pool shared on the lab SAN, scripts/test-cluster/ha-drill.sh, a container module with a 256 MiB volume): restarted on another member 144–155 s after its member's VM was force-stopped (three runs), only once that member's storage lease was dead, with its data intact. Per run (t = force-stop): Offline t+22/23/23 s; failure opened t+29/30/30 s; lease FAIL t+85/92/74 s; DEAD, fenced and recover job queued t+139/148/144 s; relinked at the same times; lock wait 7 s with 3 attempts / 5 s / none; running t+146/155/144 s. Run 1 was before the job waited for the lock; a fourth attempt was stopped before the failure because of a stale SSH host key. After run 3 the VM was started again: Online after 15 s, its lease LIVE with the next generation after 29 s, learned by the worker after 76 s, failure closed. In run 1 Incus also reported the three members that were not stopped as Offline for about 10 s (the heartbeat round was held up by the dead member); Apelora opened failures for them, saw their leases LIVE, fenced nothing and closed them.
  • Virtual machine on shared storage: HA_DRILL_VM=1, 2026-10-23, Incus 7.0.2, a debian/13 VM with migration.stateful=true, root disk and a 256 MiB volume on shared, three runs each force-stopping apelora-test-2. Running on apelora-test-1 at t+144/141/144 s; the file written beforehand read back at t+163/158/164 s; the stopped member Online again within 19 s, and the worker learned its new lease (generations 7, 8, 9) within 83 s. A cold start, on nested virtual machines, not a physical server.
  • BMC fencing: HA_DRILL_BMC=1 ha-drill.sh, 2026-10-21, REAL INCUS TESTED with the lab's lima provider standing in for a BMC: the member's Incus port was cut with nftables and its VM kept running; timeline as on the page; the restart job used 5 attempts and 26 Incus operations. The Redfish provider is CONTRACT VERIFIED only; the IPMI provider is CONTRACT VERIFIED only, and real_ipmi would be its first run on a real BMC.
  • Removing a member that never returns: NOT YET VERIFIED.

Add a server to the cluster (runbooks/member-join.md): REAL INCUS TESTED, 2026-10-20 (Incus 7.0.1): the fourth test member was registered, approved, enrolled with scripts/enrol/enrol-member.sh, held, checked and admitted this way; the second and third joined with a bare join token (steps 2–3 without the machine record). The OVN check in step 5 is the lab's check.

Maintenance on a server (runbooks/member-maintenance.md): run on 2026-10-20 on a two-member cluster, through the control plane and the Fabric Console; containers on local storage moved in about ten seconds and kept their addresses. Moving one module first: run on 2026-10-21. The pictures are sample data from tests/e2e/screenshots-console-docs.spec.ts.

Remove a server (runbooks/member-removal.md): run on 2026-10-20, Incus 7.0.1, four members: member 4 was evacuated, checked and removed through the API; the cluster kept three database members; the machine was approved again, joined with the enrolment script and admitted; every step was audited. Removing a database voter from the Console, on a real cluster, has not been run. The picture is sample data from tests/e2e/screenshots-console-docs.spec.ts.

Network incidents (runbooks/network-incident.md): the OVN central database outage is REAL INCUS TESTED, 2026-10-20 (Incus 7.0.1, OVN 25.03, four members; the test cluster runs one central database on member 1, a documented shortcut, scripts/test-cluster/ovn.sh): ovn-central stopped on that member with an application of a client and two web replicas behind a service address; traffic and names kept working; a deployment that added a replica stalled; after the database started again, the queued jobs finished within 10 s. A member's OVN controller or tunnel down, and the uplink unreachable: NOT YET VERIFIED.

Update without internet access (runbooks/offline-update.md): the bundle part is tested locally (2026-10-21, macOS): scripts/release/bundle.sh built and signed a bundle with a throwaway key, and scripts/release/verify.sh accepted it offline and refused it after a file was changed, a file was added, the signature was removed, SHA256SUMS was edited, the signer was not allowed, or the archive held a link. scripts/ops/upgrade.sh prints the staged order (worker stopped before the control plane) and, with APPLY=1, copies bin/ and web/ into a prefix while keeping the previous copies for ROLLBACK=1; that copy and the file rollback are tested by scripts/ops/test-upgrade.sh. Stopping the units, starting them, and restoring PostgreSQL are NOT YET VERIFIED on an installation.

The cluster loses quorum (runbooks/quorum-loss.md): temporary loss REAL INCUS TESTED, 2026-10-20 (Incus 7.0.1, four members: three database voters and a standby): two voters force-stopped at once and started again; the test cluster's probe instance answered throughout; the API answered again and all members were Online within 10 s of the two members booting, with the same roles. Permanent loss (recovering from the surviving members) NOT YET VERIFIED: it rewrites the cluster's database membership and was not run on the test cluster.

A replica fails its health check (runbooks/replica-unhealthy.md): REAL INCUS TESTED, 2026-10-20 (Incus 7.0.1, four members): the end-to-end demonstration (scripts/test-cluster/demo.sh) closes the port of one NGINX replica while it keeps running and checks each step on the page. The application check (readiness command) is NOT YET VERIFIED against a real failing application.

A security incident (runbooks/security-incident.md): written from the controls Apelora has. Each control is tested (API and browser tests, MOCK VERIFIED; the certificate rotation REAL INCUS TESTED). The procedure as a whole has not been drilled, and no identity provider, directory or MAAS was involved: NOT YET VERIFIED. After a drill, record the result here and add an entry to LESSONS-LEARNED.md if something non-obvious went wrong.

A storage problem (runbooks/storage-incident.md): written from Apelora's features and the Incus documentation. The operations it points to (grow, snapshot, backup, restore, delete) are REAL INCUS TESTED as listed for volumes below; the incident procedure as a whole has not been drilled (NOT YET VERIFIED).

Tenancy and quotas (runbooks/tenancy-quotas.md): the Incus side (creating a restricted project, its profile, granting it to the deploy identity, limits that Incus enforces, deleting it) REAL INCUS TESTED on the four-member lab, 2026-10-21; the API and Console steps MOCK VERIFIED.

An application's volumes (runbooks/volumes.md)

  • REAL INCUS TESTED, 2026-10-20 (Incus 7.0.1, four members, btrfs pool local): an application with two volumes deployed through the control plane and worker; one module removed from the design (its volume kept, shown Kept, not in use); that volume deleted from the Fabric Console with its name typed; the remaining volume deleted after the application was removed. The Missing case and a member outage with a volume on it NOT YET VERIFIED.
  • Grow: REAL INCUS TESTED on a running container (filesystem volume, btrfs). Snapshot delete: REAL INCUS TESTED (2026-10-21, through the Console against the cluster). Deleting a backup copy: MOCK VERIFIED (API against PostgreSQL and the mock, and the dialog in a browser); not run on the lab.
  • Restore from a backup: MOCK VERIFIED (worker and API against a real PostgreSQL, browser with a stubbed control plane), 2026-10-21. The restore job is REAL INCUS TESTED on the lab (the backup's data was read back from inside the instance; the dry run, the typed name and the Console dialog were not part of that run).

Troubleshooting (runbooks/troubleshooting.md)

  • The log lines and request ids were produced by the control plane with the mock provider (2026-10-21); the worker and Incus lines are covered by unit and contract tests (MOCK VERIFIED / CONTRACT VERIFIED). They have not yet been read off a real cluster incident.
  • From a request id to the job and the Incus operation: CONTRACT VERIFIED with the mock provider and a real PostgreSQL (2026-10-21, one_request_id_links_the_request_its_audit_row_the_job_and_the_incus_operation); not yet followed on a real cluster incident.
  • Retrying and cancelling a job: MOCK VERIFIED with a real PostgreSQL (API, worker and persistence tests, 2026-10-21); not yet used on a real cluster incident.
  • Reading an instance's logs: CONTRACT VERIFIED (adapter against spec-shaped replies) and MOCK VERIFIED (API and Console). REAL INCUS TESTED 2026-10-23 on a running Debian container on the four-member lab (a_real_instance_has_a_detail_log_files_and_a_bounded_console_log: lxc.log listed and tailed, console log 612 bytes, a path outside the listed files refused).
  • The audit table's append-only triggers come from migration 0022; retention through monthly partitions is EVERY-SIZE.md §5 item 9.
  • The support bundle's scrubber is settings::scrub; account_tests::the_support_bundle_holds_the_parts_and_never_a_planted_secret plants secrets in every source.

Deviations from the incoming specification (deliberate)

  • npm workspaces kept instead of pnpm (owner decision): no tooling churn; Fabric Console is another workspace.
  • Crates created when first needed, not all at once: Phase 01 adds crates/domain and crates/incus-adapter plus the two services; the other planned crates are listed in the architecture but created by the phase that gives them code.
  • No other product names in repository text (rule 9); the incoming documents were rewritten in our own words and the files/ drop folder removed.
  • No gradients in UI themes (design rule 5), although the incoming spec mentions gradient themes.
  • Repository stays Apelora/apelora-fabric (the spec says Apelora/apelora).

This page is a copy of docs/architecture/current-state.md in the repository, which is the source of truth. Suggest a change.