One design at every size: communication, control and recovery from three servers to the global fabric

Status: reference and plan · Date: 2026-10-21 · Decision: ADR 0011 · Builds on: connections and scale, global fabric, disconnected installations

Connections and scale lists every connection. Global fabric covers distance: regions, sites, sectors. This page covers the third axis: how each concern behaves as the installation grows, and what to change in the code so that the three-server installation and the global fabric run the same software, the same mechanisms and the same rules, with only configuration and the number of copies differing.

"Built" below is checked against the code on the date above. Everything else is a plan.

1. Sizes (profiles)

A profile is a deployment shape, not a product edition. No size has a feature switched off for convenience. Authentication, authorisation, audit and TLS are the same at three servers as at fifty thousand.

2. How the parts talk to each other

Eight rules hold at every size. The first three already hold; the rest is the plan in §5.

1. The plane calls in; nothing calls out to the plane. Members, instances and SANs never open connections to Apelora (built). This keeps the attack surface to one direction and lets a plane restart without anything waiting on it.

2. Desired state in PostgreSQL, changes as jobs. A request records intent and queues a job. Workers claim jobs with leases and fencing generations (built). This holds at every size, from one worker to hundreds sharded by sector.

3. Static stability. When a higher level is down, a lower level keeps doing its job with what it last knew (built for the plane over Incus: instances keep running, ACLs keep enforcing, OVN keeps forwarding). The regional and global planes follow the same rule (ADR 0009).

4. Events for speed, a full re-check for correctness. Today the supervisor and status views poll Incus every 30 seconds; workers are woken when a job is queued and still check every 2 seconds. Polling is cheap even at the global estimate: one list per sector every 30 s ≈ 1 request/s per region. The cost is latency, not load. The plan:

   • subscribe to each sector's Incus event stream (instance and operation events) to react within a second;
   • keep the periodic re-check as the safety net that corrects anything an event missed, because events can be lost on a reconnect;
   • wake workers through PostgreSQL LISTEN/NOTIFY when a job is queued, and keep the 2-second check as the fallback (built, 2026-10-21).

5. One transaction, then an outbox. A change, its audit row and every message it must cause (a notification, a copy to the global plane, a webhook) are written in the same transaction. A relay then delivers the outbox rows at least once, and receivers ignore duplicates by id. One mechanism solves three problems at once:

   • audit integrity: no change without its audit row, gap S4;
   • replication between planes;
   • alert and webhook delivery.

   Built (2026-10-21, MOCK VERIFIED with a real PostgreSQL): the outbox table (migration 0022), one message per audit row written in the same statement, and a relay in the worker under the outbox-relay leader lease that delivers oldest first with backoff to a pluggable sink. The only sink today writes a log line; alerts, webhooks and replication plug in later.

6. Summaries go up, intent goes down.

   • Each level sends the level above counts, worst states and capacity, each with the time it was measured. It never sends raw lists.
   • A person who needs detail is sent to the level that owns it.
   • Intent (a design, a policy, a role) flows down and is cached below with a version number.

7. Every call has a budget. A timeout, retries with backoff and jitter, and a concurrency limit per sector (a bulkhead), so one slow or broken sector cannot use up the workers or connections the others need. After repeated failures a sector is marked unreachable: its jobs wait, they do not pile up as retries, and the Console says so.

8. One correlation id end to end. The request id (built in the API) is stored on the job it queues, set on the job's log span, and recorded next to the Incus operation id and on the audit row (built for jobs, 2026-10-21: jobs.request_id, jobs.incus_operations, requestId and jobId in the request's audit detail). Then one id answers "what happened to my click", from the browser down to Incus. Across planes it travels as the standard traceparent header.

3. Concern by concern

Each row says what holds at every size, what changes as the installation grows, and what is built.

Auditing

• Every size:
  • append-only, written in the same transaction as the change (§2.5);
  • denials are audited as well as successes;
  • no content of terminal sessions;
  • who, what, on what, outcome, request id.
• Grows into:
  • monthly partitions once the table reaches tens of millions of rows;
  • the regional audit stays in the region (residency);
  • the global plane keeps an archive of exported partitions;
  • a daily digest (a hash of the day's rows, chained to the previous day's) makes tampering detectable without special hardware;
  • export to write-once storage where the operator requires it.
• Built (2026-10-21, MOCK VERIFIED with a real PostgreSQL unless noted):
  • the append-only triggers for UPDATE, DELETE and TRUNCATE;
  • same-transaction writes for every route that changes Apelora's own state, with the exceptions listed in IDENTITY-AND-AUTHORIZATION.md §8;
  • a test that every state-changing route audits, in the same transaction where it can (audit_coverage_tests.rs);
  • audit rows for successful sign-ins (provider tokens once per person per hour);
  • filters by actor, target, action, outcome and time, and export as CSV or JSON lines;
  • an outbox message per audit row.
• Missing:
  • partitions;
  • the digest;
  • retention and an archive of exported partitions.

Security

• Every size:
  • mutual TLS to Incus with a certificate restricted to the deploy projects;
  • a second, unrestricted certificate only for cluster administration;
  • no secrets in the browser, logs or repository;
  • deny-by-default application networks;
  • instances cannot reach the Incus API;
  • TLS on the API, security headers and rate limits on public routes (wave 0, S3, in progress);
  • token roles limited to the creator's permissions (S1, in progress).
• Grows into:
  • one Incus client certificate per sector;
  • an intermediate certificate authority per region, so taking one region's credentials reaches nothing else;
  • rate limits shared per region instead of per process;
  • a security-posture page per region, rolled up globally.

Roles and permissions

• Every size:

  • the same roles, scopes, approvals and emergency access;
  • deny wins;
  • assignments are audited;
  • nobody can grant what they do not hold (S1/S2, in progress).

• Grows into: two scope trees, not one.

  • Tenancy: organisation → project → environment (built).
  • Infrastructure: fabric → region → site → sector (planned with sectors).

  An operator of one site's sectors must not see or touch another region's servers. A tenant's environment must not care which sector it lands on. A permission is checked against the scope tree its target belongs to.

• Speed:

  • each process caches a person's effective permissions, keyed by a generation number;
  • any change to roles, assignments or groups bumps the number;
  • NOTIFY tells every process to drop its cache;
  • in a region, the cached copy of global assignments is invalidated the same way when replication delivers a change.

  Revocation reaches every process within seconds at every size.

Performance and efficiency

Where time goes is measured, not guessed: request latency by route, job duration by kind, Incus call latency by endpoint and the database pool's acquire time (monitoring, built).

Efficiency for the servers themselves:

• system containers by default, VMs only where a class needs them (built);
• capacity views and placement that refuse what does not fit (health and capacity work in progress);
• later, consolidation advice: members that could be emptied and powered down.

Debugging

• Every size:
  • request ids (built);
  • structured logs with APELORA_LOG_FORMAT=json (built);
  • the correlation chain (§2.8);
  • a job detail page with each step, its duration and the Incus operation it waited on (wave 1);
  • log level changed at run time by an administrator, audited, and reverting by itself after an hour;
  • a support bundle: versions, effective settings with secrets removed, the last hour of errors, health, preflight, recent failed jobs (wave 4).
• Metrics for requests, jobs, Incus calls and the database pool point at where to look; the request id in the logs then says what happened to one request (monitoring).
• Grows into:
  • logs and metrics stay in the region;
  • "look up a request id" asks the regional plane that owns it, which the id itself names (a region prefix);
  • the global plane keeps only roll-ups.

Monitoring

• Every size:
  • each layer reports for itself, scraped by the collector the operator runs inside the site;
  • Apelora exposes its own metrics on GET /metrics (control plane, behind fabric.metrics.read) and on the worker's loopback listener (built);
  • Incus servers, PostgreSQL, OVN and the SAN are collected directly, never through Apelora, so Apelora is never a bottleneck or a single point of failure for metrics;
  • labels are templates and outcome classes only, with constant region, site and sector labels (built).
• Grows into: per-site collectors; regions keep aggregates by those labels; per-instance series stay in their site. The global plane sees only roll-ups (§2.6).
• Details: monitoring, with the signals, alert thresholds and runbooks for every layer.

Application metrics

• Application metrics come from each module's own endpoint, not from Apelora. Every catalog class says where its software serves them: port, path, format, whether they are on by default and how to turn them on, access, the signals worth watching, and how well that was checked (ModuleClass.metrics, MODULE-PORTS.md, the Studio inspector's Metrics section).
• Apelora authorises the collector's path through the drawn connection. A cable from a module's mon output to a collector's scrape input lets that collector reach that module on the class's metrics port only (deny by default); nothing else may read it, and metrics ports are never published.
• Resource use comes from Incus for every instance, whatever runs inside (CPU, memory, disk, network; for VMs through the guest agent, otherwise QEMU-level numbers). Classes that run no process (gateway, network) take their numbers from Incus network state and Incus's metrics endpoint.
• Same at every size. A collector module per application at three servers; per sector or region later, with the same connections.

Failure detection

Each layer detects its own failures, and each layer has one owner of the decision.

Two rules prevent false alarms:

• one missed signal is never enough: a member is acted on only when Incus has marked it offline, and nothing on shared storage moves on Incus's word alone (on the lab Incus briefly reported every member Offline while one was force-stopped);
• an instance is replaced only after consecutive failed checks.

Two rules prevent split brain:

• fencing comes before restarting anything that writes to shared storage;
• every job writes with its fencing generation.

High availability

• Small (three servers):
  • Incus keeps working with one member down (quorum two of three);
  • replicas are spread by anti-affinity (built);
  • stateful modules on the shared pool restart after fencing (built, ADR 0013);
  • the plane itself can be two processes on two members with a database standby;
  • if the whole plane is down, workloads keep running and only changes wait (§2.3).
• Large:
  • a site's sectors in different failure domains (power, network);
  • a regional database standby in another site;
  • the global plane is never in a request's critical path (ADR 0009).

Adaptability

The same build adapts to what the installation has, by detection, not configuration flags:

• preflight and the Console show which capabilities are present: OVN, shared pools, BGP, mirrors, a directory, GPUs;
• module classes declare what they need;
• placement and the Studio offer only what can run where.

Configuration changes take effect without a restart wherever possible, and are always audited. A capability that is absent says so, with the reason, and what to set up to get it.

Backups and recovery

A backup that has never been restored is not counted as a backup. The Console shows the age of the last verified restore, not the last copy.

4. What stays the same, and what changes

5. Work, in order

Each item is small enough to review on its own. (N) marks items needed now, at three servers.

1. (N) Audit in the same transaction, TRUNCATE blocked, a test that every state-changing route audits (S4). Then the outbox table and relay, so the next items reuse it. Built (2026-10-21, MOCK VERIFIED with a real PostgreSQL): handlers write the change and its row in one transaction (authz::begin/commit, audit_scoped(&mut tx, …)), with a short, listed set of exceptions (changes made in Incus or in memory, approvals spent before the change); migration 0022 blocks TRUNCATE; audit_coverage_tests.rs exercises every state-changing route and fails on a new one that neither audits nor is exempt with a reason; provider sign-ins are audited hourly; the outbox table and the worker's relay with a log sink and metrics. Not yet: real sinks (wave 3).
2. (N) Correlation: request_id on jobs, carried into the job's span, the Incus operation id recorded beside it, both on the audit row. Built (2026-10-21, MOCK VERIFIED with a real PostgreSQL): jobs.request_id from every API route that queues a job (instance requests, application deployments, application operations and the instance jobs they queue); the worker's job span has request_id; every Incus operation a lifecycle or move job started is kept in jobs.incus_operations; the request's audit row has requestId and jobId; GET /api/v1/jobs and the Console's Operations view show the request id. Not yet: operations an adapter call waits on internally (volume copies, backups), and traceparent across planes.
3. (N) Backup and restore of Apelora's own database: a script, a restore drill, a runbook (built: scripts/ops/db-backup.sh, scripts/ops/db-restore-drill.sh, scripts/ops/db-restore-new-host.sh, scripts/ops/wal-archive.sh, scripts/ops/db-basebackup.sh, runbook), and the age of the last verified restore in the Console (built: Administration → Apelora's database backups). Tested against a real PostgreSQL 18.6, including age, continuous archiving and a restore onto a second PostgreSQL. Not on a production host.
4. (N) Supervisor leadership through a lease in PostgreSQL, so two workers can run (the first item before HA of the plane itself). Built (2026-10-21, MOCK VERIFIED with a real PostgreSQL): leader_leases, 15 s lease renewed every 5 s, every supervisor write in a transaction fenced by the lease generation; not yet run with two worker processes on the test cluster.
5. LISTEN/NOTIFY:
   • job wake-up (built, 2026-10-21, MOCK VERIFIED with a real PostgreSQL: a trigger notifies apelora_jobs in the inserting transaction; the worker listens, reconnects and keeps the 2 s check);
   • sign-in settings reload;
   • permission cache invalidation.
6. Incus event subscription per sector, with the periodic re-check kept.
7. Per-sector bulkheads and the unreachable state, with sectors (ADR 0008).
8. The infrastructure scope tree for roles, with sectors.
9. Audit partitions, the daily digest and export (wave 1 adds filters; this adds integrity and retention). Filters and export built (2026-10-21, MOCK VERIFIED with a real PostgreSQL): GET /api/v1/audit filters by actor, target, action prefix, outcome and time; GET /api/v1/audit/export streams CSV or JSON lines and is audited as audit.exported. Partitions, the digest and retention are not built.
10. Run-time log level, support bundle (wave 4).

Items 1, 3 and 4 are P0 in the Console plan. The rest follow sectors (ADR 0008) and the global plane (ADR 0009).