Implementation phases

Order of work for the Incus-first platform (ADR 0001, architecture). One phase at a time; a phase is done only when its acceptance checks have evidence. Every result is labelled MOCK VERIFIED, CONTRACT VERIFIED or REAL INCUS TESTED; anything else is NOT TESTED.

Standing rules for every phase:

• Keep Studio working; run the full Studio checks (npm run typecheck && npm run lint && npm test && npm run build && npm run test:e2e) before and after.
• Rust: cargo fmt --check, cargo clippy --workspace -- -D warnings, cargo test --workspace.
• No infrastructure mutation except on an explicitly designated, disposable test cluster, after a dry run.
• No fabricated telemetry, deployments or HA claims; no TLS verification bypass; no secrets in the repository.
• Update docs and record significant decisions as ADRs.

01 — Audit and monorepo restructure

Audit (docs/architecture/current-state.md), ADR 0001, the earlier hypervisor-based design's material retired, project rules in AGENTS.md (Codex) imported by CLAUDE.md, skills shared under .agents/skills. Cargo workspace (toolchain pinned in rust-toolchain.toml) with crates/domain, crates/incus-adapter, services/control-plane, services/deployment-worker; apps/fabric-console placeholder; .env.example. Accept: Studio and Fabric Console build; Rust workspace compiles, formats, lints and tests; Studio import/export/render unchanged; no credentials in the repo.

02 — Upstream tracking

Verify the Incus 7.0 LTS install method, host minimums, licence and the REST endpoints/extensions we need, with source links and dates. Create upstream.lock.yaml (never invent versions, hashes or CVEs). Scripts under scripts/upstream/ (status, check-updates, verify; optional read-only source checkout into a git-ignored path). Scheduled/manual workflow that opens review-only update PRs with least-privilege permissions. The adapter negotiates api_extensions and returns UnsupportedCapability instead of guessing. Accept: a clean checkout validates the tracked release; no build needs Incus source; no unattended upgrades. Status: done — pin v7.0.2 (01f81e5e), fork Apelora/incus with patch branch apelora/7.0, scripts and weekly workflow; procedure in UPSTREAM-INCUS.md. Adapter-side api_extensions negotiation is partly in place (require_extension) and is completed against real responses in Phase 03.

03 — Rust API, domain and Incus adapter

Domain types (cluster, member, role, instance and kind, project, profile, network, storage pool, image, module class/instance, application graph, deployment, operation, health evidence, role, audit event) with desired and observed state kept apart. incus-adapter: mTLS client, version discovery (GET /, /1.0), extension checks, read-only inventory (members, projects, instances, images, profiles, networks, storage pools, operations), typed errors (auth, connectivity, quota, capability, operation failure, unknown state), bounded timeouts, no blind retries of non-idempotent calls. Mock provider with fixtures. control-plane: /api/v1, auth middleware, project-scoped RBAC, OpenAPI → generated TS client. persistence: PostgreSQL migrations for Apelora state only. deployment-worker: durable jobs, leases, read-only mode. Local development: PostgreSQL from a compose file bound to loopback on a non-standard port; SQLx offline query metadata committed so builds and CI need no database; a parallel test runner for the Rust suite. Accept: contract tests against recorded Incus responses; read real member and instance inventory from an authorised test cluster (REAL INCUS TESTED) before any write operation exists. Status (slice 1 done): read-only adapter over mTLS with pinned server certificate and the unspecified TLS-name case, mock provider, API inventory endpoints with source labels, Fabric Console members/instances views — CONTRACT VERIFIED (spec-shaped fixtures, real rustls handshakes), NOT yet REAL INCUS TESTED. Slice 2 done: PostgreSQL persistence (applications, immutable revisions, jobs, audit) with migrations, and the worker job loop with leases, fencing generations, idempotent enqueue and retry backoff — tested against real PostgreSQL 18. Slice 3 done: roles and permissions, hashed API tokens with expiry/revocation and audit, enforced on every route when a database is configured (development mode only on loopback), console sign-in. Slice 4 done: typed responses with an OpenAPI 3.1 document (GET /api/v1/openapi.json, apelora-control-plane openapi), committed as packages/api-client/openapi.json; @apelora/api-client (types and a typed get client) is generated from it and used by the Fabric Console. A Rust test and the CI generated-file check fail when either file is stale. Phase 03 is complete. Real inventory (members, projects, networks, storage pools, instances) was read from the disposable test cluster on 2026-10-20 — REAL INCUS TESTED (Incus 7.0.1).

04 — Cluster management and module deployment

Read-only host preflight (kernel ≥ 6.12, Incus version, clock drift, member latency ≤ 10 ms, DNS, TLS, pools, networks, project restrictions, capacity). Surface Incus bootstrap/join with short-lived tokens and operator approval. Instance create/start/stop/delete with async operation tracking for the three runtimes, image architecture and digest checks, restricted projects by default. Catalog runtime fields (see module-catalog). Local-development storage profile; cross-member networking only claimed after it is tested. Accept: on the test cluster: members shown, project verified, disposable LXC and OCI instances created/started/stopped/deleted with operation ids, nothing unrelated touched, no secrets leaked.

Status (slice 1 done, CONTRACT VERIFIED — not yet REAL INCUS TESTED): instance create/start/stop/delete for system containers, OCI containers and VMs. The API (POST /api/v1/instances, POST …/{project}/{name}/state, DELETE …/{project}/{name}, GET /api/v1/jobs/{id}) checks the deploy permission, validates names, allows only the projects in APELORA_DEPLOY_PROJECTS (default apelora), honours Idempotency-Key, audits the request and queues a job. The worker converges level-triggered: it re-reads the instance before every call, records the Incus operation id before waiting on it (fenced), waits on an earlier attempt's operation after a crash, stops before deleting, and rejects without retrying anything lacking Apelora's ownership marker (user.apelora.managed=true). Tested with spec-shaped fixtures for the Incus calls and real PostgreSQL for the queue. In mock mode the control plane and the worker each hold their own in-memory mock, so mock deployments appear only in the worker's log. Slice 2 done: read-only preflight (GET /api/v1/preflight): Incus ≥ 7.0, kernel ≥ 6.12, required API extensions (instance_create_start, operation_wait; instance_oci for OCI), member status, deploy projects exist and are restricted, a ready storage pool and managed network. Clock drift and member latency are reported as not checked: the Incus API cannot measure them. Slice 3 done: image pinning and architecture checks. A simplestreams image is pinned by its Incus fingerprint (sent as source.fingerprint, which Incus prefers to the alias), an OCI image by a digest in the alias (repository@sha256:…); architectures lists what the image supports. Before creating, the worker refuses a spec no cluster member can run (retries while suitable members are offline); afterwards it compares Incus's volatile.base_image and architecture with the spec and refuses, without changing anything, an existing instance that does not match.

REAL INCUS TESTED (2026-10-20, Incus 7.0.1, one-member cluster on Debian 13 arm64, test cluster): automated in services/deployment-worker/tests/real_incus.rs — system container, OCI container (by tag and by digest) and virtual machine created, started, stopped, started and deleted through the job queue; re-creates and repeated deletes converge without Incus calls; a new instance created from a fingerprint pin is built from exactly that image; an existing instance on the wrong architecture is refused; a spec no member can run is refused before anything is created; an instance Apelora did not create is refused (stop and delete) and left running. Manual run the same day: members, restricted project and passing preflight through the API; the lifecycle through the HTTP API and a separately running worker with a token (Idempotency-Key replay, default project refused, every request audited, operation ids in the worker log), the Fabric Console showing the real instances in a browser, and no secrets in logs or audit. Only unit-tested: retrying while suitable members are offline, and refusing aliases in mixed-architecture clusters (one member, one architecture). The real cluster found two defects the contract tests had not: an empty profile list removed the default profile (fixed: omitted when empty) and an X.509 v1 client certificate (fixed in the setup script; the adapter now names the problem).

Slice 4 done: catalog runtime fields. ModuleClass.runtime (system container, OCI container or VM) with one image or an image chosen by an enum setting that must cover every option; OCI images only by digest; instanceRequestFor maps a placed module to the API's InstanceSpec (type-checked against the generated client). NGINX (stable 1.30.6 and mainline 1.31.7, by digest), Debian (12, 13) and Linux base were the first deployable classes; the others stay design-only until their images have run on the test cluster (current-state.md lists today's). REAL INCUS TESTED: every catalog runtime request created, running and deleted on the test cluster (catalog_runtimes_run). The local-development storage profile is the test cluster's: a btrfs pool on a loop file and a project default profile with root disk and NIC (scripts/test-cluster/up.sh).

Slice 5: cluster membership (ADR 0003). A separate, optional, unrestricted admin identity loaded only by the control plane; a fabric administrator approves a new member by asking for its single-use join token (POST /api/v1/cluster/join-tokens), which is shown once and never stored, logged or audited; pending tokens are listed without secrets and can be revoked; tokens living longer than an hour get a warning. Bootstrap of the first member stays on its host (incus admin init). REAL INCUS TESTED (2026-10-20): a second member was approved through POST /api/v1/cluster/join-tokens and joined with that token (piped to scripts/test-cluster/add-member.sh, never on a command line); the API then showed two online members and no pending token, preflight reported "2 online", and join_tokens_need_the_admin_identity_and_can_be_revoked (issue, list without secret, revoke; refused for the deploy identity) and the whole real suite passed on the two-member cluster. The join first failed because the VMs' NAT network isolates VMs from each other; members now talk over a separate network.

The lab's four members run package 7.0.2 (2026-10-23). Functional evidence before that date was produced on 7.0.1.

05 — Fabric Console

apps/fabric-console using the Studio's visual language (white shell, daisyUI tokens, no gradients). Views: overview, cluster and members (roles, groups, failure domains), instances, projects/tenants, networks, storage, catalog/images/profiles, operations, application health, administration (users/roles, Incus TLS status, upstream versions, audit). Only fields the API actually provides; unknown values shown as unavailable; mock data labelled; destructive actions confirmed and permission-checked; controls disabled during conflicting operations. Accept: browser tests and API contract tests; every action goes through the Rust API.

Status (slice 1 done): navigation by permission (whoami now returns deployProjects); Overview (status, members), Instances (Apelora-managed instances get start/stop/delete, each followed through its job with retries and errors shown on the row; delete needs the name typed; deploy dialog builds the request from the catalog runtime), Preflight, Networks & storage (projects, managed networks, pools), Membership (issue a join token shown once, list, revoke with confirmation), Audit log. Inventory now says which instances are Apelora's (Instance.managed). 14 Playwright tests with a stubbed API (including a phone layout with touch-sized controls); verified in a real browser against the two-member test cluster: an NGINX instance deployed, stopped and deleted, and a join token issued and revoked, from the console. Design review: spacing measured at 1440 and 390 (header grouping, phone navigation and table columns fixed; compact desktop controls kept per the design contract, 44px on touch screens), copy checked per view. Slice 2 done: Operations (the job queue, newest first, with outcome and error; refreshes itself while jobs are in progress; GET /api/v1/jobs), Administration (API tokens: create with a role and a 1–365-day expiry, shown once, list with last use, revoke with confirmation; GET/POST /api/v1/tokens, DELETE /api/v1/tokens/{id}, audited as token.create/token.revoke; System: control plane version, running versus pinned Incus release and its recorded evidence, GET /api/v1/system). Verified in a real browser against the test cluster (a refused deployment shown with its reason; a token created and revoked). Slice 3 done: Incus certificate status. The adapter reads when its client certificate and the pinned server certificate expire (certificate_not_after); GET /api/v1/system lists the deploy identity, the admin identity and the server certificate with days left (expiring within 30 days, expired), and Administration → System shows them, pointing to the rotation runbook when one is due. Verified in a real browser against the test cluster (the dates match openssl x509 -enddate). Slice 4 done: Applications view (each deployed application's revision, how many instances run, how many fail their port check, service addresses; its instances with member and port check on demand; phone layout) and Operations labels for restarts and network rules. Verified in a real browser against the test cluster with one replica's port closed; spacing measured at 390, 1440 and 1728; copy checked. Phase 05 is complete. While cleaning up, a removal request without connections was found to leave the application's network objects behind; a removal of every module now cleans them up whenever the last deployment enforced connections (API test; REAL INCUS TESTED).

06 — Studio deployment integration

Module classes gain runtime, image digest, resources, volumes, probes, secret refs, availability policy. Deployment flow: save graph revision → validate (types, digests, quotas, storage, network, capacity) → plan with candidates and warnings → designer approves → worker applies through the adapter → instance start and readiness checks → overlay live state on the saved graph without rewriting it. Deploy panel: runtime, CPU, memory, replicas, exposure. Demo: web (2 replicas) → API (2 replicas) → PostgreSQL (1, recovery manual-until-safe), with warnings for single points of failure. Accept: import/export round-trip, type validation, plan preview, instance mapping; a real LXC + OCI graph deployment on the test cluster when available; mock paths labelled.

Status (slice 1 done, backend; design in application-deployment.md): instances deployed for an application carry user.apelora.application / user.apelora.module labels and are named <application>-<module>; a pure planner compares the design with Incus (create, keep, remove, conflict); POST /api/v1/applications/{name}/plan previews, POST …/deployments recomputes the plan, refuses the whole deployment on any conflict, saves the document as an immutable revision (idempotent with Idempotency-Key: a retry saves and audits nothing new) and queues instance jobs (removal only with removeMissing); GET …/status reports the application's instances by module; GET /api/v1/applications lists applications with their latest revision. REAL INCUS TESTED (manual run, 2026-10-20, two members): an OCI (NGINX by digest) + system container (Debian) application planned, deployed (placed by Incus on different members), repeated without effect, reported by module, then redeployed without Debian, which was removed while NGINX was kept; labels verified in Incus. Slice 2 done: the Studio's "Deploy…" dialog (connect and sign in, application name and project, deployable versus design-only modules with the reason, the server's plan with warnings and conflicts that block applying, apply with an idempotency key, job progress) and a live overlay: each node shows its instance's state from the status endpoint, the header badge says the data comes from Incus; the design is never changed by it. Four browser tests with a stubbed API; verified in a real browser against the two-member test cluster: the sample commerce stack deployed its two NGINX modules (placed on different members) and showed them running, the other four listed as design-only. Slice 3 done: resources. Each module's CPU and memory (Studio Properties, default from its class) become Incus limits on its instances (resourceLimits: limits.memory in MiB, whole CPUs as limits.cpu, a container's fractional CPU also as a hard quota limits.cpu.allowance, VMs whole vCPUs); the deploy review shows them per module. REAL INCUS TESTED: every catalog runtime request, now with limits, plus NGINX with half a CPU, created and running on the four-member cluster (catalog_runtimes_run). Changing a module's CPU or memory now resizes its existing instances in place (slice 4, below). Readiness beyond Running: Phase 08 slice 5 (health). Remaining: volumes per module (volume resize is separate).

Slice 4 (resizing existing instances, MOCK and CONTRACT VERIFIED): the planner compares each Apelora instance's own limits.cpu, limits.cpu.allowance and limits.memory with the design and plans a resize (limits it does not manage are never touched; 1GiB equals 1024MiB); instance.resize is a level-triggered worker job that sets the keys with one instance update, without stopping the instance, and is safe to repeat; the quota check counts only the increase of a resize (a decrease adds nothing); the Studio deploy review shows the change. Not yet REAL INCUS TESTED: that every instance kind applies the change live and how Incus words a refusal.

07 — Network fabric, service discovery, publishing

Network intent compiler (graph → intents → diff → apply), stable service names mapped to healthy replicas, deny-by-default ACLs per project, MVP VLAN profile then optional OVN profile (networks, ACLs, DNS zones, forwards, load balancers). Publishing: shared HTTPS ingress (default), private, dedicated address, internal name; domains and TLS certificates; DNS integration. Accept: compiled-intent snapshot tests; isolation tests; on the test cluster, a consumer reaches a service by name across members and keeps working when a replica is replaced.

Status (slice 1 done, compiler only — nothing is enforced on the cluster yet): compileNetwork(assembly, application) in @apelora/assembly-model turns cables and link tags into allow rules (consumer output → provider input, protocol and well-known port, provider's stable name <module>.<application>.internal), services with their endpoints, and NIC segments (tagged, cabled, or default); deterministic (sorted, order-independent), deny by default (allows), generic TCP terminals flagged. Snapshot, determinism, isolation and tag tests. The Studio's deploy review lists the rules and says they are not enforced yet. OVN on the test cluster (scripts/test-cluster/ovn.sh): REAL INCUS TESTED that instances on different members reach each other by name over an OVN network, and that ACLs enforce deny by default with per-port allow rules — but only with IP/address-set rules: ACL subject selectors failed across members (details in TEST-CLUSTER.md). Enforcement design therefore: per application one OVN network; per module an ACL and an address set holding its instances' addresses (kept current by the worker); for each intent, the consumer's egress and the provider's ingress allow the declared port between the two address sets. Slice 2 done: enforcement. With APELORA_NETWORK_UPLINK set, a deployment that carries the design's connections attaches every new instance's NIC to the application's OVN network with its own ACL (unmatched traffic rejected both ways), and a network.sync job ensures the network, one ACL and one address set per module, fills the sets with the instances' addresses (retrying while instances start) and writes matching egress/ingress rules on both sides; Apelora never adopts objects without its marker. Without an uplink the plan says connections are not enforced. The Studio sends the connections between deployed modules and says whether they are enforced. REAL INCUS TESTED (manual run, two members): client on member 1 reached web-01 on member 2 by name on port 80; port 22, an unconnected module on the same member, and the reverse direction were all blocked. Network objects carry user.apelora.application, so cleanup never touches another application's objects (a name prefix never decides ownership); network.sync deletes the ACLs, then the address sets, of modules that left the design, and the network with the last module — REAL INCUS TESTED (deploy, then remove everything: instances, ACLs, address sets and network gone). Slice 3 done: publishing. A gateway in the design stands for the outside world; the Studio sends what it connects to (deployed module, transport, well-known port) as published; the API validates it, warns when publishing needs enforced connections and a service range, and records it; network.sync gives each published module a service address (even a single instance) and an ingress rule open to any source on that port only. The network outside must route the service address to the application network's router. REAL INCUS TESTED (three members): with that route, an outside client reached the published port, an unpublished one stayed blocked. Slice 4 done: internet access. Each module has an Internet access setting in the Studio (none — the default —, web and DNS, any port), sent with the deployment; network.sync adds egress rules to @external only (never an any-destination rule, which would open the other modules); the plan warns when it cannot be enforced. REAL INCUS TESTED (three members): a client with web access fetched from a public mirror, port 22 outside stayed blocked, its connection inside still worked. Slice 5 done: service discovery (ADR 0004). Service addresses are allocated at deployment, before instances exist; every consumer is created with APELORA_SERVICE_<PROVIDER> per connection — the provider's service address, or its instance name (resolved by OVN's DNS) when it has none; recreated and substituted instances get the same values; a design cannot set these variables itself; the Studio's deploy review shows them. REAL INCUS TESTED (three members): inside a client, $APELORA_SERVICE_WEB_01 reached the replicated web service and $APELORA_SERVICE_DB_01 resolved to the database. DNS names remain possible later as an optional resolver module.

08 — Coordination, HA and mobility

PostgreSQL leases/generations; separate member/instance/service/ingress/storage health with hysteresis, grace periods, cooldowns; stateless replica policy with anti-affinity via verified Incus mechanisms; replace-then-retire relocation; policy-aware evacuation with dry run and per-item reasons (M1), replica-aware recovery (M2), advanced HA with tested fencing and shared storage (M3). Stateful single writers blocked on ambiguity. One recovery authority per workload; Incus healing stays off unless enabled deliberately. Status (slice 1 done: replicas, anti-affinity, restoration): classes declare runtime.availability (stateless, singleton — the default — or stateful; NGINX is stateless); a stateless module takes 1–9 replicas (Studio Properties → Availability), named <application>-<module> then -r2…, labelled user.apelora.replica; the API refuses replicas for anything not stateless and the plan names every single point of failure. Each deployment's desired state is recorded (application_deployments); the worker's replica supervisor (every APELORA_SUPERVISE_SECS, default 30) queues an ordinary create job for a lost stateless replica — never while one is in flight, with a five-minute back-off after a failure — audits application.replica.recreate, and re-syncs the application's address sets; singleton and stateful instances are reported, never recreated. Anti-affinity uses Incus's placement scriptlet (infra/incus/placement.star, installed by the Incus operator; scripts/test-cluster/placement.sh on the lab): fewest replicas of the same module first. REAL INCUS TESTED (two members, 2026-10-20): two replicas placed on different members although member 2 carried more load (without the scriptlet both landed on member 1); a deleted replica was recreated by the supervisor on the member without its sibling. Slice 2 done: service addresses and restarts. With APELORA_SERVICE_ADDRESSES (a routed range of the uplink), every stateless module others connect to gets an OVN load balancer over its replicas with health checks (address allocated in PostgreSQL, released with the module; listed by the status API and in the Studio overlay, which now shows every replica and a degraded state). The supervisor also restarts a stateless replica that stopped behind Apelora's back (application.replica.restart) but respects a stop made through Apelora. REAL INCUS TESTED end to end through the control plane and worker (two members, 2026-10-20): requests to the service address alternated between replicas on different members; after one replica was force-stopped, all requests went to the other within four seconds; the supervisor restarted it within a pass and it rejoined; after it was deleted, the supervisor recreated it at the same address and the load balancer rebound to the new instance; removing the application deleted instances, network, ACLs, address sets and load balancer. Slice 3 done: evacuation with a dry run. GET /api/v1/cluster/members/{member}/evacuation explains, per instance on the member, what Incus will do (from its effective cluster.evacuate mode) and what that means for its application (replicas elsewhere keep the service up; singletons and stateful modules are down while they move or until restore), names what blocks it (no other online member, no member of the instance's architecture, member not Online) and warns when replicas will share the last member. POST …/evacuate and …/restore need manage-infrastructure, the admin identity and the member's name typed again; evacuating re-runs the dry run and refuses while anything blocks it; both are audited. The Fabric Console's Membership view lists members with Evacuate…/Restore… (dry run table, type-to-confirm). The supervisor leaves instances on a member that is not Online to Incus. REAL INCUS TESTED (two members, 2026-10-20, through the control plane): the dry run for member 1 predicted both instances moving and the client being down; evacuation took about ten seconds, the instances kept their addresses on the OVN network, the service address answered from both replicas throughout, the supervisor did not interfere; restore moved them back; both actions audited. Verified in a real browser against the cluster. Slice 4 done: recovery from a failed member (replace, then retire). Replica slots are matched by labels, so a slot may be filled by a substitute <replica>-s<n>; an instance whose member is unreachable (listed with no configuration) still counts in its slot by name. A member that stays Offline longer than APELORA_RECOVER_AFTER_SECS (default 120; hysteresis kept by the supervisor) has its stateless replicas substituted on another member; when it returns, the slot's extra instance is retired, keeping the one that puts replicas back on different members; singletons and stateful modules are only reported. Address sets and load balancer backends follow the slot's reachable instances; a failed network sync is retried after a minute; a load balancer whose health check went missing is repaired. REAL INCUS TESTED (two members, member VM force-stopped three times): the substitute appeared 31 s after the recovery delay began, the load balancer swapped the dead backend for it and requests stopped failing; on return the supervisor retired the substitute and kept the original on the returned member; a lost health check (state unknown) was reinstated by the next sync. Findings in TEST-CLUSTER.md (instances with config: null, health checks blind to a dead member, a restricted identity losing its project after a project was recreated). Slice 5 done: health with hysteresis. The worker decides each running service replica's health from the OVN load balancer's per-backend port check (apelora_domain::health: unhealthy after failing 20 s, healthy after passing 20 s, unknown neither confirms nor clears; APELORA_HEALTH_FAIL_AFTER_SECS), records it (instance_health, migration 0009) and restarts a stateless replica unhealthy for 60 s (APELORA_HEALTH_RESTART_AFTER_SECS) with a one-shot instance.restart job — at most one replica per module per pass, least recently restarted first, the same one not again within five minutes — audited application.replica.restart-unhealthy; singleton and stateful modules are only reported. The status API reports health for running instances; the Studio counts unhealthy replicas as not serving (Degraded, Unhealthy). Runbook: docs/runbooks/replica-unhealthy.md. REAL INCUS TESTED (four members, demo.sh, 2026-10-20): with one NGINX replica's port closed while it kept running, it was reported unhealthy after 36 s, the service answered from the other replica only, the supervisor restarted it once after 66 s and not again while it stayed broken, and it was healthy 40 s after the port reopened. The check is a TCP connection: an application that hangs behind an open port is not detected. Worker restart and duplicate requests REAL INCUS TESTED (scripts/test-cluster/drill-worker-restart.sh: worker killed mid-create, the next one waited on its Incus operation after the lease expired, exactly the instances asked for; a repeated request queued nothing new). Remaining: partition tests, and a preflight check that the deploy identity can use its projects. Service names are delivered as APELORA_SERVICE_NAME_* and published on the application network; nothing inside the application answers them yet. Replica spread is Apelora's own plan (APELORA_REBALANCE=1), not Incus's rebalance interval. REAL INCUS TESTED for containers (four members, 2026-10-22): two replicas on one member were spread by one supervisor pass.

VM modules on the shared pool: REAL INCUS TESTED 2026-10-23 (HA_DRILL_VM=1, three runs; design). Studio data-replicated and the BMC fence provider are built. Still not run: the partition drill and the confirmation drill.

Accept (disposable lab): stop one stateless replica → ingress reroutes and the desired count is restored without duplicates; ambiguous database/host failure → alert and block, not a second writer; worker restart and duplicate requests handled; all actions audited.

09 — Bare-metal provisioning and node enrolment (ADR 0002 accepted 2026-10-20)

maas-adapter (discovery, commissioning, deployment, power), node-enrollment (approval, preflight, cloud-init/preseed join with single-use token, validation). Physical servers page with lifecycle and power actions; evacuation offered before power-off. Separate provisioning, management, storage and BMC networks. Accept: a test server is discovered, approved, installed, joined and becomes eligible for placement, with every step audited.

Status (slice 1 done: provider-neutral enrolment with the manual provider; design in node-enrolment.md): machine records (machines, migration 0008) with a provider (manual or maas) and their capabilities; the enrolment state is derived from Incus, never stored — registered, approved (a join token is pending), approval-expired, held (member with scheduler.instance=manual), eligible, left. POST /api/v1/machines registers (an existing member is adopted), …/approve returns the join token once (no-store) with the enrolment command, DELETE …/approval withdraws it, GET …/admission runs the admission checks (online, Incus version and same release, kernel ≥ 6.12, architecture, storage pools and per-member networks created on it; OVN chassis and clock reported as not checked), POST …/admit re-runs them and, when none fails, returns the member to Incus's default placement, DELETE … forgets a machine that is not a member; all need manage-infrastructure, the admin identity and the database, and are audited without secrets. scripts/enrol/enrol-member.sh (run on the server; token on standard input or a hidden prompt) checks the host, configures its OVN controller, joins and holds the member; the lab's add-member.sh now uses it. Fabric Console: Membership → Machines (add, approve with the token and command shown once, withdraw, check and admit with the checks listed, remove; next step per row; phone layout). REAL INCUS TESTED (2026-10-20, Incus 7.0.1): a fourth Lima VM was registered, approved, enrolled in 129 s and came up held — four new instances avoided it; admission passed and Incus then placed two of eight new instances on it; an OVN instance on it reached one on member 1 and resolved it by name; every step audited, no token in the audit log or the control plane's log. The same flow (including a held member's admission, approve/withdraw/remove) was verified in a real browser against the cluster. The real cluster found a decoding defect the mocks had not (locations: null on unmanaged interfaces), fixed with a contract test. MAAS adapter slice 1 (2026-10-21, design): discovery, registration with the MAAS system id, power state and power-off as a fence provider. Burn-in is built (MAAS hardware tests on a machine not in use; a failed or unfinished run blocks admission; CONTRACT VERIFIED). Remaining: a physical servers page. Commission, OS install and release are built (CONTRACT VERIFIED against a MAAS-shaped server; not REAL MAAS TESTED; the cloud-init is not stored). Member removal is built.

10 — CI/CD, security hardening, releases, runbooks

Path-aware GitHub Actions: Rust fmt/clippy/test with --locked, npm typecheck/lint/test/build, Playwright, OpenAPI compatibility, dependency audit and licence scan (cargo deny with our own allow-list of AGPL-compatible licences and the advisory database; the npm equivalent for JS), secret scanning, upstream checks. CI never holds production Incus credentials or runs destructive tests on pull requests. Independent artefacts for each app and service, pinned dependencies, staged upgrades and rollback. Runbooks: member join/removal, quorum loss, instance recovery, stateful restore, rolling upgrades, certificates, API outage, storage and network incidents, access revocation, security incidents. Status (partial): CI runs on every push and pull request — frontend typecheck/lint/unit/build, generated-file drift (JSON Schemas, catalog reference, API client), Rust fmt/clippy/test with --locked, Playwright for both apps, and a dependency job: cargo deny check (deny.toml: permissive licences only, RustSec advisories, no wildcards, crates.io only) plus the same licence allow-list for the npm packages the apps ship and npm audit. The upstream check verifies the Incus pin weekly. Runbooks (docs/runbooks/, each step labelled REAL INCUS TESTED or NOT YET VERIFIED): member join, maintenance (evacuate/restore), member failure, deploy identity without its project, access revocation, control plane or worker outage. The test cluster then had three members (the third joined through Apelora's join-token flow, 2026-10-20); it has four since Phase 09, Lima VMs standing in for physical servers. Secret scanning (scripts/security/scan-secrets.sh, in CI). Release workflow (.github/workflows/release.yml): on a v* tag it builds the control plane and worker for linux x86_64 and aarch64 and the two apps, each packaged on its own with SHA-256 checksums, into a draft release a maintainer publishes. End-to-end demonstration: scripts/test-cluster/demo.sh — REAL INCUS TESTED three times on three members (2026-10-20, all checks passed: replicas apart, service address, enforced connections, internet access, publishing, recovery from a force-stopped member, evacuation; each run removed everything again); status matrix and known limitations: docs/architecture/current-state.md. Certificate rotation (rotate-identity.sh, runbook) REAL INCUS TESTED for both identities. Path-aware jobs: pull requests and pushes to main run only the areas they change (scripts/ci/changed-areas.sh: Rust, frontend and end-to-end; the dependency and secret job always runs), manual runs run everything. Tiered since 2026-10-21 (CI.md): every change runs the @smoke browser tests, the whole suite runs nightly in four shards; actions pinned to commit SHAs; Rust lint and tests in parallel jobs; CodeQL on shipping code; passing Dependabot patch and minor updates merge automatically. Release workflow exercised: v0.1.0-alpha.1 built both services for x86_64 and aarch64 and both apps, with checksums, into a draft release (2026-10-20). Network incident runbook: OVN central database outage REAL INCUS TESTED (running applications kept serving and resolving names; a change made during the outage waited and completed within 10 s of the database returning). Remaining: installing the Incus package and rolling it back (Apelora does not), and runbooks for storage incidents and security incidents. The rolling-upgrade runbook exists and is NOT YET VERIFIED. The Console assistant (evacuate, record the package upgrade, restore) is built, MOCK VERIFIED. Quorum-loss runbook: temporary loss REAL INCUS TESTED (two of three voters force-stopped: instances kept serving, Apelora answered 502 instead of stale data, quorum back within 10 s of the members booting); permanent-loss recovery written, NOT YET VERIFIED. Accept: end-to-end demonstration on a 3-member lab with logs, plus a final status matrix — implemented, mock-tested, real-tested, not implemented, known limitations.

Proposed additions (designs written, not started)

• Applications view and server-side drafts (extends 05 and 06): design.
• Persistent volumes, snapshots, backups (finishes 06, feeds 08 M3): storage.
• Interactive access to modules (extends 05 and 06): terminal sessions built (MOCK and CONTRACT VERIFIED, not yet REAL INCUS TESTED), PowerShell built but off by default, remote desktop built but off by default (APELORA_RDP_GATEWAY, design; never run against a Windows guest). Contract, details.
• Project scoping, role bindings, viewer role, OIDC (extends 03 and 05): access and roles, ADR 0005.