Storage: how Incus does it, and how Apelora abstracts it

Status: target design, partly built (2026-10-21; the authoritative list is current state). Built and REAL INCUS TESTED: the storage pool list with how full each is (GET /api/v1/storage/pools, Fabric Console); root disks from the project's default profile on the member's local pool (btrfs in the lab); data-local volumes (<instance>-<volume> on the pool in APELORA_VOLUME_POOL, on the instance's member), with records, listing, grow and explicit delete; snapshots (manual and scheduled, restore); backups copied into the backup pool on another member (scheduled or on request); moving an instance with its volumes (instance.move). Built 2026-10-21 (MOCK VERIFIED; the restore job REAL INCUS TESTED on the lab, the dry run and the Console dialog were not part of that run): the backup API and the Console's Storage page — backups per volume, back up now, delete a backup with its name typed (MOCK VERIFIED, not run on the lab), schedules, restore from a backup (§10, step 5) and volumes no application uses. Built 2026-10-21 (HA slice B, ADR 0013): the data-replicated class on the shared pool (APELORA_SHARED_POOL), with the module's root disk there too, and restart on another member after its member fails and is fenced — REAL INCUS TESTED on the lab's shared pool (clustered LVM on an iSCSI SAN, scripts/test-cluster/shared-storage.sh; drill timings in the member-failure runbook). Not built: the Console view for classes, choosing data-replicated in the Studio (the API accepts it), deleting a backup, Ceph-specific fencing. How a module's volumes fit the rest of the chain: platform architecture §4.1. Statements about Incus below come from its documentation unless §10 records them as REAL INCUS TESTED on the pinned release (7.0.1); each build step verifies what it relies on, on the disposable test cluster, and records the result. Date: 2026-10-20. Related: platform architecture §8, ADR 0001, skills storage-recovery and storage-volumes.

1. How Incus models storage

Diagram source

flowchart LR
  subgraph CL["Incus cluster"]
    M1["Member 1"]
    M2["Member 2"]
    M3["Member 3"]
  end
  subgraph PR["Project: team-a (restricted)"]
    PROF["Profile<br/>root disk, NIC"]
    INST["Instance<br/>web-01"]
    CV["Custom volume<br/>data-db-01"]
  end
  POOL1["Pool: local<br/>driver: btrfs in the lab (zfs, lvm, dir possible)<br/>one copy per member"]
  POOL2["Pool: shared<br/>driver: lvmcluster on a SAN, or distributed storage<br/>visible to all members"]

  INST -- "root disk device" --> POOL1
  PROF -- "supplies defaults" --> INST
  INST -- "disk device: source=data-db-01" --> CV
  CV -- "lives in" --> POOL2
  M1 --- POOL1
  M2 --- POOL1
  M3 --- POOL1
  M1 --- POOL2
  M2 --- POOL2
  M3 --- POOL2

1.1 Local pools and shared pools (the part that decides HA)

Rule from the architecture: a local pool across several members is not shared data. Apelora must say so in plans instead of implying failover that cannot happen.

Two kinds of shared pool, side by side: clustered LVM on a SAN (lvmcluster: every member logs in to the same block device over iSCSI, Fibre Channel or NVMe over fabrics, and a lock manager coordinates the volume group; thick volumes, no snapshots of shared custom volumes) and distributed storage (a separate cluster of storage machines that replicates data). The lab runs the first: pool shared on an iSCSI LUN from the apelora-san VM (test cluster). Apelora puts data-replicated volumes and their modules' root disks there and restarts them on another member once a failed member is fenced (slice B of moving modules, ADR 0013). On lvmcluster Incus activates container file systems exclusively but VM disks and block volumes shared, so only the failed host's dead sanlock lease, not the LV lock, rules out a second writer of a VM disk.

2. Where Apelora fits: one authority per piece of state

Apelora never writes Incus's database and never invents its own volume store or cluster filesystem.

3. The abstraction: designers pick a class, not a pool

A design must not contain pool names, driver names, hosts or addresses (same rule as hosts: architecture §4). Instead:

• An module class declares the volumes it can use: name, mount path, default size, whether required, and a suggested class. (A database class asks for data; a web server asks for none.)
• A volume class is an Apelora policy an administrator defines and maps to a pool. The designer chooses a class and a size.
• Administrators map each class to a pool in the Console. If no pool can satisfy a class, the plan says so and refuses; it never silently falls back to a weaker pool.

Diagram source

flowchart TB
  subgraph D["Design (Studio) — provider-neutral"]
    A["Module: postgresql<br/>volume data · class data-replicated · 50 GiB"]
  end
  subgraph AP["Apelora control plane"]
    PL["Planner<br/>capacity, class → pool, anti-affinity,<br/>replica and volume rules"]
    DB[("PostgreSQL<br/>desired volumes, ownership, retention")]
    VP["VolumeProvider<br/>create · attach · detach · snapshot ·<br/>restore · grow · delete"]
    AD["incus-adapter"]
  end
  subgraph IN["Incus"]
    PO["Pool (admin-chosen)"]
    VO["Custom volume<br/>labelled user.apelora.*"]
    DV["disk device on the instance"]
  end
  A --> PL --> DB --> VP --> AD --> VO
  PO --> VO --> DV

3.1 Rules (carried from the storage-volumes skill, restated for this design)

1. Removing a module from the canvas never deletes data. Deleting a volume is an explicit, typed-name confirmed action; retained volumes show as orphaned until an owner decides.
2. A read-write block volume attaches to at most one instance. A design that gives several replicas the same exclusive volume is invalid (validation error with a hint: use one volume per replica or a shared filesystem class).
3. Resize is grow-only by default; shrinking is not offered.
4. Snapshots are taken stopped or with an application-aware hook unless the backend guarantees consistency; Apelora records which one was achieved. Snapshots are not backups.
5. All operations are idempotent and level-triggered like every other desired state: re-read Incus, then act.
6. Encrypt at rest where the backend supports it; keys are referenced, never stored in designs or the database.
7. Ownership labels user.apelora.application, user.apelora.module, user.apelora.volume are written only by Apelora; anything without them is never touched.

4. Data model (PROPOSED)

Diagram source

erDiagram
  APPLICATION ||--o{ REVISION : has
  APPLICATION ||--o{ VOLUME : owns
  REVISION ||--o{ VOLUME_SPEC : declares
  VOLUME_CLASS ||--o{ VOLUME : "defines policy of"
  VOLUME_CLASS }o--|| POOL_MAPPING : "resolves to"
  VOLUME ||--o{ SNAPSHOT : has
  VOLUME ||--o| ATTACHMENT : "attached by"
  MODULE_INSTANCE ||--o{ ATTACHMENT : uses

  VOLUME {
    string id
    string application
    string project
    string module_id
    string name
    string class
    int size_gib
    string pool "resolved, never in the design"
    string state "requested|creating|available|attached|retained|deleting|error"
    string retention "delete-with-app|retain"
    int generation "fences stale workers"
  }
  VOLUME_CLASS {
    string name
    string kind "scratch|data-local|data-replicated|shared-read-only|backup-target"
    int rpo_minutes
    int rto_minutes
    string snapshot_schedule
    string snapshot_expiry
  }

5. Lifecycle

Diagram source

stateDiagram-v2
  [*] --> Requested : plan approved
  Requested --> Creating : worker claims job
  Creating --> Available : volume exists in the pool
  Creating --> Error : pool full / class unavailable
  Available --> Attached : disk device added
  Attached --> Available : stop and detach
  Attached --> Attached : grow (online if supported)
  Available --> Retained : application removed, retention = retain
  Retained --> Available : application redeployed
  Available --> Deleting : explicit delete, typed name
  Retained --> Deleting : explicit delete, typed name
  Deleting --> [*]
  Error --> Requested : operator fixes cause, retries

5.1 Deploying a module with a volume

Diagram source

sequenceDiagram
  autonumber
  actor D as Designer / operator
  participant S as Studio
  participant API as Control plane
  participant W as Worker
  participant I as Incus

  D->>S: Set volume data = 50 GiB, class data-replicated
  S->>API: POST /applications/shop/plan
  API->>API: Resolve class to pool, check capacity and rules
  API-->>S: Plan: create volume, create instance, attach (or refusal with reason)
  D->>S: Approve
  S->>API: POST /applications/shop/deployments
  API->>API: Save revision, desired volumes, queue jobs, audit
  W->>I: Create custom volume (project, pool, size, labels)
  W->>I: Create instance with root disk
  W->>I: Add disk device for the volume, start
  W->>I: Readiness check
  W-->>API: Record result
  S->>API: GET status (volume state, instance state)

6. Interaction with mobility and recovery

The evacuation dry run lists every volume that blocks a move and why. It never reports "no downtime" for stateful workloads.

7. Security

• Volumes are created in the application's project, with project limits on disk and volume counts; tenants cannot reach other projects' volumes.
• Restricted projects allow only managed (Apelora-created) disk devices: no host path mounts, no raw device passthrough for tenants.
• Pool configuration and driver credentials (for example Ceph keys) are administrator-only and live in the backend's secret store, never in the browser, designs, logs or the repository.
• Storage actions require their own permission (manage-storage, see roles) and are audited.
• Backup copies are independent of the pool they protect.

8. Failure handling

9. UI

• Studio: a Volumes section in the module properties (add volume, class, size, mount path); a small disk badge on the node; validation messages for rule 2 and unavailable classes; the deploy review lists volumes to create, keep, grow or leave retained.
• Fabric Console, Storage: pools (exists), volume classes and their pool mapping (admin), volumes per project with state, attachment, size and used space, snapshots, orphans; destructive actions need the typed name.

10. Build order

Step 1 progress (REAL INCUS TESTED, 2026-10-20, Incus 7.0.1, btrfs pool local on four members, through the restricted deploy identity):

Step 1 is complete. Consequences for step 2 and later: create order is instance (stopped) → volume on its member → disk device → start; grow-only and deletion are Apelora's rules, not Incus's; a project needs restricted.snapshots=allow before snapshots are offered; evacuation of a member with data-local volumes means downtime, never a move.

Step 2 progress (part A, 2026-10-20): volumes are part of the instance spec (apelora_domain::storage::VolumeMount: name, size in MiB, filesystem or block, class, mount path; at most four; block only for virtual machines; stateless modules cannot have them). The control plane resolves the class to the pool in APELORA_VOLUME_POOL and refuses a design with volumes when it is unset or Incus has no such pool; a caller can never name a pool. The worker creates the instance stopped, then each volume <instance>-<volume> on the instance's member with ownership labels, attaches it as vol-<volume> and starts the instance; a volume that exists without Apelora's labels or for another application is refused. Deleting the instance keeps the volume. The adapter's operations are methods of Provider (as for networks) rather than a separate trait; the mock holds volumes and injects create and attach failures. REAL INCUS TESTED: volumes_follow_the_instance_and_outlive_it. Part B (2026-10-20): the worker records each volume as soon as it exists (volumes, migration 0012; an application with recorded volumes cannot be deleted from the database). GET /api/v1/applications/{name}/volumes lists them with what Incus reports now (attached, kept, missing); DELETE …/volumes/{volume} deletes one with its snapshots after the name is typed again, refused while an instance uses it, needs application.delete, audited application.volume.delete. The Fabric Console shows them under Applications → Show details. REAL INCUS TESTED (four members): volumes on two members, one kept after its module was removed, both deleted from the console and the API. Runbook: docs/runbooks/volumes.md. Step 3 (2026-10-20): @apelora/assembly-model has VolumeSchema (name, size in MiB, mount path; unique names and paths, at most four), class defaults in runtime.volumes and per-module volumes (absent: the class's). instanceRequestFor sends them as spec.volumes and refuses them for stateless classes. The Studio's Properties window has a Volumes section (add, rename, mount path, size in GiB, remove; every change checked before it is kept) and the deploy review lists what each module keeps. Filesystem volumes only in the Studio for now; block volumes for virtual machines are accepted by the API. Grow: PATCH /api/v1/applications/{name}/volumes/{volume} with sizeMib (needs application.deploy, audited application.volume.grow) and Grow… in the Fabric Console; grow-only, refused before Incus is asked; REAL INCUS TESTED on a running instance. Snapshots (step 5, first part): GET|POST /api/v1/applications/{name}/volumes/{volume}/snapshots, POST …/snapshots/{snapshot}/restore (name typed, refused while an instance using the volume runs, approval where the policy asks), DELETE …/snapshots/{snapshot}; snapshots are named snap-YYYYMMDD-HHMMSS and the response says crash-consistent or stopped; a project without restricted.snapshots=allow is reported as such. REAL INCUS TESTED (adapter test, and take, refused restore, restore after stop and delete from the Fabric Console against the cluster). Next: the Console Storage view for classes and orphans (step 4), scheduled snapshots and backups to a separate pool.

Step 4 and 5, backups and restore (2026-10-21, MOCK VERIFIED with a real PostgreSQL):

• Read side. GET /api/v1/storage/pools (cluster.view): each pool's use per member from the members' state, a shared pool counted once, and what Apelora uses it for (APELORA_VOLUME_POOL, APELORA_BACKUP_POOL, which the control plane now reads too). GET /api/v1/storage/volumes (per application the caller may view): every recorded volume with what Incus reports (one custom-volume list per project and pool), its schedules and how many backups it has. GET /api/v1/storage/orphans (cluster.view): custom volumes in Apelora's projects (deploy projects and every environment's) that neither the volumes nor the volume_backups table records, with their size and why they are listed; read-only.
• Per volume. GET …/volumes/{volume}/backups (application.view) lists its backups, oldest first, with whether each still exists in Incus and the volume's schedules. POST …/backups (application.operate) queues a volume.backup job (the existing worker job; its audit row application.volume.backup.requested is written with the job). PUT …/schedules (application.deploy) sets or clears the snapshot and backup schedules (15 minutes to a week, keep 1–50; backups need a backup pool), audited application.volume.schedules in the same transaction.
• Restore from a backup. GET …/backups/{backup}/restoration is the dry run (apelora_domain::storage::plan_restore): the volume, the backup and when it was taken, the volume's snapshots that will be deleted, the instances using it and which run, where the safety backup goes, what blocks it and the steps. POST …/restore runs the same check again, needs the volume's name typed, application.operate (and an approval where a policy asks) and queues a volume.restore job with its audit row (application.volume.restore.requested) in one transaction. The job (deployment-worker/src/storage.rs) re-reads Incus before each change: refused (nothing changed) while an instance using the volume runs or when the volume or the backup is missing; then a safety backup of the current data into the backup pool, recorded with the job (volume_backups.before_restore, migration 0024), so a retried attempt finds it instead of taking another; then a safe point (a cancellation stops here, keeping the safety backup); the check again; then an Incus refresh copy of the backup over the volume — one Incus operation, no safe point inside; a copy that fails part-way is repeated whole by the next attempt and the safety backup holds what was there. Incus deletes the volume's own snapshots during a refresh and would accept it under a running instance (REAL INCUS TESTED earlier, test cluster): Apelora refuses that first. The instances stay stopped; the person starts them after checking the data. A failed restore is not retried as is (the dry run is read and the name typed again). Audited application.volume.restored by the worker. Runbook: volumes.

1. Verify on the test cluster, labelled REAL INCUS TESTED: custom volume create, attach to a container and to a VM, snapshot, restore, grow, delete, and the project restrictions in §7; write down what the pinned release actually does.
2. VolumeProvider trait, fake provider with fault injection, adapter implementation, volumes table and migration.
3. Module class volumes field and Studio properties section; planner rules; deploy review.
4. Console Storage: classes, volumes, orphans.
5. Snapshots and scheduled backups with a restore drill; then relocation of data-local volumes (stop, copy, start).
6. Only with a verified shared pool and tested fencing: data-replicated recovery (M3).