Identity and authorization

Status: built and tested against stubs and a mock provider; browser sign-in tested in a real browser against a local development identity provider; not yet verified against an enterprise provider in production or MAAS. Environments' restricted Incus projects and their limits are REAL INCUS TESTED on the lab (§4). Design rationale: access and roles and ADR 0005. Working checklist for contributors: the identity-authorization skill.

Evidence labels: BUILT (in the repository, covered by tests), NOT VERIFIED (written from documentation, exercised only by stubs).

1. Model

Diagram source

flowchart LR
  P["Person<br/>(OIDC sign-in)"] --> A
  G["Group<br/>(from the groups claim<br/>or created in Apelora)"] --> A
  S["Service identity<br/>(automation)"] --> A
  A["Role assignment<br/>role + scope + optional<br/>conditions and expiry"] --> E["Evaluator<br/>deny by default"]
  R["Role<br/>(11 predefined, custom)"] --> A
  E --> D{"Allowed?"}
  D -- "needs approval" --> AP["Approval by a<br/>different person"]
  D -- yes --> ACT["Action"]
  ACT --> AU[("Append-only audit")]
  D -- no --> AU

• Scopes nest: platform → organisation → project → application (and pool for infrastructure). An assignment at a scope applies to everything below it.
• Permission classes: Tenant (applications, projects, members), Infrastructure (cluster, networks, storage, hardware) and Security (identity, policy, emergency access). Each permission has a floor, the narrowest scope it may be granted at; only Tenant permissions can be granted below platform level, so a project role can never reach servers, the cluster or the platform's own security settings.
• Predefined roles (ten) and custom roles are sets of catalogue permissions. Nothing checks a role name.
• Groups from outside Apelora (ADR 0016). A group can take its members from an identity provider's groups claim or a directory. The mapping records its source (the provider's issuer, or directory:<id>) and the external name (for a directory, the group's full DN in one spelling, ldap::normalize_dn). A credential's groups match only mappings of its own source, so another provider, or another branch of the directory, cannot produce a member. Mapping a group (PUT /api/v1/identity/groups/{group}/external) passes the escalation guard for each role the group holds, as adding a member does (membership_refused). Mappings made before sources were recorded match nobody: at the first start after migration 0033 the control plane binds them to the only source when exactly one identity provider (and nothing else) is configured; otherwise the API marks them needsSource and the Console asks for one.
• Service identities hold roles like people but are withheld from security and people-management permissions, and never get interactive sessions.
• Personal tokens (apt_) belong to one person and act as that person, with what they hold at each request; they never open interactive sessions or make credentials (section 3).
• Legacy apl_ tokens stay platform-wide and exempt from approvals and multi-factor rules (they cannot present either); they pass the same route gate as everyone else, through their role's legacy mapping, and cannot create fabric-admin tokens. Set APELORA_LEGACY_TOKENS=disabled once people sign in through OIDC.

2. Request path

Diagram source

sequenceDiagram
  participant B as Browser
  participant API as Control plane
  participant DB as PostgreSQL
  B->>API: request + bearer (OIDC access token or apl_ token)
  API->>API: validate token (signature, issuer, audience, expiry)
  API->>API: route gate (GATES): route listed? else deny
  API->>DB: resolve resource in caller's scope (404 if not visible)
  API->>API: allow(permission, exact resource), MFA/approval policy
  API->>DB: one transaction: change + audit row + outbox message
  API-->>B: result (+ per-resource `actions` for the UI)

Rules that are enforced and tested (API integration tests, 22 for identity, 13 for application management):

• A route not in GATES is refused for every caller (people, services, legacy tokens and the development principal).
• Every handler checks the exact resource; objects the caller cannot see answer 404.
• The escalation guard (check_assignment) stops anyone granting more than they hold or at a wider scope; the preview endpoint uses the same checks as create. Adding someone to a group is granting them the group's roles: group.manage is not enough when the group holds roles; the caller must be able to grant each of the group's live assignments to that person (403 membership_refused, audited as a denied group.member.add). Removing someone from a group takes those roles away, so it passes the revocation test below for each of them (403 membership_refused, audited as a denied group.member.remove). A group with no roles stays a group.manage matter. Revoking uses the same test (check_revocation): only someone who could grant the role at that scope may revoke it (403 revocation_refused, audited); emergency grants can also be ended with platform.emergency_access.
• Tokens never exceed their creator. An API token's role is refused (403 token_exceeds_caller, audited as a denied token.create) when it carries any permission the creator does not hold at platform level. A fabric-admin token needs another person's approval of token.create_admin on token:fabric-admin; legacy tokens cannot present one and are refused (approval_unavailable). Every token expires: 1–365 days, 90 by default, through the API and the command line (--expires-in-days); a service token asking for more is refused with 400, not shortened.
• The last super admin stays. Disabling a person, revoking an assignment or removing a group member is refused (409 last_super_admin, audited as denied) when it would leave no enabled person holding platform-super-admin at platform level, directly or through a group membership recorded in Apelora. The change and the count run in one transaction under an advisory lock, so concurrent changes cannot both pass. Membership through an identity provider's group claim is not counted (it is only known at sign-in). With no super admin at all (a platform run on API tokens), nothing is refused.
• Approvals: flagged permissions need a recorded approval by a different person for the exact action and target, single use, 30 minutes (Apelora-Approval header). Self-approval is refused. Only a signed-in person decides: approving or rejecting from an API token (legacy apl_; personal and service tokens are already refused) answers 403 approval_needs_a_person, audited as denied, because a token is a principal of its own and its holder could otherwise approve their own request.
• Emergency access: at most 4 hours, a written reason, cannot be self-granted, every request under it audited.
• Audit is append-only (database triggers refuse UPDATE, DELETE and TRUNCATE) and written in the same transaction as the change it describes; reads are scoped to what the caller may see. Denials of sensitive actions are recorded. Details in section 8.

3. Sign-in

How people and administrators use it, with screenshots and troubleshooting: Signing in.

OIDC access tokens are validated by crates/identity: asymmetric algorithms only (no none, no shared-secret algorithms), issuer and audience pinned, exp/nbf/iat checked with 30 s leeway, maximum lifetime, key set fetched over TLS only (loopback allowed for development) and cached. An application end-user token with another audience is refused. MFA is derived from configured amr/acr values and can be required by an access policy.

Where sign-in is configured. Platform administrators (platform.security.manage) add, edit, turn off and remove identity providers in the Fabric Console (Identity & Access → Sign-in); changes apply at once, without a restart, and are audited (identity.provider.saved, identity.provider.deleted). Any OpenID Connect provider works. Per provider: name on the button, issuer, audiences, an optional public browser client ID (no secret; the browser uses authorization code with PKCE, state and nonce), scopes, the groups and username claims, the amr/acr values that count as multi-factor, and the clients a token must be issued to (azp). Check connection reads the provider's discovery document and key set before saving. A provider set by environment variables (APELORA_OIDC_ISSUER, APELORA_OIDC_AUDIENCE, APELORA_OIDC_CLIENT_ID, APELORA_OIDC_SCOPES, APELORA_OIDC_LABEL and the claim variables; see .env.example) keeps working and is shown read-only.

Choosing the provider for a token. The control plane reads the token's iss (unverified) only to pick the configured provider with that issuer, then verifies the token against that provider alone. An issuer nobody configured is refused.

API tokens (apl_) are created by a person or token that holds everything the token's role can do, expire within a year, and a fabric-admin one needs an approval (see section 2). They can be turned off in the Console once people sign in with a provider (identity.api_tokens.disabled). APELORA_LEGACY_TOKENS=disabled turns them off for good; the Console cannot turn them back on.

Personal tokens (built 2026-10-21; services/control-plane/src/api/account.rs, crates/persistence/src/personal_tokens.rs, migration 0026_personal_tokens.sql). A person makes tokens for their own scripts under Your tokens in the Console's user menu (GET/POST /api/v1/me/tokens, POST …/{id}/rotate, DELETE …/{id}).

• Prefix and storage. apt_ + 64 hex (256 random bits), stored as SHA-256, shown once (Cache-Control: no-store), never logged or audited. A separate prefix (not apl_) because these are not legacy role tokens: they keep working when APELORA_LEGACY_TOKENS=disabled, the secret scan and the credential refused log line can name them, and nothing that checks apl_ changes meaning.
• What a token can do. Nothing of its own: each request made with it resolves the person's role assignments and Apelora group memberships at that moment (caller_for), so a revoked role, a removed membership or a disabled account takes effect on the next request. Groups that come only from an identity provider's token are not known and do not apply. A token never counts as multi-factor, so policies that require it refuse tokens. A local account's tokens stop when local accounts are turned off.
• What it can never do (account::REFUSED_TO_PERSONAL_TOKENS, checked by the route gate, 403 personal_token_refused, audited as access.denied): open terminal or PowerShell sessions (also refused by the session access check), create or rotate personal tokens, create API tokens, service identities, local accounts or setup links, cluster join tokens, change a password, approve someone else's request, grant emergency access. A leaked token therefore cannot outlive its own expiry or mint a stronger credential.
• Lifetime. 1–365 days, no default (400 when missing). At most 20 live tokens per person (409 too_many_tokens, counted under the person's row lock).
• Rotation. POST /api/v1/me/tokens/{id}/rotate {overlapMinutes 0–1440 (default 60), expiresInDays?} makes a successor with the same name (the old token's lifetime unless given) and shortens the old token to the overlap; a token is rotated once (409 already_rotated; rotate the successor). Both are listed: the old one as Replaced until it expires.
• Revocation. The person revokes their own (DELETE /api/v1/me/tokens/{id}; another person's id answers 404); administrators with token.manage list and revoke anyone's (GET /api/v1/identity/personal-tokens[?user=], DELETE …/{id}; Console: Identity & Access → Sign-ins & tokens). Disabling a person revokes their tokens in the same transaction (revoked_by = disabled), so enabling them again brings no old token back.
• Audit. personal_token.create, personal_token.rotate, personal_token.revoke, each in the change's transaction, with the token's id and name, never the secret.

Active sign-ins (built 2026-10-21). A person sees their own Apelora sessions (password and directory sign-ins) under Your sign-ins (GET /api/v1/me/sign-ins: method, signed in, last used, the absolute limit and when it ends unless used again, which one is this browser) and ends any of them (DELETE /api/v1/me/sign-ins/{id}). Administrators with user.manage see and end anyone's (GET /api/v1/identity/sign-ins[?user=], DELETE …/{id}). Audited as auth.session.ended. A sign-in with an identity provider's token is not a session Apelora holds: it is not listed, it ends when the provider's token expires (at most an hour), and the page says so (credential: identity-provider in the response and in whoami).

Local accounts (ADR 0007, design; services/control-plane/src/api/sessions_auth.rs, crates/persistence/src/local_accounts.rs, user_sessions.rs, migration 0017_sign_in_methods.sql). A local account is a users row with the issuer apelora:local and the lowercase username ([a-z0-9._-]{2,64}, also a database CHECK) as subject, plus a local_credentials row. Usernames are unique across every issuer. The setting auth.local_accounts is off by default (PUT /api/v1/identity/sign-in/local-accounts, platform.security.manage, audited). An administrator with user.manage creates an account (POST /api/v1/identity/local-accounts) or resets one (POST …/{id}/reset); both return a one-time setup token (apw_ + 64 hex, 24 hours, stored as SHA-256, Cache-Control: no-store) that the Console turns into the link #/set-password?token=… (a fragment, so it never reaches a server log). The person sets the password with POST /api/v1/sign-in/setup; the link is spent atomically. A reset removes the password and revokes every session; nobody can reset their own account.

• Hashing. Argon2id (RustCrypto argon2), memory 19 456 KiB, 2 passes, 1 lane, PHC string in password_hash, computed on the blocking pool. Policy: 12 to 128 characters, not the username; no composition rules.
• Sign-in (POST /api/v1/sign-in/password, public). Every refusal answers 401 invalid_credentials "The username or password is incorrect.", and every path costs one Argon2id verification: unknown, disabled, locked and not-yet-set-up accounts verify against a dummy hash made with the same parameters (sessions_auth::plan decides; tests count the dummy verifications). Five failures in a row lock the account for 15 minutes (locked_until, database clock); attempts during a lock are not counted; a success resets the count. Each client address gets 10 attempts a minute across sign-in, setup and password change. The address is the one the rate limits use (rate_limit::client_ip): the TCP peer, or, when the peer is a proxy listed in APELORA_TRUSTED_PROXIES, the address it forwarded for; IPv6 clients count per /64. The same address is recorded in the sign-in audit rows and terminal sessions (in memory; 429 rate_limited with Retry-After). Passwords longer than 512 bytes are never hashed. With the optional directory field the same endpoint binds to that directory as the person (directories::verify_directory_sign_in), creates or refreshes them (upsert_directory_user) and issues a session with the directory's groups as external groups; a directory not offered for sign-in answers 400 directory_unavailable. With local accounts off: 400 local_accounts_disabled.
• Sessions. issue_session(pool, user_id, method, external_groups) is the one place apu_ tokens are made (32 random bytes as hex, stored as SHA-256). resolve_principal accepts a session that is not revoked, before its absolute expiry (8 hours), used within the last hour, whose person is not disabled, and (for password sessions) while local accounts are on; every use updates last_used_at. Groups are the person's Apelora groups plus groups mapped to the session's group_source (a directory sign-in records directory:<id>) whose external name is in the session's external_groups; mfa is false, so policies that require multi-factor refuse local accounts. Sessions end on POST /api/v1/sign-out, a reset, a password change (all others), disabling the person (in the same transaction) and turning local accounts off. whoami.localAccount tells the UI to offer Change password (PUT /api/v1/me/password, current password checked and counted towards the lock).
• Audit. auth.password.succeeded, auth.password.failed (username, reason unknown|wrong|locked|disabled, address), auth.locked, auth.setup.completed, auth.password.changed, auth.signed_out, identity.local_account.created|reset, identity.local_accounts.enabled|disabled; never a password or token. Failures by unknown usernames use the actor anonymous. A username typed into the password field could still end up in the audit as an unknown username; that is the price of auditing failed attempts by name, and the per-address limit bounds the volume.

Nobody is locked out. The server refuses: turning API tokens off while signed in with one, or while no other way in remains; removing or turning off the last answering provider while API tokens are off and no local account can sign in; turning local accounts off while signed in with one, or while API tokens are off and no provider answers. Local accounts count as a way in when they are on and at least one account has a password and is not disabled. The guards are enforced by the API (auth_config::lockout_guard, set_api_tokens, guard_leaving_local_accounts), not only by the Console.

Sign-in page. The Fabric Console shows one button per provider with a browser client ID, a username and password form when local accounts are on (first without a provider, otherwise below the buttons after "or"), then "Use an API token instead" while API tokens are accepted. The access token is kept for the browser tab only (sessionStorage). If the provider signs someone in but Apelora refuses its token, the page says why (for example "the token lives longer than Apelora accepts": tokens may live at most an hour).

Troubleshooting. Each refused credential is logged at warn as credential refused with method, path, credential kind (identity-provider token, Apelora session, service token, API token), the token's issuer and the reason; the token itself is never logged. Refusals are not written to the audit log (one row per bad request would let anyone fill it); a disabled account signing in is (auth.refused). At startup and after every change the control plane logs each provider as identity provider accepted for sign-in or identity provider not reachable with the error; unreachable providers are retried every minute.

Trying it locally. docker compose -f infra/dev/compose.yaml --profile oidc up -d starts a development identity provider (infra/dev/dex.yaml; development credentials only). Add it in the Console with issuer http://127.0.0.1:5556/dex and audience and client ID apelora-console. A realm setup guide with no secrets is in infra/identity/keycloak.

Evidence: provider configuration, lockout guards, discovery and token verification against a local HTTP provider in the API tests (MOCK VERIFIED); the full browser round trip with PKCE against the local development provider in a real browser (sign-in, whoami as the new user, refusal of a 24-hour token with its reason) on 2026-10-21. Enterprise providers in production are NOT VERIFIED. Local accounts: every rule above has an API test against PostgreSQL (sessions_auth_tests.rs); REAL-BROWSER TESTED on 2026-10-21 against a real control plane and the development PostgreSQL (setup link, wrong then right password, the user menu, sign-out revoking the session on the server). The Argon2id cost was not timed on production hardware.

Directories (LDAP over TLS) (ADR 0007). Saved in the directories table (migration 0018) and managed under Identity & Access → Sign-in → Directories (GET/POST /api/v1/identity/sign-in/directories, PUT/DELETE …/{id}, POST …/check; platform.security.manage; audited identity.directory.saved|deleted; removal needs the name typed). Rules, in services/control-plane/src/api/ldap.rs and directories.rs:

• ldaps:// or ldap:// with StartTLS only; plain LDAP is refused when saving (API and a database check). TLS through rustls (aws-lc-rs) with certificate and host-name verification always on, against the directory's CA (PEM, 1 to 16 certificates) or the system's roots. TLS 1.2 or later.
• Apelora binds as the person: the bind name is the template with {username} (a DN, or a principal name such as {username}@example.org, which then needs a base DN and user filter). No service-account password exists. The person's own bind reads their entry (base search on the bind DN, or the user filter under the base DN; exactly one entry) and groups (an attribute such as memberOf, a group search with {dn}/{username}, or both).
• The username is escaped per RFC 4514 inside DNs and RFC 4515 inside filters; a principal-name username with @, \ or / is refused; an empty password is refused before connecting (an empty-password bind is anonymous). Settings are validated: one {username} in the template, filters parsed, attribute names [A-Za-z][A-Za-z0-9-]* or numeric OIDs, URL only scheme, host and port.
• A whole sign-in has 10 s (connect, TLS, bind, searches); the list's reachability check (a TLS connection, run in parallel on GET /api/v1/identity/sign-in) has 5 s.
• Identity: users.issuer = the directory URL, users.subject = the configured id attribute, else entryUUID, else objectGUID as hex, else the DN. A directory's groups are named by their full DN (normalized: CN=Ops, OU=Groups and cn=ops,ou=groups are one group; groups found by the search are named by their entry's DN), become the session's external groups with directory:<id> as their source, and map onto Apelora groups mapped to that directory and DN (ADR 0016).
• Sign-in functions: verify_directory_sign_in(state, directory_id, username, password) → DirectoryIdentity (audits auth.directory.failed with username and reason; 401 invalid_credentials, 503 directory_unreachable, 502 directory_entry_unreadable, 400 directory_unavailable) and upsert_directory_user (refuses a disabled person with 403 account_disabled; audits auth.directory.succeeded). POST /api/v1/sign-in/password with directory calls both, then issues the session with method directory (wired together with local-account sessions). GET /api/v1/sign-in lists enabled directories as { id, name }.
• Passwords are never logged, audited, returned or stored; the check's test password is used for one bind.
• Directories do not yet count as a way in for the lockout guards (they cannot sign anyone in until the password endpoint exists).

Evidence: escaping, validation and attribute/group mapping in unit tests; TLS failures (untrusted CA, host-name mismatch, unreachable) and the deadline against local TLS servers in the API tests; sign-in, wrong password, injection attempts, StartTLS, plain LDAP refused, untrusted CA and host-name mismatch REAL DIRECTORY TESTED against the development directory (OpenLDAP in infra/dev, profile directory) on 2026-10-21; the Console's Directories section REAL-BROWSER TESTED against the same directory (add, test sign-in, save, status). Principal-name directories (objectGUID, userPrincipalName) are written from documentation and NOT VERIFIED.

4. Backends

Apelora never connects people to Incus. It maps its decisions onto each backend's own controls:

Tenancy isolation and quotas

Built 2026-10-21 (design, ADR 0012, migration 0027, services/control-plane/src/api/tenancy.rs, crates/domain/src/tenancy.rs, crates/incus-adapter/src/tenancy.rs, runbook).

• The environment's Incus project. Created when the environment is created (or on POST /api/v1/identity/environments/{id}/incus-project, environment.manage on its project, or at its first deployment) with a fixed baseline of restricted.* keys, its own default profile, and the project added to the deploy identity's list under a cluster-wide lock, then read back with the deploy identity. An existing project is adopted only when restricted, and keeps its own restrictions (differences shown as findings). Projects in APELORA_DEPLOY_PROJECTS are adopted only on request. Audited environment.incus_project.prepare (success or failed) after Incus answered, with the new state in the same transaction.
• Quotas (tenancy_quotas): instances, CPUs, memory and disk on organisations, projects and environments. GET|PUT /api/v1/identity/quotas/{scope}/{id}; GET /api/v1/identity/tenancy for the page.
  • Setting one needs quota.manage (new; infrastructure, platform, held by Fabric Administrator and Platform Super Admin; legacy fabric-admin tokens through manage-infrastructure); or, below the organisation, organization.update on the organisation (projects, environments) or project.update on the project (environments). Tenants may never exceed a quota above them (400 quota_above_parent); nobody but quota.manage sets an organisation's quota.
  • Seeing one needs organization.view (organisation) or project.view (project, environment). An organisation's use is shown only to those who see the whole organisation.
  • quota.set is audited in the change's transaction with before and after; mirroring to Incus runs after the commit (environment.incus_limits.apply, success or failed).
• Enforcement. Plans (POST /applications/{name}/plan) carry quota (use, what the deployment adds, limit, per scope and dimension); deployments and POST /instances over a quota are refused with 409 quota_exceeded before anything is saved or queued, and with 503 quota_unverifiable when a scope with a quota cannot be read from Incus (fail closed). Each environment's Incus project gets the tightest of its own, its project's and its organisation's quota as limits.*, so Incus refuses what Apelora would.

5. User interfaces

• Fabric Console → Identity & Access: Users, Groups, Organizations (→ projects → environments), Tenancy & quotas (each environment's Incus project, restrictions and limit use; quotas and their use; Set quota…, Prepare Incus project), Project memberships, Roles & permissions, Service identities, Access policies (+ Approvals), Audit history. The assignment dialog asks the server for a preview and shows effective permissions with their origin before anything is saved; emergency access has its own warning dialog; service tokens appear once in a copy-once dialog and leave the DOM when closed. Pages appear only if whoami.capabilities holds a matching permission.
• Studio → Applications: every action is enabled from the actions the API returns for that caller and application, with the reason when disabled.

6. Known gaps

1. ApplicationInfo has no id, so an application-scoped assignment needs a pasted UUID.
2. preview_assignment does not run the principal checks that create runs (disabled user, unknown service identity); the error appears at save time.
3. Assignments and memberships carry ids; names are resolved from lists the caller may not be allowed to read, so some show as ids. Audit rows written since migration 0022 also carry the actor's name (actorName); older rows have only the id.
4. No endpoint previews a custom role's assignable levels before creating it.
5. Remote desktop (RDP) is built but off unless APELORA_RDP_GATEWAY=enabled: the same tickets and rules carry the RDP stream between the person's own client (through apelora-desktop-tunnel) and a fixed relay inside the guest (design); without the setting the API reports it unavailable with that reason. Never run against a Windows guest (NOT VERIFIED). PowerShell into Windows modules is wired through the same exec channel but has never run against a Windows guest (NOT VERIFIED).
6. Terminal sessions have only run against a labelled mock shell and an Incus-shaped contract fixture, never a real Incus (see section 7).
7. Real Keycloak and MAAS 3.8 behaviour must be verified before these are called production-ready. Incus restricted projects, their grant and their limits are REAL INCUS TESTED on the lab (ADR 0012); a deployment of real instances into an Apelora-created project has not run there yet.
8. The Console's token dialog cannot send an approval, so a person creates a fabric-admin token through the API (with Apelora-Approval) or with the command line on the server. The dialog still offers every role and shows the server's refusal (which roles a token may have is not in any response the Console reads).
9. Legacy apl_ tokens remain exempt from approvals and multi-factor rules for everything except creating fabric-admin tokens; turning them off is the only way to apply those rules to everyone.

7. Interactive access to modules (terminal, PowerShell)

Contract: the secure-remote-access skill. Built as one slice; evidence MOCK VERIFIED (control plane: 15 API tests, 4 unit tests, with a real websocket against PostgreSQL and the mock Incus; 10 Studio browser tests with a stubbed API and websocket) and CONTRACT VERIFIED (Incus exec protocol: 8 tests over real mutual TLS and websocket upgrades). Not run against a real Incus, Windows guests or a live identity provider.

8. Audit trail

Status: BUILT (2026-10-21), MOCK VERIFIED with a real PostgreSQL (ADR 0011, EVERY-SIZE.md §2.5 and "Auditing"; gap S4 in the Console plan).

One transaction. A change and its audit row commit or roll back together. A handler begins a transaction (authz::begin), makes the change through the persistence functions on it, writes the row with audit_as or audit_scoped(&mut tx, …) and commits (authz::commit). If the row cannot be written, the change is rolled back and the caller gets 500 store_error. audit_as and audit_scoped take a connection (the transaction's), not the pool, so writing the row on the side takes a deliberate audit_alone. Events that change nothing in Apelora's database are written on their own with audit_alone: refusals, failed attempts, exports, and changes carried out in Incus or in the control plane's memory (listed below).

Append-only, completely. Row triggers refuse UPDATE and DELETE; a statement trigger (migration 0022) refuses TRUNCATE.

What a row holds. Who (stable id such as user:<uuid>, kind, and since migration 0022 the name they had, actor_name), what (action), on what (target), outcome (success, denied, failed), the organisation, project and application concerned, the request id, and a detail object without secrets.

Outbox. Every audit row also writes one message (topic audit.recorded, the row as the API shows it) to the outbox table, in the same statement. The worker relays it at least once (see MONITORING.md and services/deployment-worker/src/outbox.rs). Today the relay writes one log line per message; alerts, webhooks and copies for another plane plug in as sinks later.

Sign-ins.

• Password and directory sign-ins write auth.password.succeeded or auth.directory.succeeded in the same transaction as the session they issue; failures are written on their own.
• A sign-in with an identity provider's token has no such moment, so the first request a person makes in each hour writes auth.oidc.signed_in (method, issuer, whether multi-factor, request id; never the token). A conditional update of users.sign_in_audited_at takes the person's row lock, so concurrent first requests write one row. Refused tokens are logged, never audited. If the row cannot be written, the request still goes ahead and the failure is logged at error; every change still needs its own row.

Reading. GET /api/v1/audit and GET /api/v1/identity/audit take the same filters, newest first, each served by an index:

• actor: the stable id, or the name the actor had (ignoring case);
• target: exact;
• action: a prefix, taken literally (% and _ are not patterns);
• outcome: success, denied or failed;
• from and to: RFC 3339, from before to;
• after (the next of the previous page; the answer has no next on the last page) and limit (1 to 500, default 100).

Both need audit.read; below platform level a reader sees only the events of their organisations, projects and applications.

Export. GET /api/v1/audit/export?format=csv|jsonl takes the same filters (not limit), needs audit.read and streams the events newest first, 500 rows per database read, so a large export never sits in memory. CSV has a header row; a cell that starts with a tab or a carriage return, or whose first character after white space is =, +, - or @, is prefixed with ' so spreadsheet programs do not run it. JSON lines hold one event per line as the list returns it. The export is itself audited as audit.exported with its format and filters, before the first byte is sent; an export that cannot be audited does not start.

Every route is checked. services/control-plane/src/api/audit_coverage_tests.rs drives every state-changing route of the OpenAPI document through the real router. For each, it checks the route writes its audit row and, where the change is Apelora's own state, makes the audit table unwritable, sends the request again and checks that it fails and that every other table is unchanged. A route that is neither exercised there nor listed in its EXEMPT list, with the reason, fails the test. Exempt today: dry runs and checks that change nothing (…/plan, …/assignments/preview, …/sign-in/check, …/directories/check) and terminal sessions, which are exercised with their audit rows by sessions_tests.

Exceptions to the one-transaction rule:

• Changes made in Incus, written with audit_alone right after Incus accepted them (if the row cannot be written the request fails, but the Incus change stands): join tokens (create, revoke), evacuating, restoring and removing members, and volume snapshots (take, restore, delete).
• Incus first, then the record and its row in one transaction (the Incus step is not undone): growing and deleting an application's volume; approving, withdrawing, admitting and removing a machine; preparing an environment's Incus project and mirroring quotas as its limits (a refused mirror is written on its own).
• In memory: opening a terminal session (its ticket is discarded when its row cannot be written) and ending one (written before it takes effect).
• Approvals presented with Apelora-Approval are spent before the change, outside its transaction: a change that then fails leaves the approval spent. That errs on the safe side.
• Reads that are audited: an audit export (audit.exported) and a support bundle download (support.bundle.downloaded) are written on their own before the first byte is sent; one that cannot be audited is not sent.
• After the commit: reloading sign-in settings (providers, local accounts, API tokens), and the reachability probes of providers and directories. Reachability is shown in the response and no longer recorded in the audit row.
• Worker: a backup's record and its row share a transaction, as the supervisor's actions already did; instance.moved after a move is written on its own.