Runbooks
Step-by-step procedures for operating an Apelora fabric: find your situation in the table and follow its page. When part of a procedure has not yet been carried out in practice, the page says so in a short list at its end.
Apelora never writes Incus's database, and it moves instances only by asking Incus (the instance.move
job). Membership, placement, evacuation and migration are Incus operations that Apelora requests,
previews and audits.
| Situation | Runbook |
|---|---|
| Add a server to the cluster | member-join.md |
| Maintenance on a server | member-maintenance.md |
| A server fails | member-failure.md |
| Remove a server for good | member-removal.md |
| The cluster loses quorum | quorum-loss.md |
| Read quorum; upgrade Incus one member at a time | incus-rolling-upgrade.md |
| OVN or the uplink fails | network-incident.md |
| A replica fails its health check | replica-unhealthy.md |
| A volume is kept, missing or to be deleted | volumes.md |
| A pool is filling up or down, or backups stop | storage-incident.md |
| The deploy identity cannot see its project | deploy-identity.md |
| Raise a quota; an environment's Incus project is not prepared | tenancy-quotas.md |
| Revoke access (people, automation, servers) | access-revocation.md |
| A credential is stolen or an account or session is misused | security-incident.md |
| The control plane or worker is down | api-outage.md |
| Find out why something failed (logs, request ids, audit) | troubleshooting.md |
| Read the Health page (components, servers, what each state means) | health-page.md |
| Set up and respond to alerts (webhooks, acknowledge, silence, each rule) | alerts.md |
| Rotate Apelora's Incus certificates | certificates.md |
| Back up Apelora's own database, drill a restore, restore the plane onto a new host | apelora-database-backup.md |
| Update Apelora without internet access | offline-update.md |
This page is a copy of docs/runbooks/README.md in the repository, which is the source of truth. Suggest a change.