When something goes wrong
Start with Health in the Console, then open Operations for failed jobs and Audit for what changed. Running instances are not affected by Apelora being down: Incus runs them and Apelora is not in the data path.
Which runbook
| Symptom | Runbook |
|---|---|
| A server is down | A server fails |
| The cluster cannot reach a majority | The cluster loses quorum |
| A replica fails its health check | A replica fails its health check |
| The Console or Studio cannot reach the API | The control plane or the worker is down |
| A pool is filling up, or backups stopped | A storage problem |
| Applications cannot talk to each other | Network incidents |
| A deploy fails, or a job keeps failing | Troubleshooting with logs and the audit trail |
| A certificate is about to expire | Rotate Apelora's Incus certificates |
| A stolen token or misused account | A security incident |
| You need to service a server | Maintenance on a server |
When a server fails
Incus marks it offline after about 20 seconds. Stateless replicas are replaced on other servers after a delay. A module with data on shared storage restarts elsewhere only once the failed server is fenced: proven stopped, so two servers never write the same disk. This takes a few minutes: Incus first has to notice the server is gone, and the server has to be fenced. A server fails has the timeline and says what is automatic and what you do.
Looking after Apelora itself
Back up and restore its database: Back up and restore Apelora's own database. Upgrade Incus without downtime: a rolling Incus upgrade. Update without internet access: offline update. All runbooks: Runbooks.