Operate

Day-to-day tasks and what to do when something fails, step by step.

• Runbooks: start hereThe runbooks, and how far each procedure has been verified.
• Add a server to the clusterRegister, approve, enrol and admit a new server.
• Maintenance on a serverEvacuate a server for maintenance, then restore it.
• A server failsWhat happens on its own when a server fails, what to do, and the measured timings.
• Remove a server from the cluster for goodRemove a server that is leaving for good.
• The cluster loses quorumThe cluster cannot reach a majority: what it means and how to recover.
• Reading quorum, and a rolling Incus upgradeReading quorum, and upgrading Incus one server at a time.
• A replica fails its health checkA replica fails its health check: what Apelora does and what you can do.
• An application's volumes: kept, missing or to be deletedAn application's volumes: kept, missing, or to be deleted.
• A storage problem: a pool filling up, a pool that is down, or backups that stopA pool filling up, a pool that is down, or backups that stopped.
• Network incidents (OVN)Application networks (OVN) misbehaving: what to check and what is known.
• The control plane or the worker is downWhat still works when the control plane or the worker is down, and how to bring them back.
• Reading the Health pageHow to read the Health page, and what each status means.
• AlertsWhat each alert means and what to do about it.
• Troubleshooting with logs and the audit trailUsing logs and the audit trail, and retrying or cancelling jobs.
• Rotate Apelora's Incus certificatesRotate the certificates Apelora uses to reach Incus.
• The deploy identity cannot see its projectThe deploy identity cannot see its project: symptoms and fixes.
• Revoke accessRevoke someone's access, then check what they did since.
• A security incident: a stolen credential, a misused account or a session you did not expectA stolen credential, a misused account or a session you do not trust: contain, investigate, recover.
• Tenancy: environments' Incus projects and quotasEnvironments' Incus projects and quotas: checking and fixing them.
• Back up and restore Apelora's own databaseBack up and restore Apelora's own database, with a restore drill.
• Update Apelora without internet accessUpdate Apelora and its images without internet access.