The control plane or the worker is down

Use this page when the API does not answer, when deployments stay queued, or when PostgreSQL is down.

  • Running instances are unaffected. Incus runs them; Apelora is not in the data path. Service addresses keep working (they are OVN load balancers).
  • Control plane down: the Studio and the Fabric Console say the API is not reachable and show no invented data. Nothing new can be deployed. Restart it; it needs PostgreSQL (DATABASE_URL) and its Incus settings.
  • Worker down: deployments are accepted and queued but not carried out. The replica supervisor does not run, so lost replicas are not restored. Restart the worker; it claims queued jobs. A job whose 30 s lease expired is taken over, fenced by its generation, and waits for any Incus operation an earlier attempt started, so nothing is created twice. A deployment request sent twice queues its jobs once.
  • PostgreSQL down: both services fail requests that need the database; Incus keeps running everything. Restore the database from backup if needed: job and revision history live there (apelora-database-backup.md).

Before you rely on it

  • A worker stopped in the middle of a job has been tried; a longer outage under load has not.

This page is a copy of docs/runbooks/api-outage.md in the repository, which is the source of truth. Suggest a change.