The global fabric: from three servers to tens of thousands

Status: target design · Date: 2026-10-21 · Decisions: ADR 0008 — sectors, ADR 0009 — global fabric · Numbers: capacity estimate · Today's limits and connections: connections and scale · Where the product is going: cloud roadmap

Apelora starts on three servers and is meant to grow into a cloud spread over cities, countries and continents. This page explains how the same building blocks scale to that, and what has to be added at each step. One control plane and one sector are what runs. The members report names that site and refuses to plan a second deployment until another site is connected; a second Incus cluster is not managed. Every section ends with what is built today.

1. Levels

How it grows:

2. Control planes

• Regional plane (today's control plane, scoped to a region):

  • desired state, deployments, jobs, workers, health, capacity, terminal sessions and audit for its region;
  • its own PostgreSQL: a primary plus a synchronous standby in another site of the region;
  • workers sharded by sector;
  • it talks to its sectors' Incus APIs over at most ~30 ms.

• Global plane:

  • people, groups, roles, sign-in methods;
  • module classes, application designs and versions;
  • placement and residency policies;
  • the address plan; regions and sites;
  • read-only roll-ups; the audit archive.

  It sees few writes and many reads. Its PostgreSQL has a primary in one region and read replicas in every region.

• Static stability. Each region keeps local copies of what it needs from the global plane, so it keeps operating when the global plane or the links between regions are down:

  • verification keys for identity-provider tokens;
  • role assignments;
  • module classes;
  • the designs it runs;
  • policies.

  Revoking a role takes effect in a region as soon as the copy reaches it (seconds). Identity-provider tokens are short-lived (at most an hour), which bounds the rest.

• One Console. It asks the global plane for shared things and the region's plane for operations, with the same sign-in. Lists across regions come from the global roll-ups and say how fresh they are.

Built today: one control plane, one sector. Servers → Members names that site and sector and says a second deployment is not planned (MOCK VERIFIED). A second Incus cluster is not connected.

3. Networking

What is there and how far it carries

What to augment, and when

The current stack (OVN on Geneve, driven through Incus) does not need replacing to grow. These additions make it carry more, run faster and need fewer hand-made steps. All of them are standards-based and can be driven through Incus or the switches' own APIs.

The address plan is hierarchical, for example:

• IPv6: a /32 per fabric → /40 per region → /48 per site → /56 per sector → /64 per application network.
• IPv4: private space split the same way; small public blocks only for published addresses.

Apelora allocates; Incus and OVN apply.

Built today: OVN networks per application with ACLs, service addresses and published addresses in one sector (REAL INCUS TESTED on the lab).

4. Moving things

One mechanism, used at every distance; what changes is bandwidth, downtime and policy.

How long a copy takes (pre-copy, before any downtime):

The final refresh only moves what changed since the pre-copy (typically 1–5 %), so downtime stays in the seconds-to-minutes range even for large volumes on 10 Gbit/s links.

5. Backups and recovery

• Recovery classes per application, chosen in the Studio and shown in the Console:
  • Gold: recovery point 15 minutes, recovery time 1 hour (tiers 0–2 plus database-level replication in the application).
  • Silver: 4 h / 8 h.
  • Bronze: 24 h / 72 h.
• Restore drills. A scheduled job restores a sample of backups into an isolated project and checks that they start. The result is part of the Health page.
• Losing a region. The global plane holds the designs and the last desired state. The standby region redeploys from them and from tier 2 or 3 copies. People see what is older than the recovery point before anything starts.

6. Other things that change with distance

• Residency. Policies per organisation or project list allowed regions or countries. Placement, moves, backups and the audit archive obey them; a refusal says which policy and which country.

• Identity.

  • Identity-provider tokens are checked in each region with cached keys, so no call to the global plane is needed.
  • Local accounts and their sessions live in the global plane, with regional read copies; revocation reaches regions within seconds.
  • Directories can be per region (a country's own directory).

• Images. Each region keeps a cache of module images; a new image version is pulled once per region, not once per server.

• Time. Every site runs NTP (or PTP for timing-sensitive workloads). Incus and the Raft database need synchronised clocks.

• Upgrades. In waves:

  1. a canary sector;
  2. one site;
  3. one region;
  4. the rest.

  Incus is upgraded per sector (LTS releases; schema changes only between LTS releases). Regional planes are upgraded region by region and must accept one version of skew with the global plane.

• Observability. Metrics and logs stay in the region. The global plane keeps roll-ups and alert states. Every request carries a request id across planes.

• Security. Each sector has its own Incus client certificates; each region has its own intermediate certificate authority. Taking one region's credentials does not reach another.

• Operations. On-call per region, with a follow-the-sun handover. A region can be put in maintenance as a whole.

7. Order of work

1. Sectors (ADR 0008): a sectors table, an adapter per sector, sector on applications.
2. Regions and sites as labels, residency policies, global address management.
3. The per-process fixes (supervisor leader, sign-in reload, shared session tickets), so a plane runs as several processes.
4. Moves between sectors, then between sites (§4).
5. Backup tiers 1–3 and restore drills (§5).
6. A routed underlay with BGP from Incus, anycast for published addresses (§3).
7. Splitting the global plane from the regional plane, with replication and static stability (§2).
8. Global traffic steering.