Monitoring: metrics for every layer, collected inside the installation

Status: built for Apelora's own processes (CONTRACT/MOCK VERIFIED); Incus /1.0/metrics REAL INCUS TESTED 2026-10-23; example collector in infra/observe · Date: 2026-10-23 · Builds on: connections and scale, one design at every size (ADR 0011), disconnected installations (ADR 0010)

Operators watch an Apelora installation with the metrics collector they already run inside it. Apelora exposes its own metrics in the standard text exposition format. Every layer below it (Incus, instances, PostgreSQL, OVN, shared storage) is collected directly from that layer, never through Apelora. That way Apelora is never a bottleneck or a single point of failure for metrics, and nothing leaves the site.

1. Principles

1. Each layer reports for itself. Apelora measures what only Apelora knows: requests, decisions, jobs, its calls to Incus and its database pool. Incus measures servers and instances. PostgreSQL measures itself.
2. Cheap at every size. Instrumentation sits at choke points only: the request middleware, the job runner, the Incus HTTP client, the supervisor loop and the scrape-time readers. State read from PostgreSQL or Incus for metrics is refreshed when a collector scrapes, at most every 15 s (database) and 30 s (inventory) per process, never per API request.
3. Low cardinality, no personal data. Label values come from small fixed sets: route templates (/api/v1/applications/{name}), endpoint templates (/1.0/instances/{id}/state), status classes (2xx), outcome classes (ok, unreachable), catalogue permission ids and job kinds. A label never carries a raw path, a name, an id, an email address, a token or an error text. Each family keeps at most 1,000 label sets; anything beyond that is counted under one series whose labels are _overflow, so a mistake shows up instead of using unbounded memory.
4. Summaries roll up (ADR 0011 §2.6). Every series carries the constant labels region, site and sector (APELORA_REGION, APELORA_SITE, APELORA_SECTOR; default default). A regional collector aggregates by them without relabelling, and the global level only ever sees the aggregates.
5. Honest about the source. apelora_provider_info{mode="mock"} marks a process that talks to the mock, so mock numbers are never mistaken for a cluster.

2. Where to scrape Apelora

Default and why. The safer default is the authenticated endpoint. It uses the API's TLS and the route gate, and refusals are audited like any other 403. Unauthenticated listeners exist only for collectors that cannot hold a token; they bind to loopback by default, and a non-loopback address is logged as a warning at start-up. Put them on a management network only.

The collector's identity. Create an automation service identity and give it the built-in role Metrics Reader (metrics-reader) at platform scope. That role holds only fabric.metrics.read: the collector cannot read inventory, change anything or see people. Fabric Administrator, Viewer and Auditor (at platform scope) also hold the permission. A legacy apl_ token can scrape when its role reads inventory (every legacy role does), because metrics reveal less than cluster.view. The Console's Administration → Monitoring panel shows the endpoints, whether the signed-in caller may scrape, and the labels in use (GET /api/v1/monitoring, which needs fabric.metrics.read or cluster.view).

Logs. Both services write RUST_LOG lines (default info). APELORA_LOG_FORMAT=json writes one JSON object per line, with the current request or job span's fields (request_id, and on the worker job, kind, target). The example collector tails those files into Loki.

Traces. Off by default. APELORA_OTLP_ENDPOINT (an http or https URL of a Tempo or collector inside the site) posts completed request and job spans as OTLP/HTTP JSON. The same request_id hashes to the same trace in both processes, so a job the API queued shows with its request. Nothing is sent unless this is set. Keep it on loopback.

Profiles. Apelora does not push CPU profiles. The example collector runs Pyroscope so an operator can add a profiler later (eBPF on Linux, or an SDK). Until then the Pyroscope UI is empty.

Format. By default the answer is text/plain; version=0.0.4. A collector that sends Accept: application/openmetrics-text gets the OpenMetrics variant (counter families named without _total, ending with # EOF). Answers carry Cache-Control: no-store.

Why our own registry. The crate apelora-metrics (a few hundred lines with tests, no dependencies) holds counters, gauges, histograms and the exposition writer. Three things led to it:

• Three crates (the Incus adapter, the identity crate and the persistence crate) must record into one registry, and Apelora needs neither push, exemplars nor dynamic registration.
• A hand-written registry adds no third-party code to review, licence or mirror for air-gapped sites.
• The cardinality cap is part of the registry itself.

The worker's listener uses the HTTP stack the control plane already uses (no new crate in the tree). If we later need features we do not have, a permissive, well-maintained crate can replace the registry behind the same family declarations.

3. Apelora's metric families

All names start with apelora_. Counters end in _total, durations are in seconds, and timestamps are Unix seconds. The declarations, with their help text, are in crates/metrics/src/lib.rs.

Health and capacity views (per-member load, pool fill) belong to the Health work and to Incus's own metrics (§4). These families do not copy them.

4. Layers Apelora does not own: collect them directly

Incus servers and instances. Each Incus server exposes GET /1.0/metrics in the same text format. It reports CPU, memory, disk, file system and network per instance, plus internal figures: member status, roles, running operations, active warnings, storage pool size and use, and daemon uptime. In a cluster each server reports only the instances it runs, so scrape every server. Set-up, from the Incus documentation:

1. Expose only the metrics endpoint on a management address: incus config set core.metrics_address=<address>:8444 on each server. core.https_address would expose the whole API instead.
2. Create a key and a self-signed client certificate for the collector (metrics.key, metrics.crt) with your usual X.509 tooling; an elliptic-curve key is fine. Keep the key on the collector's host only.
3. Trust it for metrics only: incus config trust add-certificate metrics.crt --type=metrics. A metrics certificate cannot manage instances or anything else, so a leaked collector credential reads numbers and nothing more.
4. Point the collector at https://<server>:8444/1.0/metrics on every server. The metrics listener presents the cluster certificate (not each member's server.crt). Use that certificate as the CA, or pin its public key: OpenSSL 3's curl --cacert rejects the self-signed leaf because it is not marked CA:TRUE. A project parameter limits the answer to one project; without it, all projects are reported.

Never turn off metrics authentication (core.metrics_authentication=false). Never give the collector Apelora's deploy or admin certificates.

REAL INCUS TESTED 2026-10-23 on the four-member lab (scripts/test-cluster/metrics.sh): a type=metrics certificate, core.metrics_address=127.0.0.1:8444 on each member (lab server certificates name only loopback), 38 families from every member, /1.0/instances 404 on the metrics listener and 403 on the API. Authentication stayed on.

PostgreSQL. Read its statistics views with a dedicated read-only role:

• connections by state (pg_stat_activity);
• transactions, rollbacks, conflicts and deadlocks (pg_stat_database);
• replication lag per standby (pg_stat_replication) and on the standby (pg_last_xact_replay_timestamp());
• WAL archiving success and failures (pg_stat_archiver);
• database and table size;
• long-running transactions.

Apelora's own view of the database is apelora_db_* and apelora_store_errors_total.

OVN. On the members that run the central databases, check cluster health with each database's cluster status (ovn-appctl -t <database control socket> cluster/status <database>): role, leader, term, and connected servers. On every member, check that ovn-controller is running and connected to the southbound database, and read the datapath's flow and drop counters. Apelora only sees OVN through Incus: network.sync job failures show up as apelora_worker_job_attempts_total{kind="network.sync",outcome!="succeeded"}.

SAN and shared storage. On every member, read: iSCSI or Fibre Channel session state, multipath path counts, the lock manager's lockspace state (sanlock) and the watchdog. Read pool use from Incus (incus_storage_pool_used_bytes). Use the same node agent your collector already uses for host metrics. Apelora reads the lockspace itself only to fence a failed member (the fence probe, ADR 0013) and exports apelora_member_failures, apelora_fence_probes and apelora_recoveries; alert member-unfenced.

Image mirrors. Probe each mirror's HTTPS endpoint from inside the site, and watch its disk. Failed pulls appear in Apelora as failed instance.create attempts.

5. Which signals cover each layer

Layers as in connections and scale §1. The thresholds are starting points; tune them to the installation.

Built-in alerts and the collector's alerts

Apelora also raises a small set of built-in alerts itself (design, runbook): the worker holding the alert-evaluator lease checks sixteen rules every 30 s against what Apelora already knows (jobs, worker heartbeats, leader leases, supervisor actions, replica health, application status, Incus reachability and members, certificate expiry, volume schedules, the outbox, the restore drill's record), shows them in the Console and sends them to webhooks through the outbox.

• They do not replace the rows above. They cannot see what only the lower layers measure (instance memory, OVN, SAN paths, PostgreSQL replication, the API's error ratio), and they stop when no worker runs. Alert on those in your collector.
• Overlap is intended. Where a row above and a built-in rule watch the same thing (oldest runnable job, expired leases, member status, certificate expiry, backup age, outbox age), the built-in rule exists so that an installation without a collector still hears about it; with a collector you get both, which is harmless. Turn a built-in rule off (Console → Alerts → Rules) if you prefer to route that condition through your collector only.
• Watch the watcher from outside. apelora_alert_evaluation_duration_seconds stops growing and apelora_alerts_firing disappears when no worker evaluates: alert on "no worker reports apelora_alerts_firing for 5 min" in the collector. The restore-drill rule defaults to 35 days (the runbook's monthly drill); the row above asks for 8 days for weekly drills: set the rule's threshold to your drill schedule.

6. At every size, and without internet access

• Small installation: one collector inside the site scrapes the control plane, the worker, every Incus server, PostgreSQL and the members' node agents. At 15–30 s intervals the extra load is negligible.
• Several sectors, sites or regions: each site's collector scrapes only its own site. The region, site and sector labels make roll-ups a plain aggregation. The regional level keeps aggregates; per-instance series stay in the site (ADR 0011 §2.6, residency).
• Disconnected sites (ADR 0010): the collector, its storage and its alerting run inside the site. Apelora sends nothing anywhere: there is no push and no telemetry, and metrics are only answered when the collector asks.

7. Adding a metric

See the observability skill (.agents/skills/observability/SKILL.md, "Metrics in Apelora today"): declare the family in crates/metrics/src/lib.rs and add it to FAMILIES. Record it only at a choke point. Use labels from small fixed sets, and extend the allowed label list in the crate's test only for a new small set. Add a test that the family appears.

8. Example collector (infra/observe)

The operator's collector stays a separate program. Apelora never vendors it. infra/observe is an example, like infra/serve and infra/systemd: Prometheus (metrics), Loki (logs), Tempo (traces), Pyroscope (profiles, empty until a profiler is added), Grafana (the UI), and Alloy (tails JSON logs). Licences: Prometheus and Alloy Apache-2.0; Grafana, Loki, Tempo and Pyroscope AGPL-3.0, run as their own processes, never linked.

bash scripts/test-cluster/observe.sh          # plan
APPLY=1 bash scripts/test-cluster/observe.sh  # start on loopback

Then start the control plane and worker with APELORA_LOG_FORMAT=json, APELORA_METRICS_ADDR=127.0.0.1:9470 and APELORA_OTLP_ENDPOINT=http://127.0.0.1:4318, and tee JSON logs into $APELORA_OBSERVE_LOG_DIR (default ~/.config/apelora/test-cluster/logs). Grafana is http://127.0.0.1:3000 (development login in the compose file). Incus /1.0/metrics is still scraped on each member (metrics.sh), not through this compose file.

Studio and the Fabric Console log failures as JSON lines in the browser console (secret field names redacted). They do not send those lines anywhere.

9. Evidence and open points

• Evidence:

  • unit tests for the registry (escaping, histograms, the overflow cap, label rules);
  • contract tests for the Incus client (endpoint templates, outcomes, reachability);
  • API tests against a real PostgreSQL with the mock provider: who may scrape; the expected families after a request and a job through the real worker loop; no token, name, email address, id or key text in the output;
  • a worker test for the listener;
  • a Console Playwright test with a stubbed API;
  • OTLP JSON for request and job spans (crates/telemetry; same request_id shares a trace id);
  • scripts/ci/verify-observe-examples.sh (compose file and the Grafana dashboard parse).

  Labels: CONTRACT VERIFIED for the Incus call metrics, MOCK VERIFIED for the rest of Apelora's /metrics in unit tests. Collecting Incus's own /1.0/metrics is REAL INCUS TESTED 2026-10-23 (metrics.sh). Apelora's own /metrics scraped into the example collector while talking to the four-member lab is REAL INCUS TESTED 2026-10-23 (observe.sh: Prometheus targets up, apelora_incus_up=1 on control plane and worker, four members Online, Loki labels from JSON logs, Tempo held request traces, Grafana dashboard apelora-overview). Pyroscope is running and empty until a profiler is added.

• Pool acquire time is sampled, once per scrape. The database library offers no hook for every acquire, and a pool wrapper would touch every query. The sample shows a starved pool, but not a short burst.

• Several API processes expose the same queue and inventory gauges. Aggregate them with max, not sum.

• Rate limiting: any 429 is counted per route. A dedicated limiter can add its own reason label when it lands.

• Request id ↔ metrics: metrics carry no request id (that would be unbounded). Use the logs for single requests (EVERY-SIZE §2.8).