Atlas
kb://architecture/slo-auto-targets-pipelinestable2026-05-21

SLO auto-targets — file_sd flow from manifest to live scrape

architectureobservabilityprometheusfile-sdatlasslo

Wave 🆃 (2026-05-21) wired manifest endpoints into Prometheus via file_sd_configs. The pipeline is fully declarative end-to-end.

The flow

.atlas-deps.yml (per repo)
        │
        │ exposes:
        │   - kind: http
        │     url:  https://service.example/metrics
        │     slo:  ...
        ▼
ingestor/src/manifest.rs
        │
        │ writes JSON to:
        ▼
/srv/atlas/prom-targets/*.json   (volume on blal.de)
        │
        │ mounted into prometheus container
        ▼
prometheus.yml
  scrape_configs:
    - job_name: atlas-auto
      file_sd_configs:
        - files: ['/etc/prometheus/atlas-targets/*.json']
        ▼
Prometheus reloads file_sd every 30s
        │
        ▼
Live scrape → Grafana dashboards

Why this matters

Adding a new service to the SLO board is one PR: declare an exposes block with a /metrics URL in .atlas-deps.yml. The next ingestor scan writes a new JSON target file. Prometheus picks it up automatically. Grafana surfaces it via existing dashboards keyed on service name.

No prometheus.yml edits. No manual target lists drifting from reality.

Network reach prerequisite

Prometheus must be on the same Docker network as the target services it scrapes, or the target hostnames will not resolve. See kb://learnings/2026-05-21-prometheus-network-bridge for the multi-network bridge fix.

Per-target SLO

The manifest exposes[].slo field carries per-target latency / error budgets:

exposes:
  - kind: http
    name: persona-http
    url: https://persona.blal.pro/healthz
    slo:
      availability: 99.5
      latency_p99_ms: 800

Grafana dashboards read these via atlas's REST API (/api/v1/repos/<name> returns the parsed exposes array) and render per-service SLO panels without per-service config.

State of the pipeline (2026-05-21)

  • 19 probable endpoints discovered from manifests
  • Ingestor publishes to /srv/atlas/prom-targets/
  • 6 services dim due to network reach gap (see linked learning)
  • Open question 🅚 in docs/open-questions-2026-05-21.md chooses between option 1 (prom joins each network) and option 2 (single metrics-scrape net that every service opts into)