Wave 🆃 wired manifest endpoints into Prometheus file_sd_configs. The JSON targets landed, Prometheus read them, but 6 services stayed dim on the dashboard. Root cause: Docker network reach.
Symptom
/metrics fanout audit at end of 2026-05-21 showed:
- Reachable: atlas-api, atlas-ingestor, openalice (alice main)
- Dim: chat-bridge, scheduler, radio, voice-agent, persona, anim
The dim services' /metrics endpoints returned 200 when curled from inside their own container. Prometheus saw the file_sd targets land in its config. But scrape requests timed out at the DNS layer.
Root cause
Prometheus runs in the monitoring Docker network. It was attached to two more (openalice, atlas) for legacy reasons. The 6 dim services live in other project networks (openalice-persona, openalice-chat-bridge, etc.). From monitoring + openalice + atlas, Prometheus could not resolve hostnames in the other networks — DNS lookups returned NXDOMAIN.
The file_sd contract has no built-in awareness of Docker network topology. Targets land regardless of reach.
Two clean architectures
| Option | Cost | Pros | Cons |
|---|---|---|---|
| 1. Attach prom to every service network | low | works today | grows linearly with services |
2. Single metrics-scrape net + every service joins | low-mid | scales, isolates traffic | needs every compose to opt in |
The right move is option 2 — pays off the second the lab adds a seventh service. Decision pending NAO call (open question 🅚 in docs/open-questions-2026-05-21.md).
Operational lesson
Auto-discovery declarations (file_sd, DNS-SD, consul) describe what to scrape; they do not describe how to reach it. Reach is a separate concern that must be designed into the deploy topology, not inferred from the discovery file.
When adding a new manifest-driven observability mechanism, the design needs three layers explicitly:
- Declaration — manifest
exposesblock - Discovery — ingestor writes file_sd target
- Reach — Docker network bridge or equivalent
Missing layer 3 produces silent partial success: targets land but never scrape. Worse than an obvious failure.
Mitigation until decision lands
A temporary attach (option 1) for the 6 specific services unblocks the SLO board now without committing to long-term topology. The metrics-scrape net rollout becomes a deliberate sequenced change rather than a rushed fix.