kb://learnings/2026-05-21-prometheus-network-bridgestable2026-05-21

Prometheus needs multi-network reach for file_sd targets

learningprometheusdocker-networksobservabilityfile-sd

Wave 🆃 wired manifest endpoints into Prometheus file_sd_configs. The JSON targets landed, Prometheus read them, but 6 services stayed dim on the dashboard. Root cause: Docker network reach.

Symptom

/metrics fanout audit at end of 2026-05-21 showed:

  • Reachable: atlas-api, atlas-ingestor, openalice (alice main)
  • Dim: chat-bridge, scheduler, radio, voice-agent, persona, anim

The dim services' /metrics endpoints returned 200 when curled from inside their own container. Prometheus saw the file_sd targets land in its config. But scrape requests timed out at the DNS layer.

Root cause

Prometheus runs in the monitoring Docker network. It was attached to two more (openalice, atlas) for legacy reasons. The 6 dim services live in other project networks (openalice-persona, openalice-chat-bridge, etc.). From monitoring + openalice + atlas, Prometheus could not resolve hostnames in the other networks — DNS lookups returned NXDOMAIN.

The file_sd contract has no built-in awareness of Docker network topology. Targets land regardless of reach.

Two clean architectures

OptionCostProsCons
1. Attach prom to every service networklowworks todaygrows linearly with services
2. Single metrics-scrape net + every service joinslow-midscales, isolates trafficneeds every compose to opt in

The right move is option 2 — pays off the second the lab adds a seventh service. Decision pending NAO call (open question 🅚 in docs/open-questions-2026-05-21.md).

Operational lesson

Auto-discovery declarations (file_sd, DNS-SD, consul) describe what to scrape; they do not describe how to reach it. Reach is a separate concern that must be designed into the deploy topology, not inferred from the discovery file.

When adding a new manifest-driven observability mechanism, the design needs three layers explicitly:

  1. Declaration — manifest exposes block
  2. Discovery — ingestor writes file_sd target
  3. Reach — Docker network bridge or equivalent

Missing layer 3 produces silent partial success: targets land but never scrape. Worse than an obvious failure.

Mitigation until decision lands

A temporary attach (option 1) for the 6 specific services unblocks the SLO board now without committing to long-term topology. The metrics-scrape net rollout becomes a deliberate sequenced change rather than a rushed fix.