fix: port api-watchdog mitigation into the real installer compose
The 2026-09-12 CT 101 502 incident fix (api-watchdog: detects the API container hanging while docker still reports it healthy, and restarts it) only ever landed in argus-appliance's docker/docker-compose.yml, the source-build dev compose. It never reached this repo, which is what update.sh/install.sh on every real Docker-based install actually use -- so the mitigation never shipped to a real install. Ported the service (built locally from a bundled Dockerfile, since there is no released, cosign-verified argus-api-watchdog image yet) plus its build context.
This commit is contained in:
root
@@ -183,6 +183,46 @@ services:
|
||||
retries: 3
|
||||
start_period: 360s
|
||||
|
||||
# api-watchdog (2026-09-12, see docker/watchdog/README.md in the
|
||||
# argus-appliance repo): closes a real production gap -- `restart:
|
||||
# unless-stopped` above only ever reacts to the api container actually
|
||||
# EXITING. A real incident (CT 101, 2026-09-11/12) had it hang/wedge
|
||||
# internally for ~11 hours while `docker ps` still reported it `Up`; the
|
||||
# HEALTHCHECK above correctly kept detecting the hang the entire time,
|
||||
# but nothing was watching that status and acting on it. This polls the
|
||||
# same already-scoped docker-socket-proxy (zero new grants) and restarts
|
||||
# argus-api after several consecutive unhealthy checks.
|
||||
#
|
||||
# Built locally rather than pulled/pinned/cosign-verified like api/ui/
|
||||
# nginx above: there is not yet a released, signed argus-api-watchdog
|
||||
# image in the registry (tracked as a follow-up to extend ADR-0051's
|
||||
# verification set to a 4th component). The build context ships in this
|
||||
# repo so it stays fully reproducible from source, not a hand-built
|
||||
# local image nobody can regenerate.
|
||||
api-watchdog:
|
||||
build:
|
||||
context: ./watchdog
|
||||
container_name: argus-api-watchdog
|
||||
restart: unless-stopped
|
||||
security_opt:
|
||||
- "no-new-privileges:true"
|
||||
cap_drop:
|
||||
- ALL
|
||||
logging:
|
||||
driver: "json-file"
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
environment:
|
||||
WATCHDOG_TARGET_CONTAINER: argus-api
|
||||
WATCHDOG_INTERVAL_SECONDS: "30"
|
||||
WATCHDOG_UNHEALTHY_THRESHOLD: "5"
|
||||
depends_on:
|
||||
docker-socket-proxy:
|
||||
condition: service_started
|
||||
api:
|
||||
condition: service_started
|
||||
|
||||
# React UI (Admin Panel), served over HTTPS
|
||||
ui:
|
||||
# See the api service's own comment above -- same ADR-0051 pinning.
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
FROM alpine:3.20
|
||||
|
||||
RUN apk add --no-cache curl jq
|
||||
|
||||
COPY entrypoint.sh /entrypoint.sh
|
||||
RUN chmod +x /entrypoint.sh
|
||||
|
||||
USER nobody
|
||||
ENTRYPOINT ["/entrypoint.sh"]
|
||||
@@ -0,0 +1,45 @@
|
||||
# api-watchdog
|
||||
|
||||
Recovers from a real production incident class (2026-09-11/12, CT 101): the `api` container's
|
||||
process hung/wedged internally while `docker ps` still reported it `Up` -- it never actually
|
||||
exited, so Docker's own `restart: unless-stopped` policy (which only reacts to a container
|
||||
exiting) never fired. The container's `HEALTHCHECK` (`wget --spider http://localhost:8080/health`)
|
||||
correctly kept detecting the hang the entire time -- `docker inspect`'s `.State.Health.Status`
|
||||
genuinely flipped to `unhealthy` -- but nothing was watching that status and acting on it, so the
|
||||
outage lasted roughly 11 hours until a human happened to notice and ran `docker restart` by hand.
|
||||
|
||||
## What it does
|
||||
|
||||
Polls `docker-socket-proxy` (the same already-existing, scoped Docker API proxy the `api` service
|
||||
itself uses -- see `docker/docker-socket-proxy/README.md`) for `argus-api`'s health status every
|
||||
`WATCHDOG_INTERVAL_SECONDS` (default 30s). After `WATCHDOG_UNHEALTHY_THRESHOLD` (default 5)
|
||||
*consecutive* `unhealthy` readings (~2.5 minutes sustained, not one blip), it issues
|
||||
`POST /containers/argus-api/restart` through the same proxy and resets its counter. A `healthy`
|
||||
reading at any point resets the counter immediately.
|
||||
|
||||
Deliberately talks to `docker-socket-proxy`, never a raw `/var/run/docker.sock` mount -- zero new
|
||||
grants needed (`CONTAINERS=1`/`ALLOW_RESTARTS=1` are already on for the `api` service's own
|
||||
diagnostics recovery actions), and this container never gets docker.sock access at all.
|
||||
|
||||
## Why not just shorten `retries`/rely on `restart: unless-stopped` alone
|
||||
|
||||
`restart: unless-stopped` restarts a container Docker itself observes as exited -- it has no
|
||||
concept of "unhealthy," and this incident's whole failure mode was a container that never
|
||||
exited. Docker Engine does not ship a built-in "restart on unhealthy" action; some setups solve
|
||||
this with the third-party `willfarrell/autoheal` image, but that requires mounting the real
|
||||
docker.sock into a new container. Reusing the already-present, already-audited
|
||||
`docker-socket-proxy` instead keeps this fix inside ARGUS's existing security boundary.
|
||||
|
||||
## Tuning
|
||||
|
||||
- `WATCHDOG_TARGET_CONTAINER` (default `argus-api`)
|
||||
- `WATCHDOG_INTERVAL_SECONDS` (default `30`, matches the api healthcheck's own interval)
|
||||
- `WATCHDOG_UNHEALTHY_THRESHOLD` (default `5`) -- consecutive unhealthy checks before restarting
|
||||
|
||||
## Verification
|
||||
|
||||
Sandbox-tested (`docker-compose.local-sandbox.yml`) by temporarily forcing the `api` service's
|
||||
healthcheck to always fail (`test: ["CMD", "false"]`), confirming: the watchdog logs 5 consecutive
|
||||
`unhealthy` checks, then a real `POST .../restart` through the proxy, and `argus-api`'s container
|
||||
start time visibly advances. Reverted the forced-failure healthcheck override afterward -- this
|
||||
watchdog is not itself part of that sandbox scenario's cleanup.
|
||||
@@ -0,0 +1,73 @@
|
||||
#!/bin/sh
|
||||
# api-watchdog: closes a real gap found during a 2026-09-11/12 production
|
||||
# incident (see docker/watchdog/README.md) -- Docker's own `restart:
|
||||
# unless-stopped` policy only ever acts on a container actually EXITING; it
|
||||
# does nothing for a container that hangs/wedges while `docker ps` still
|
||||
# reports it "Up". The api service's HEALTHCHECK (wget against /health)
|
||||
# correctly kept detecting the hang for the incident's entire ~11-hour
|
||||
# duration -- `docker inspect`'s `.State.Health.Status` genuinely flips to
|
||||
# "unhealthy" -- but nothing was ever watching that status and acting on it,
|
||||
# so the outage went unrecovered until a human happened to notice and ran
|
||||
# `docker restart` by hand.
|
||||
#
|
||||
# Talks to the EXISTING docker-socket-proxy (already scoped with
|
||||
# CONTAINERS=1 + ALLOW_RESTARTS=1 for the api service's own diagnostics
|
||||
# recovery actions -- see docker/docker-socket-proxy/README.md) rather than
|
||||
# mounting the raw host docker.sock into a new container: zero new grants
|
||||
# needed, no new attack surface, same security boundary the api service
|
||||
# itself already relies on.
|
||||
set -eu
|
||||
|
||||
DOCKER_API="${WATCHDOG_DOCKER_API:-http://docker-socket-proxy:2375}"
|
||||
TARGET="${WATCHDOG_TARGET_CONTAINER:-argus-api}"
|
||||
INTERVAL="${WATCHDOG_INTERVAL_SECONDS:-30}"
|
||||
THRESHOLD="${WATCHDOG_UNHEALTHY_THRESHOLD:-5}"
|
||||
|
||||
log() {
|
||||
printf '[watchdog] %s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*"
|
||||
}
|
||||
|
||||
log "starting -- target=$TARGET interval=${INTERVAL}s threshold=$THRESHOLD checks (~$((INTERVAL * THRESHOLD))s sustained unhealthy before acting)"
|
||||
|
||||
consecutive=0
|
||||
while true; do
|
||||
sleep "$INTERVAL"
|
||||
|
||||
body="$(curl -sf --max-time 10 "$DOCKER_API/containers/$TARGET/json" 2>/dev/null || true)"
|
||||
if [ -z "$body" ]; then
|
||||
log "WARNING: could not reach docker-socket-proxy or inspect $TARGET -- leaving consecutive count unchanged"
|
||||
continue
|
||||
fi
|
||||
status="$(printf '%s' "$body" | jq -r '.State.Health.Status // "none"' 2>/dev/null || echo "parse_error")"
|
||||
|
||||
case "$status" in
|
||||
healthy)
|
||||
if [ "$consecutive" -gt 0 ]; then
|
||||
log "recovered on its own after $consecutive unhealthy check(s) -- resetting counter"
|
||||
fi
|
||||
consecutive=0
|
||||
;;
|
||||
unhealthy)
|
||||
consecutive=$((consecutive + 1))
|
||||
log "unhealthy check $consecutive/$THRESHOLD"
|
||||
if [ "$consecutive" -ge "$THRESHOLD" ]; then
|
||||
log "sustained unhealthy for $consecutive consecutive checks -- restarting $TARGET"
|
||||
if curl -sf --max-time 30 -X POST "$DOCKER_API/containers/$TARGET/restart?t=30" >/dev/null 2>&1; then
|
||||
log "restart request accepted"
|
||||
else
|
||||
log "WARNING: restart request failed -- will keep retrying every $INTERVAL"
|
||||
fi
|
||||
consecutive=0
|
||||
fi
|
||||
;;
|
||||
starting | none)
|
||||
# "starting" = still inside start_period (normal on boot/redeploy);
|
||||
# "none" = the target has no HEALTHCHECK at all -- either way, not
|
||||
# this watchdog's concern.
|
||||
consecutive=0
|
||||
;;
|
||||
*)
|
||||
log "unexpected/unparseable health status ($status) -- leaving consecutive count unchanged"
|
||||
;;
|
||||
esac
|
||||
done
|
||||
Reference in New Issue
Block a user