fix: port api-watchdog mitigation into the real installer compose

The 2026-09-12 CT 101 502 incident fix (api-watchdog: detects the API
container hanging while docker still reports it healthy, and restarts it)
only ever landed in argus-appliance's docker/docker-compose.yml, the
source-build dev compose. It never reached this repo, which is what
update.sh/install.sh on every real Docker-based install actually use --
so the mitigation never shipped to a real install. Ported the service
(built locally from a bundled Dockerfile, since there is no released,
cosign-verified argus-api-watchdog image yet) plus its build context.
This commit is contained in:
root
2026-09-12 13:21:45 +00:00
parent b124e02886
commit 72db2b9718
4 changed files with 167 additions and 0 deletions
+45
View File
@@ -0,0 +1,45 @@
# api-watchdog
Recovers from a real production incident class (2026-09-11/12, CT 101): the `api` container's
process hung/wedged internally while `docker ps` still reported it `Up` -- it never actually
exited, so Docker's own `restart: unless-stopped` policy (which only reacts to a container
exiting) never fired. The container's `HEALTHCHECK` (`wget --spider http://localhost:8080/health`)
correctly kept detecting the hang the entire time -- `docker inspect`'s `.State.Health.Status`
genuinely flipped to `unhealthy` -- but nothing was watching that status and acting on it, so the
outage lasted roughly 11 hours until a human happened to notice and ran `docker restart` by hand.
## What it does
Polls `docker-socket-proxy` (the same already-existing, scoped Docker API proxy the `api` service
itself uses -- see `docker/docker-socket-proxy/README.md`) for `argus-api`'s health status every
`WATCHDOG_INTERVAL_SECONDS` (default 30s). After `WATCHDOG_UNHEALTHY_THRESHOLD` (default 5)
*consecutive* `unhealthy` readings (~2.5 minutes sustained, not one blip), it issues
`POST /containers/argus-api/restart` through the same proxy and resets its counter. A `healthy`
reading at any point resets the counter immediately.
Deliberately talks to `docker-socket-proxy`, never a raw `/var/run/docker.sock` mount -- zero new
grants needed (`CONTAINERS=1`/`ALLOW_RESTARTS=1` are already on for the `api` service's own
diagnostics recovery actions), and this container never gets docker.sock access at all.
## Why not just shorten `retries`/rely on `restart: unless-stopped` alone
`restart: unless-stopped` restarts a container Docker itself observes as exited -- it has no
concept of "unhealthy," and this incident's whole failure mode was a container that never
exited. Docker Engine does not ship a built-in "restart on unhealthy" action; some setups solve
this with the third-party `willfarrell/autoheal` image, but that requires mounting the real
docker.sock into a new container. Reusing the already-present, already-audited
`docker-socket-proxy` instead keeps this fix inside ARGUS's existing security boundary.
## Tuning
- `WATCHDOG_TARGET_CONTAINER` (default `argus-api`)
- `WATCHDOG_INTERVAL_SECONDS` (default `30`, matches the api healthcheck's own interval)
- `WATCHDOG_UNHEALTHY_THRESHOLD` (default `5`) -- consecutive unhealthy checks before restarting
## Verification
Sandbox-tested (`docker-compose.local-sandbox.yml`) by temporarily forcing the `api` service's
healthcheck to always fail (`test: ["CMD", "false"]`), confirming: the watchdog logs 5 consecutive
`unhealthy` checks, then a real `POST .../restart` through the proxy, and `argus-api`'s container
start time visibly advances. Reverted the forced-failure healthcheck override afterward -- this
watchdog is not itself part of that sandbox scenario's cleanup.