Watchdog API

Health monitoring, crash detection, auto-repair, and notifications for agent gateways

Watchdog API

The watchdog monitors agent gateway processes for health and automatically recovers from failures. It detects crash loops, performs auto-repair, and sends notifications via Telegram and Discord.

How It Works

Gateway running
      │
      ▼
Health check every 2 min
      │
      ├── Healthy → continue monitoring
      │
      └── Unhealthy
           │
           ├── First failure → mark degraded
           ├── Retry every 5 sec
           ├── 3 failures → trigger repair
           │
           ▼
      Auto-repair
           │
           ├── Kill gateway process
           ├── Wait 5 seconds
           ├── Restart gateway
           ├── Verify health
           │
           ├── Success → resume normal monitoring
           └── Failure → retry (max 2 attempts)
                              │
                              └── Crash loop detected → notify + stop

Configuration

Variable	Default	Description
WATCHDOG_CHECK_INTERVAL	120	Health check interval in seconds
WATCHDOG_DEGRADED_CHECK_INTERVAL	5	Retry interval when degraded (seconds)
WATCHDOG_MAX_REPAIR_ATTEMPTS	2	Max auto-repair attempts before crash-loop
WATCHDOG_CRASH_LOOP_WINDOW	300	Window for crash-loop detection (seconds)
WATCHDOG_CRASH_LOOP_THRESHOLD	3	Crashes in window to trigger crash-loop
WATCHDOG_AUTO_REPAIR	true	Enable/disable auto-repair
TELEGRAM_BOT_TOKEN	—	Telegram bot token for notifications
TELEGRAM_ADMIN_CHAT_ID	—	Chat ID for Telegram notifications
DISCORD_WEBHOOK_URL	—	Discord webhook URL for notifications

Lifecycle States

State	Description
stopped	Gateway is not running
starting	Gateway process started, waiting for health check
running	Gateway is healthy
degraded	Health check failed, retrying
crash_loop	Too many crashes, auto-repair exhausted
repairing	Auto-repair in progress

Notifications

The watchdog sends notifications for:

• Crash detected — Gateway process exited unexpectedly
• Crash loop — 3+ crashes in 5-minute window
• Auto-repair started — Attempting to restart the gateway
• Auto-repair succeeded — Gateway is healthy again
• Auto-repair failed — Could not recover, needs manual intervention

Telegram Format

🐺 Agentbot Watchdog
🔴 Crash loop detected - View logs
Trigger: `crash_loop`
Attempt count: 3

Discord Format

Embedded message with color coding:

• 🔴 Red: crashes, failures
• 🟡 Yellow: repairing, degraded
• 🟢 Green: recovered, healthy

Test Script

Test the watchdog by injecting an invalid config:

#!/bin/bash
CONTAINER_NAME="agentbot-<userId>"
docker exec "$CONTAINER_NAME" node -e "
  const fs = require('fs');
  const cfg = JSON.parse(fs.readFileSync('/data/.openclaw/openclaw.json'));
  cfg.hooks = cfg.hooks || {};
  cfg.hooks.transformDir = '/tmp/does-not-exist';
  fs.writeFileSync('/data/.openclaw/openclaw.json', JSON.stringify(cfg, null, 2));
"

The watchdog should detect the gateway failure and auto-repair within 2 health check cycles.