> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agentbot.raveculture.xyz/llms.txt
> Use this file to discover all available pages before exploring further.

# Watchdog API

> Health monitoring, crash detection, auto-repair, and notifications for agent gateways

# Watchdog API

The watchdog monitors agent gateway processes for health and automatically recovers from failures. It detects crash loops, performs auto-repair, and sends notifications via Telegram and Discord.

## How It Works

```
Gateway running
      │
      ▼
Health check every 2 min
      │
      ├── Healthy → continue monitoring
      │
      └── Unhealthy
           │
           ├── First failure → mark degraded
           ├── Retry every 5 sec
           ├── 3 failures → trigger repair
           │
           ▼
      Auto-repair
           │
           ├── Kill gateway process
           ├── Wait 5 seconds
           ├── Restart gateway
           ├── Verify health
           │
           ├── Success → resume normal monitoring
           └── Failure → retry (max 2 attempts)
                              │
                              └── Crash loop detected → notify + stop
```

## Configuration

| Variable                           | Default | Description                                |
| ---------------------------------- | ------- | ------------------------------------------ |
| `WATCHDOG_CHECK_INTERVAL`          | `120`   | Health check interval in seconds           |
| `WATCHDOG_DEGRADED_CHECK_INTERVAL` | `5`     | Retry interval when degraded (seconds)     |
| `WATCHDOG_MAX_REPAIR_ATTEMPTS`     | `2`     | Max auto-repair attempts before crash-loop |
| `WATCHDOG_CRASH_LOOP_WINDOW`       | `300`   | Window for crash-loop detection (seconds)  |
| `WATCHDOG_CRASH_LOOP_THRESHOLD`    | `3`     | Crashes in window to trigger crash-loop    |
| `WATCHDOG_AUTO_REPAIR`             | `true`  | Enable/disable auto-repair                 |
| `TELEGRAM_BOT_TOKEN`               | —       | Telegram bot token for notifications       |
| `TELEGRAM_ADMIN_CHAT_ID`           | —       | Chat ID for Telegram notifications         |
| `DISCORD_WEBHOOK_URL`              | —       | Discord webhook URL for notifications      |

## Lifecycle States

| State        | Description                                       |
| ------------ | ------------------------------------------------- |
| `stopped`    | Gateway is not running                            |
| `starting`   | Gateway process started, waiting for health check |
| `running`    | Gateway is healthy                                |
| `degraded`   | Health check failed, retrying                     |
| `crash_loop` | Too many crashes, auto-repair exhausted           |
| `repairing`  | Auto-repair in progress                           |

## Notifications

The watchdog sends notifications for:

* **Crash detected** — Gateway process exited unexpectedly
* **Crash loop** — 3+ crashes in 5-minute window
* **Auto-repair started** — Attempting to restart the gateway
* **Auto-repair succeeded** — Gateway is healthy again
* **Auto-repair failed** — Could not recover, needs manual intervention

### Telegram Format

```
🐺 Agentbot Watchdog
🔴 Crash loop detected - View logs
Trigger: `crash_loop`
Attempt count: 3
```

### Discord Format

Embedded message with color coding:

* 🔴 Red: crashes, failures
* 🟡 Yellow: repairing, degraded
* 🟢 Green: recovered, healthy

## Test Script

Test the watchdog by injecting an invalid config:

```bash theme={"dark"}
#!/bin/bash
CONTAINER_NAME="agentbot-<userId>"
docker exec "$CONTAINER_NAME" node -e "
  const fs = require('fs');
  const cfg = JSON.parse(fs.readFileSync('/data/.openclaw/openclaw.json'));
  cfg.hooks = cfg.hooks || {};
  cfg.hooks.transformDir = '/tmp/does-not-exist';
  fs.writeFileSync('/data/.openclaw/openclaw.json', JSON.stringify(cfg, null, 2));
"
```

The watchdog should detect the gateway failure and auto-repair within 2 health check cycles.