​

Queue operations runbook

​

First signal: jobs aren’t running

modusbrain doctor --json | jq '.checks[] | select(.name == "queue_health")'

• stalled-forever: active job whose started_at is older than 1h.
• waiting-depth: any per-name queue deeper than 10 (override via MODUSBRAIN_QUEUE_WAITING_THRESHOLD). Signals a missing maxWaiting.

​

The worker is alive but wedged (dead pool)

• The worker exits on its own dead pool. Under a supervisor, the worker’s DB-liveness probe runs and self-exits (db_dead) after ~3 minutes; the supervisor respawns it with a fresh pool.
• The supervisor restarts a worker that stops making progress. If a queue has claimable work, 0 live-lock active jobs, and no completions for 15 minutes while the child is alive, the supervisor restarts it (covers stuck handlers too, not just dead pools). Tune with --wedge-restart-minutes / --wedge-restart-checks on modusbrain jobs supervisor (0 disables).

modusbrain jobs stats --queue default          # prints a WEDGED QUEUE line
modusbrain doctor --json | jq '.checks[] | select(.name == "wedged_queue")'

modusbrain jobs supervisor stop && modusbrain jobs supervisor start   # fresh pool
modusbrain jobs retry <id>                                        # dead-lettered jobs

​

Triage commands

# Who's active right now?
modusbrain jobs list --status active

# Who's waiting, biggest pile first?
modusbrain jobs list --status waiting --limit 50

# What's wrong with a specific job?
modusbrain jobs get <id>

​

Rescue actions (in order of escalation)

# Force-kill a single stuck job:
modusbrain jobs cancel <id>

# Clear a specific job entirely (last resort):
modusbrain jobs delete <id>

# Health smoke on the mechanism itself:
modusbrain jobs smoke --wedge-rescue

​

What each subcheck means

• stalled-forever — A worker claimed a job, started executing, and has held the row for over an hour. The wall-clock sweep evicts jobs past 2× timeout_ms; if one’s still active, either no timeout_ms was set or the sweep is newly deployed and this job predates it. Cancel it.
• waiting-depth — Submitters are piling up jobs faster than workers drain them. Set --max-waiting N on the submission or on the programmatic queue.add() call. If you want a taller pile, raise the threshold via MODUSBRAIN_QUEUE_WAITING_THRESHOLD=50 modusbrain doctor.

​

Self-check: is a worker even running?

# If you're running autopilot with --no-worker, check that your external
# worker (systemd / Docker / OpenClaw service-manager) is alive:
modusbrain jobs list --status active | head -5

MODUSBRAIN_ALLOW_SHELL_JOBS=1 modusbrain jobs work --concurrency 4

​

Follow-ups tracked for v0.20+

• B7 — minion_workers heartbeat table for ground-truth liveness (the --no-worker probe and the dropped queue_health worker-heartbeat subcheck both need this).
• B3 — modusbrain doctor --fix learns to rescue queue wedges.