> ## Documentation Index
> Fetch the complete documentation index at: https://docs.modusbrain.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Queue Operations Runbook

> Manage the operation queue in production environments.

# Queue operations runbook

"My queue looks wedged — what do I run?" The commands below are in the order
you probably want them. Shipped with v0.19.1 after a production incident
where the queue held for 90+ minutes before the operator noticed.

## First signal: jobs aren't running

```bash theme={null}
modusbrain doctor --json | jq '.checks[] | select(.name == "queue_health")'
```

`queue_health` flags two patterns:

* **stalled-forever**: active job whose `started_at` is older than 1h.
* **waiting-depth**: any per-name queue deeper than 10 (override via
  `MODUSBRAIN_QUEUE_WAITING_THRESHOLD`). Signals a missing `maxWaiting`.

## The worker is alive but wedged (dead pool)

The nastiest stall: the worker process is *running* (passes `ps` / `kill -0` /
container health), but its DB connection died (common behind a transaction
pooler) and never came back, so it claims no jobs and finishes nothing. Jobs
pile up with **0 active**. Liveness checks all pass; nothing crashes.

As of v0.42.22.0 this self-heals — you usually won't have to do anything:

* **The worker exits on its own dead pool.** Under a supervisor, the worker's
  DB-liveness probe runs and self-exits (`db_dead`) after \~3 minutes; the
  supervisor respawns it with a fresh pool.
* **The supervisor restarts a worker that stops making progress.** If a queue
  has claimable work, **0 live-lock active jobs**, and no completions for 15
  minutes while the child is alive, the supervisor restarts it (covers stuck
  handlers too, not just dead pools). Tune with `--wedge-restart-minutes` /
  `--wedge-restart-checks` on `modusbrain jobs supervisor` (0 disables).

The signal is loud now — check either:

```bash theme={null}
modusbrain jobs stats --queue default          # prints a WEDGED QUEUE line
modusbrain doctor --json | jq '.checks[] | select(.name == "wedged_queue")'
```

`wedged_queue` is a per-queue health **error** (0 active\_healthy + waiting > 0 +
stale completions). Manual fix if you ever need it:

```bash theme={null}
modusbrain jobs supervisor stop && modusbrain jobs supervisor start   # fresh pool
modusbrain jobs retry <id>                                        # dead-lettered jobs
```

## Triage commands

```bash theme={null}
# Who's active right now?
modusbrain jobs list --status active

# Who's waiting, biggest pile first?
modusbrain jobs list --status waiting --limit 50

# What's wrong with a specific job?
modusbrain jobs get <id>
```

## Rescue actions (in order of escalation)

```bash theme={null}
# Force-kill a single stuck job:
modusbrain jobs cancel <id>

# Clear a specific job entirely (last resort):
modusbrain jobs delete <id>

# Health smoke on the mechanism itself:
modusbrain jobs smoke --wedge-rescue
```

## What each subcheck means

* **stalled-forever** — A worker claimed a job, started executing, and has
  held the row for over an hour. The wall-clock sweep evicts jobs past
  2× `timeout_ms`; if one's still active, either no `timeout_ms` was set
  or the sweep is newly deployed and this job predates it. Cancel it.
* **waiting-depth** — Submitters are piling up jobs faster than workers
  drain them. Set `--max-waiting N` on the submission or on the programmatic
  `queue.add()` call. If you want a taller pile, raise the threshold via
  `MODUSBRAIN_QUEUE_WAITING_THRESHOLD=50 modusbrain doctor`.

## Self-check: is a worker even running?

```bash theme={null}
# If you're running autopilot with --no-worker, check that your external
# worker (systemd / Docker / OpenClaw service-manager) is alive:
modusbrain jobs list --status active | head -5
```

If the list is empty AND your submissions keep piling up, no worker is
claiming. Start one:

```bash theme={null}
MODUSBRAIN_ALLOW_SHELL_JOBS=1 modusbrain jobs work --concurrency 4
```

## Follow-ups tracked for v0.20+

* B7 — `minion_workers` heartbeat table for ground-truth liveness (the
  `--no-worker` probe and the dropped `queue_health` worker-heartbeat
  subcheck both need this).
* B3 — `modusbrain doctor --fix` learns to rescue queue wedges.