​

Frontmatter scan: DB-backed incremental state (Phase 2 design sketch)

​

Why this exists

​

Goal

​

Schema

CREATE TABLE frontmatter_scan_state (
  source_id    TEXT NOT NULL REFERENCES sources(id) ON DELETE CASCADE,
  path         TEXT NOT NULL,  -- relative to source.local_path
  mtime_ms     BIGINT NOT NULL,
  content_hash TEXT NOT NULL,  -- sha256 of file content at scan time
  codes        JSONB NOT NULL DEFAULT '[]'::jsonb,  -- ParseValidationCode[]
  last_scanned_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  PRIMARY KEY (source_id, path)
);

CREATE INDEX frontmatter_scan_state_has_issues_idx
  ON frontmatter_scan_state (source_id)
  WHERE codes != '[]'::jsonb;

• mtime_ms + content_hash: incremental check picks one. mtime is faster (no read); content_hash is the truth (defeats touch-without-change cases). The incremental walker uses mtime as a fast gate and content_hash as the fallback when mtime suggests change.
• codes JSONB: per-row error code list, NULL/[] means clean. Doctor aggregates with jsonb_array_length(codes) > 0.
• Partial index on WHERE codes != '[]'::jsonb: doctor’s aggregate query only walks rows with issues, which is a small fraction of pages.

​

Migration shape

// src/core/migrate.ts — append after the v80 entry
const migrations = [
  // ...existing v1-v80...
  {
    version: 81,
    name: 'frontmatter_scan_state',
    sql: `
      CREATE TABLE IF NOT EXISTS frontmatter_scan_state (...);
      CREATE INDEX IF NOT EXISTS frontmatter_scan_state_has_issues_idx ...;
    `,
  },
];

​

Writers

1. Sync-side write (canonical). src/core/sync.ts:performSync already parses every file it touches. After the existing parseMarkdown call, UPSERT into frontmatter_scan_state with the file’s path / mtime / content_hash / codes. Cost: one row per file synced. Zero extra parse work — the parse already happened.
2. Incremental scan (modusbrain frontmatter scan --incremental). Walks the disk via walkBrainTree, for each file checks mtime > last_scanned_at OR content_hash != stored, only re-parses changed files. Most ticks: zero work after the first full backfill. Also exposed as an autopilot cycle phase (frontmatter_scan) so it runs alongside the other periodic maintenance phases.

• Files edited outside sync (user opens an editor, saves, never git commits).
• Sources whose local_path isn’t a git repo (sync only sees git-touched files).

​

Doctor reader

// src/commands/doctor.ts:frontmatter_integrity (Phase 2 shape)
const rows = await engine.executeRaw<{ source_id: string; issues: number }>(
  `SELECT source_id, count(*) FILTER (WHERE jsonb_array_length(codes) > 0)::int AS issues
   FROM frontmatter_scan_state
   GROUP BY source_id`,
);

​

Sequencing concerns

1. First-ever scan. A fresh upgrade has no rows in frontmatter_scan_state. Two options:
   • Lazy: doctor reports “no scan state yet; run modusbrain frontmatter scan --incremental once” (operator-driven).
   • Eager: the migration that creates the table also enqueues an autopilot cycle job to do the first full scan.
   Recommendation: lazy, with a clear hint. The autopilot path is heavier surface (must add the new frontmatter_scan phase to the existing cycle.ts machinery + the doctor-routed background job system).
2. Source archival / deletion. frontmatter_scan_state has ON DELETE CASCADE on sources(id), so soft-delete + 72h TTL + purge already clean it up. No additional logic needed.
3. Path renames inside a source. Sync would DELETE the old row by path (via a periodic reconcile step) and INSERT the new row. Without that step, the table accumulates stale path rows. Either:
   • A reconcile step in the incremental scanner: any path-row not seen during the walk gets deleted.
   • Or: doctor reports “N stale rows in frontmatter_scan_state” as a freshness signal, with modusbrain frontmatter scan --reconcile as the remediation.

​

Cost estimate

• One UPSERT per file synced. Negligible vs the parse + DB write that sync already does.
• Incremental refresh runtime: dominated by mtime stats. ~ms per 1000 files on SSD.
• Doctor read: one indexed SQL query. Sub-100ms on any brain size.

​

What this design deliberately does NOT do

• Replace v0.38.2.0’s bounded-walk safety net. Phase 2 makes the steady-state cheap, but the disk walker (with its deadline check) stays as the source-of-truth fallback for sources whose scan state is missing or stale. Belt-and-suspenders.
• Introduce a separate frontmatter validation rule set. Reuses parseMarkdown(..., {validate: true}) and the existing ParseValidationCode enum. Single source of truth.
• Add a new background daemon. Wires into the existing autopilot-cycle Minion handler as a new phase, alongside sync / extract / embed / etc.

​

Open questions for the implementer

1. Path normalization. pages.source_path and the disk walker’s relative path computation are similar but not identical (slashes, leading ./, etc.). The incremental scanner needs to match what sync stores so UPSERTs key correctly. Audit before writing.
2. Soft-delete interaction. A page that gets soft-deleted in the DB (v0.26.5) still has a file on disk. Should the incremental scan continue to track its frontmatter state? Probably yes (so a future restore_page doesn’t surprise with stale frontmatter), but worth confirming with the soft-delete owner.
3. Two-phase rollout. Land the table + writes first, let it backfill for a release cycle, then switch the doctor reader. Avoids the “Phase 2 ships but the table is empty” case where doctor regresses to reporting “no scan state.”

​

TODO file entry

- [ ] Implement Phase 2: DB-backed frontmatter scan state.
      Design lives at docs/architecture/frontmatter-scan-incremental.md.
      Schema migration v81 + sync-side UPSERT + incremental scan command
      + autopilot cycle phase + doctor reader. Two-phase rollout: ship
      table + writes first; flip the reader one release later.