> ## Documentation Index
> Fetch the complete documentation index at: https://docs.modusbrain.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Storage tiering

# Storage Tiering: db-tracked vs db-only directories

## Overview

ModusBrain supports storage tiering to separate version-controlled content from bulk machine-generated data. This prevents git repositories from becoming bloated with large amounts of automatically generated content while still preserving it in the database.

> Note on naming: prior to v0.22.11 the keys were `git_tracked` / `supabase_only`. The canonical names are now `db_tracked` / `db_only` (engine-agnostic — works on both PGLite and Postgres). The deprecated keys still load with a once-per-process warning. Run `modusbrain doctor --fix` for an automated rename when that path lands.

## Configuration

Add a `storage` section to your `modusbrain.yml` file in the brain repository root:

```yaml theme={null}
storage:
  # Directories that are version-controlled (human-edited, committed to git).
  db_tracked:
    - people/
    - companies/
    - deals/
    - concepts/
    - yc/
    - ideas/
    - projects/

  # Directories persisted via the brain database only (bulk machine-generated
  # content). Written to disk as a local cache but not committed to git;
  # `modusbrain sync` auto-manages .gitignore for these paths. `modusbrain export
  # --restore-only` repopulates missing files from the database.
  db_only:
    - media/x/
    - media/articles/
    - meetings/transcripts/
```

Path requirements:

* Each directory must end with `/` for canonical form. The validator auto-normalizes missing trailing slashes (one-time info note shows what changed).
* A directory cannot appear in both tiers — that's a tier-overlap error and `loadStorageConfig` throws `StorageConfigError`. Edit `modusbrain.yml` to remove the overlap and try again.

## Behavior Changes

### 1. `modusbrain sync` — automatic .gitignore management

When storage configuration is present, `modusbrain sync` automatically manages `.gitignore` entries on every successful sync:

* Adds missing `db_only` directory patterns to `.gitignore`.
* Idempotent — re-running adds no duplicate entries.
* Stable comment header so the managed block is grep-able.
* Skipped on `--dry-run` (don't mutate disk in preview mode).
* Skipped on `blocked_by_failures` status (sync state is inconsistent).
* Skipped when the repo is a git submodule (`.git` is a file, not a directory) — submodule .gitignore changes don't survive parent updates. A warning explains.
* Skipped entirely when `MODUSBRAIN_NO_GITIGNORE=1` is set (escape hatch for shared-repo setups where a maintainer wants modusbrain to leave .gitignore alone).
* Failures (write permission denied, etc.) are caught and logged, never crash sync.

Example `.gitignore` addition:

```gitignore theme={null}
# Auto-managed by modusbrain (db_only directories)
media/x/
media/articles/
meetings/transcripts/
```

### 2. `modusbrain export --restore-only` — repopulate missing db\_only files

```bash theme={null}
# Restore only missing db_only files from the database.
modusbrain export --restore-only --repo /path/to/brain

# Filter by page type.
modusbrain export --restore-only --type media --repo /path/to/brain

# Filter by slug prefix.
modusbrain export --restore-only --slug-prefix media/x/ --repo /path/to/brain

# Combine filters.
modusbrain export --restore-only --type media --slug-prefix media/x/ --repo /path/to/brain
```

The `--restore-only` flag:

* Resolves repoPath via the chain `--repo` → typed `sources.getDefault()` → hard error.
  Never falls through to the current directory.
* Only exports pages that match `db_only` patterns AND are missing from disk.
* Ideal for container restart recovery and fresh clones.

### 3. `modusbrain storage status` — storage-tier health dashboard

```bash theme={null}
# Human-readable status.
modusbrain storage status --repo /path/to/brain

# JSON output for scripts and orchestrators.
modusbrain storage status --repo /path/to/brain --json
```

Output includes:

* Total page counts by storage tier.
* Disk usage breakdown by tier.
* Missing files that need restoration (top 10 shown; full list in `--json`).
* Configuration validation warnings.
* Current tier directory listing.

Example output:

```
Storage Status
==============

Repository: /data/brain
Total pages: 15,243

Storage Tiers:
-------------
DB tracked:     2,156 pages
DB only:        12,887 pages
Unspecified:    200 pages

Disk Usage:
-----------
DB tracked:     45.2 MB
DB only:        2.1 GB

Missing Files (need restore):
-----------------------------
  media/x/tweet-1234567890
  media/x/tweet-0987654321
  ... and 47 more

Use: modusbrain export --restore-only --repo "/data/brain"

Configuration:
--------------
DB tracked directories:
  - people/
  - companies/
  - deals/

DB-only directories:
  - media/x/
  - media/articles/
  - meetings/transcripts/
```

## Validation

`loadStorageConfig` runs `normalizeAndValidateStorageConfig` after parsing:

* Auto-fixes (silent, with one-time info note showing what changed):
  * Missing trailing `/` is added: `'media/x'` → `'media/x/'`.
* Throws `StorageConfigError` (caller sees a clean exit-1 with actionable message):
  * Same directory in both `db_tracked` and `db_only` (ambiguous routing).

## Use cases

### Brain repository scaling

Perfect for brain repositories crossing 50K-200K+ files where:

* Core knowledge (people, companies, deals) remains git-tracked.
* Bulk data (tweets, articles, transcripts) moves to db\_only.
* Development stays fast with smaller git repos.
* Full data remains available via the database.

### Container-based deployments

Essential for ephemeral container environments:

* Git repo contains only essential files.
* Container restarts don't lose db\_only data.
* `modusbrain export --restore-only` quickly restores bulk files when needed.
* Local disk acts as a cache layer.

### Multi-environment consistency

Enables consistent data access across environments:

* Development: small git clone, restore bulk data on demand.
* Production: full dataset via the database, selective local caching.
* CI/CD: fast tests with git-tracked data only.

## Migration strategy

1. **Assess current repository**: use `modusbrain storage status` to understand current distribution.
2. **Plan directory structure**: identify which directories should be db\_tracked vs db\_only.
3. **Create `modusbrain.yml`**: add storage configuration to the repository root.
4. **Test with dry-run**: `modusbrain sync --dry-run` to verify behavior; `.gitignore` is NOT touched on dry-run.
5. **Run a real sync**: `modusbrain sync` updates `.gitignore` automatically on success.
6. **Verify restore**: test `modusbrain export --restore-only --repo .` against a small db\_only directory.

## Best practices

* **Directory naming**: end storage paths with `/` (canonical form). The validator normalizes if you forget.
* **Start small**: begin with clearly machine-generated directories in `db_only`.
* **Address validation errors**: tier overlap is an error, not a warning. Fix it before sync.
* **Test restore**: regularly test `--restore-only` in staging environments.
* **Document decisions**: comment your `modusbrain.yml` to explain tier choices.

## PGLite engine note

On the PGLite engine (modusbrain's local-only embedded Postgres), the "DB" your db\_only pages live in IS the local file modusbrain uses for everything else. The `.gitignore` housekeeping still helps (keeps bulk content out of git history), but the offload-to-DB promise is technically vacuous. A once-per-process soft-warn explains when the engine is detected. To get full tiering, migrate to Postgres with `modusbrain migrate --to supabase`.

## Compatibility

* **Backward compatible**: systems without `modusbrain.yml` work unchanged.
* **Progressive enhancement**: add configuration when needed.
* **Database unchanged**: all data remains in Postgres regardless of tier.
* **Existing workflows**: all existing `sync` and `export` behavior preserved.
* **Deprecated keys**: `git_tracked` / `supabase_only` still load with a once-per-process warning.