Skip to main content

​
Auto-improve a skill with modusbrain skillopt

You have a SKILL.md. Sometimes the agent following it does a great job, sometimes it forgets a step or pads the output. This tutorial takes you from that skill to a measurably better version of it, in one session, without you hand-editing the prose. By the end you’ll have written your first benchmark, watched the optimizer propose and test edits, and accepted an improvement that actually scored higher. Time: ~20 minutes. Cost: ~$1 in API calls for the worked example. Based on SkillOpt (Microsoft Research, May 2026).

​
The mental model (two sentences)

Your SKILL.md is the trainable parameter; the agent that reads it never changes. SkillOpt runs the agent against a benchmark of realistic tasks, proposes specific edits to the skill body, re-tests, and keeps a change only when it measurably beats the current version on a held-out slice. That’s the whole idea. The benchmark is how “better” gets defined — which is why writing it is the one part you can’t skip. Everything else is mechanical.

​
The easiest path: generate a starter, then strengthen it

You don’t start from a blank file. One command reads the SKILL.md and writes a full starter benchmark for you:
It infers what the skill produces, writes ~15 tasks (each with rule judges) to skills/meeting-prep/skillopt-benchmark.jsonl, and appends a # BOOTSTRAP_PENDING_REVIEW sentinel so nothing runs until a human has looked. Then you review and strengthen the judges (the generated checks are weak drafts), delete the sentinel line, and run:
If you run an agent over this brain (OpenClaw, Claude Code, Cursor, any MCP client with the modusbrain skills installed), it does this for you: just say “improve my meeting-prep skill.” It runs --bootstrap-from-skill, strengthens the judges, dry-runs for cost, runs the optimizer, and reports the diff + score delta back. You keep or discard. Read the rest of this tutorial to understand what that command produces — the benchmark format, how to strengthen a draft (or write one by hand), how to read the outcome, and where the output lands.

​
What you’ll need

  • modusbrain installed and a brain initialized (modusbrain --version works).
  • One embedding/chat provider configured. SkillOpt makes real LLM calls. modusbrain models doctor should show at least one reachable chat model.
  • A skill you want to improve, living at skills/<name>/SKILL.md. This tutorial uses a skill called meeting-prep — substitute your own name everywhere.
  • A clean git working tree for that skill file (SkillOpt refuses to run over uncommitted changes so it can never clobber your edits; --force overrides).
If you don’t have a skill yet, scaffold one first:

​
Step 1: Get a benchmark — generated or hand-written

A benchmark is a .jsonl file — one JSON object per line — where each line is a task plus a way to score the agent’s answer. It’s the crux: the benchmark IS your definition of “better.” The recommended way is to generate a starter (the section above): modusbrain skillopt meeting-prep --bootstrap-from-skill writes the file for you, then you strengthen the judges. The format below is exactly what it produces, so this section doubles as your guide to reviewing and sharpening a generated draft. To follow this tutorial verbatim (or to hand-curate from scratch), paste this complete 15-task starter. It’s deliberately generic — once you’ve seen the loop work, replace these tasks with your skill’s real cases (that’s Step 6):
Each line has three fields:
  • task_id — a unique label. Anything; you’ll see it in the audit trail.
  • task — the prompt the agent gets, exactly as a user would phrase it.
  • judge — how the answer is scored. kind: "rule" is deterministic and free (no LLM call): it runs a list of checks, and the task’s score is the fraction that pass.
The rule checks you can use: Rule judges are the right place to start. They’re free, deterministic, and they force you to say concretely what a good answer looks like. (judge.kind can also be "llm" with a rubric, or "qrels" for retrieval tasks — see the reference guide once you outgrow rules.)

​
The one gotcha: how many tasks you need

SkillOpt splits your benchmark three ways — train (propose edits against), sel (the held-out gate that decides accept/reject), and test (final score). The sel slice must have at least 5 tasks or the run refuses, so noise can’t masquerade as improvement. The default split is 4:1:5, which means sel is 1/10th of your tasks — so the default needs ~50 tasks before it’ll run. That’s too many for a first benchmark, which is why every command below passes --split 1:1:1: with the 15-task starter that’s a clean 5 train / 5 sel / 5 test, and sel hits the floor exactly.
If you ever see D_sel has N task(s) after split (need >=5), you either added fewer than 15 tasks or used a split whose middle number is too small a share. --split 1:1:1 on 15+ tasks is the simplest thing that works.
When you swap in your own tasks (Step 6), keep at least 15 and cover the boring middle, not just the edge cases. The benchmark IS your definition of quality; a thin benchmark optimizes for a thin definition.

​
Step 2: Preview the cost (dry run)

Before spending anything, see what the run will cost:
This makes zero LLM calls — it just prints the plan and the cost estimate. A ~15-task benchmark with defaults runs around 0.70–0.70–1.00. The preflight refuses to start a real run whose estimate exceeds --max-cost-usd (default $5.00), so you can’t get surprise-billed mid-run.
--dry-run exits with code 2 (“aborted”). That’s the convention for “did not run the optimization,” not a failure. The cost line is what you came for.

​
Step 3: Run it for real

You’ll watch it work: a baseline eval to set the bar, then per-step forward passes (run the skill), backward passes (propose edits), and a validation gate that runs each sel task’s judge 3 times and takes the median — accepting only if the median beats the current best by more than 0.05. When it finishes, the last lines tell you everything:

​
Reading the outcome

no_improvement is not a failure. It’s the gate doing its job: it would rather keep your known-good skill than accept a change it can’t prove is better.

​
Step 4: See what changed

The optimizer leaves a full audit trail under the skill:
The actual change to your skill is a normal git diff:
Run-level events (cost, model, scores per run) also land in the rotating audit log at ~/.modusbrain/audit/skillopt-YYYY-Www.jsonl.

​
Step 5: Accept or reject — and the bundled-skill rule

For a skill you own (your own skills/ dir): an accepted run rewrites SKILL.md in place. It’s already a git diff — review it, then git commit to keep it or git checkout to throw it away. Nothing is committed for you. For a skill that ships with modusbrain (anything under the modusbrain repo’s own skills/): SkillOpt refuses to overwrite it by default and writes the winner to skills/<name>/skillopt/best.md instead, so an optimization pass can never silently mutate a skill other people depend on. Two ways to handle that:
Rewriting a bundled skill in place now requires BOTH --allow-mutate-bundled AND --held-out <path> (a JSONL with the same shape as your benchmark, but at least 5 tasks whose IDs don’t appear in the benchmark). The held-out set is how the run proves the edit didn’t just learn the benchmark: a candidate that climbs the benchmark but slips on the held-out tasks is refused. Drop --held-out and the run hard-refuses and points you at proposed.md instead. Rule of thumb: --no-mutate when you want to read the diff before trusting it (no held-out needed); --allow-mutate-bundled --held-out only when you intend to commit a proven change to a shared skill.

​
Step 6: Iterate

The loop that actually makes skills better:
  1. Run it. If no_improvement, the benchmark probably can’t distinguish good from bad yet.
  2. Add tasks that capture what you wish the skill did differently. Saw the agent skip citations? Add {"op":"min_citations","arg":2}. Saw it ramble? Tighten max_chars.
  3. Re-run. A sharper benchmark gives the optimizer a real gradient to climb.
  4. When a run lands accepted, read the diff, commit it, and bank the win.
The skill you ship gets better every time the benchmark gets sharper. That’s the whole game: you’re not editing prose, you’re improving the definition of done and letting the optimizer chase it.

​
What you built

You wrote a benchmark that encodes what “good” means for one skill, previewed the cost, ran the optimizer, and either accepted a measurably better skill or learned your benchmark needs sharpening. Same loop scales to every skill you own — and modusbrain skillopt --all runs it across every skill that has a benchmark, under a brain-wide cost cap.

​
Where to go next

  • Full flag + exit-code reference, cost model, safety guards: docs/guides/skillopt.md
  • Every flag inline: modusbrain skillopt --help
  • Batch + fleet + background runs (--all, --target-models, --background), LLM and qrels judges, held-out test sets, and resume after a crash (--resume <run-id>): all in the reference guide above.
  • Generate a starter benchmark from the SKILL.md (the recommended way to start): modusbrain skillopt <name> --bootstrap-from-skill → review + strengthen the judges → delete the sentinel → --bootstrap-reviewed --split 1:1:1. Tune the count with --bootstrap-tasks N (max 50).
  • Bootstrap from existing routing fixtures instead: modusbrain skillopt <name> --bootstrap-from-routing (routing tasks test dispatch, not quality — tighten them).