Skip to content
All letters

Do Skills Make Models Better or Worse?

Both, and the answer changes line by line. Knowledge the model already has can cost accuracy. The moment to act and the move to block are where a skill earns its place.

Your team has a folder of agent skills by now: markdown files the agent loads when a task matches. Sooner or later someone asks whether they help, and nobody knows, because nobody has run the agent without them. In an earlier letter I wrote about a skill that took a frontier model from 97% correct to 77% on its own company's product. The obvious follow-up is whether skills in general make models better or worse.

That question has the wrong shape. A skill is not one thing. Most skill files mix three kinds of line, and they push in opposite directions.

Three kinds of line

  • Knowledge. What the API does, what the framework is, what the book says. "Our auth endpoint returns a session token."
  • Invocation. When to run a procedure, and the instruction to commit to it instead of improvising. "Before writing the proposal, pull the client's two budget numbers from the call transcript."
  • Guards. A move to stop, enforced by code that runs before the move: a hook or a script shipped with the skill. "If the next action is buy-now and a required option is unselected, block it."

The third kind is the one most skill files get wrong. Written as a sentence, a guard is only advice, and the model can read straight past it. The guards in the research below were code.

The public work from the last three months lines up along that split.

Knowledge: negative where the model was trained

In September, Brad Bonanno published a walkthrough of book-to-skill, a free tool that turns a book you own into an agent skill. His own diagnosis undercuts the tool. Ask a model about a famous marketing book and it names the frameworks back to you. Ask it to write with them and you get the same generic copy as everyone else. The model had the knowledge. It failed at using it.

Then he says one of his best skills came from a book he does not own. He built it from five or six free written summaries and three summary videos, and calls the result "almost as good as the real thing." For a book whose value is a procedure, the full text was never the input.

My read, and the one my notes on that video record: a skill like that supplies invocation, not knowledge. The model needs to be told when to run the procedure and held to it, because left alone it drifts back to the most probable answer. It also fits the 97-to-77 result: that skill was generated from documentation the model had largely absorbed already, and the version that worked kept only the gotchas. Knowledge the model already holds adds little and still costs attention.

The exception is the one Bonanno names himself. A style book, where the value is the author's actual wording, needs the real text. Knowledge earns its place only where the model's base rate is low.

Guards: the largest lift, and the largest risk

A paper posted in August by researchers at Shanghai Jiao Tong University and Xiaohongshu, walked through here, trained a 9B model to repair broken agents by editing four hook points around them. The training rule was that a fix counts only if the agent scores higher when the tasks are rerun. The small trained editor reached 53.6% against 45.8% for a 397B model of the same family that was prompted to suggest fixes.

The ablation is the useful part. Removing the pre-action guard, which can veto or rewrite the agent's next move, cost 3.9 points. Removing post-feedback recovery cost 3.3. The two hooks that add context or instructions cost under a point each. Most of the value sat in code that stopped bad moves, not in better instructions, and the hooks closest to a written skill were the ones worth least.

The same paper prints the cost. When Gemini 3.5 Flash wrote the patches, 21 of 39 valid ones made things worse, and the worst took a batch from 7 of 10 tasks passing to 0 by overriding moves the agent had been getting right. A guard has authority, which is why it helps and why it can break everything. The caveats are in the source: three toy benchmarks, no coding agent, and plain reflection scored 55.8 in the same table.

What that leaves for invocation

Put the findings together and the answer to the title is not "fewer skills." My read is that skills help when they supply the moment to act, or a block that runs as code, and hurt when they restate what the model already knows. The invocation half is the part nobody in these sources measured. Bonanno's "almost as good" is his judgment, not a score. That makes it the half to test, not the half to assume.

The skills working developers keep tend to look like that. Theo's three trigger-phrase skills, shown this month, are procedures he kept running by hand, each started by a phrase. One has a model write code and a second model review it, capped at three rounds. The other two take a change through proof and a pull request, and one of them is allowed to merge only under a written six-item gate. He also says that letting agents edit his skills made one "quite bloated", which is why he writes them himself.

Where I stand

I counted my own before writing this: 68 skills, test folders in 5 of them, a frozen golden set in 1, and that one says in its header that it "CANNOT judge whether a summary is good". None has a with-and-without run. So I am not claiming my skills help. I am claiming I know which lines to suspect first, and so can you.

For any team with a skill folder:

  1. Sort every line into knowledge, invocation or guard. It takes minutes per file.
  2. Cut knowledge first, unless the model's base rate on it is low: your proprietary API, a framework released after its training cutoff, a convention it cannot guess. This is where negative lift hides.
  3. Keep invocation lines and measure them with the afternoon test from the earlier letter: five real tasks, with and without the skill, scored blind.
  4. Move guards out of prose and into code, then keep each one only with a rerun that includes the tasks already passing. A guard that rescues six tasks and breaks one is visible only if you rerun all of them.
  5. Rerun after every model upgrade. Every line was written against something the model could not do at the time. Anthropic's prompting guide for Opus 5 says the model now checks its own work, and that removing explicit verification instructions "reduces wasted tokens with no loss in quality." That is the vendor's claim, not a published measurement, but it describes exactly the kind of line that turns from help into cost without anyone touching the file.

Do skills make models better or worse? Line by line, both. Sort the lines and you know which half you are paying for.

Back to all lettersayal.tech / letters