# Measuring What a Claude Code Skill Actually Adds

> What a Claude Code skill eval measures, why we run every prompt with and without the plugin, and how a passing suite can hide checks that only test the model.

- Source: https://verygood.ventures/blog/evaluating-claude-code-skills-with-ablation/
- Published: 2026-11-16
- Author: Dominik Simonik
- Tags: AI, Claude Code, Flutter, Open Source, Testing, Developer Tools

---

Our Flutter plugin ships fifteen skills. Each one is a markdown file of instructions, an opinion we wrote down for a model to read: use Equatable, name the loading state `LoginInProgress`, keep events past tense. What we had no way of knowing was whether the model read any of it.

Ask a model to write a login bloc without the plugin and you still get decent code. Sealed classes, pattern matching, sensible names. It is just not how our team does it, and that gap is the only thing a skill exists to close.

**We use evals to check whether a skill closes that gap.** An eval, short for evaluation, is a test that asks the model to do something and grades what comes back. Ours do two jobs: measure whether a skill is doing anything, and keep measuring it as the model underneath changes.

Both jobs behave differently from unit tests. The same prompt can load a skill on one run and skip it on the next, and the model changes even when nothing in your project does, so the suite has to be rerun and reread.

## Why we built a 102-case suite

The [VGV AI Flutter Plugin](/blog/very-good-ai-flutter-plugin/) ships fifteen skills: bloc, testing, layered architecture, accessibility, the rest of how we build Flutter apps. [VGV Wingspan](/blog/how-we-built-wingspan-claude-code-plugin/) ships twelve more alongside it, but the suite in this post covers the Flutter plugin. Every skill is our opinion, written down, and our evidence that any of it worked was that it had looked fine when we tried it by hand. We had never measured it.

So we built a suite: 102 cases, open source, running on [`claude plugin eval`](https://code.claude.com/docs/en/plugin-evals). It answers four questions we had been guessing at: whether a skill gets loaded when it should, which of our conventions the model would have followed anyway, whether a change to a skill broke it, and whether any of those answers still holds a month later.

Reading the results was harder than building the suite, because three testing instincts got in the way: same input, same output; a test that passes is a good test; red means bug. Each holds for a function and fails for a model. The second one cost us the most, and most of this post is about it.

## When a skill does not trigger

The first question is whether the model ever read the skill.

Someone types "write a LoginBloc for email and password authentication" and hits enter. Fifteen [skill](https://code.claude.com/docs/en/skills) descriptions, the short summary at the top of each skill file, are sitting in front of the model. Our bloc skill's description lists the name `LoginBloc`, so the model has a fair chance of picking it. Nothing else in the sentence says Equatable, or sealed classes, or VGV. The model reads the descriptions, decides whether to load a skill, and starts typing. That decision is called routing, and **the answer does not show whether our skill was loaded**. Good Dart comes back either way.

So we measure it instead of assuming. Each check in an eval is called a grader, and it returns pass or fail. Our routing grader reads the run's trace, the log of every tool the model called, and looks for a call to the Skill tool, which is how Claude Code loads a skill. Across the three runs behind this post, the skill loaded every time. If that rate drops, edit the description, because the description is what the model reads before deciding.

![How a Claude Code skill eval checks routing: the prompt, fifteen skill descriptions in context, the model's choice, and the Skill tool call the skill-fired grader reads in the trace, which fired in 3 of 3 runs](/assets/images/blog/evaluating-claude-code-skills-with-ablation/skill-routing.png)

One thing to get right before trusting the number: we measure with the whole plugin loaded, not with the one skill under test. Fifteen descriptions compete for the same decision, and that is the decision a real session puts in front of the model.

## What the model actually wrote

The other half of an eval reads the code that came back. Each of those graders asks one question about the output: does the state class extend Equatable, do the event names end in a past-tense verb.

A skill can load and still be ignored here. The model follows the overall shape of the skill and drops the one rule the case was written for. That did not happen in the case below, but it is the reason the content graders sit next to the routing one.

## How we score a case

Here is a real prompt from the suite. It reads like a ticket, never names the skill, and never mentions sealed classes or [Equatable](https://pub.dev/packages/equatable), because the whole question is whether the model gets there on its own:

```markdown
Write a LoginBloc for email and password authentication, with a submit
event and a logout event. While the request is in flight the button shows
a spinner. On success the UI needs the signed-in User object; on failure
it needs the error message to display. Nothing else is on screen before
the first attempt. Output Dart code only.
```

That prompt has no expected output attached to it. It has eight graders, each returning pass or fail, and each carrying a weight:

| Grader | Type | Weight | Checks |
|---|---|---|---|
| `skill-fired` | `tool_used` | 3 | a Skill call naming `bloc` in the trace |
| `extends-equatable` | `regex` | 1 | `extends Equatable` |
| `declares-props` | `regex` | 1 | a `props` getter |
| `sealed-event-hierarchy` | `regex` | 1 | `sealed class LoginEvent` |
| `sealed-state-hierarchy` | `regex` | 1 | `sealed class LoginState` |
| `pinned-in-progress-name` | `regex` | 1 | the name `LoginInProgress` |
| `final-class-subclasses` | `llm` | 1 | subclasses declared `final class` |
| `past-tense-event-names` | `llm` | 1 | event names start with the subject and end in a past-tense verb |

The weights add up to 10. A run scores the weight that passed, divided by 10, and the case score is the mean of its runs.

![How one eval case is scored: eight weighted graders, run scores of 0.50, 0.70 and 0.40, and a case score of 0.53 as their mean](/assets/images/blog/evaluating-claude-code-skills-with-ablation/score-pipeline.png)

A case passes when its score clears a threshold. The runner, the `claude plugin eval` command, defaults to 1.0. We set ours to 0.8, because two graders are scored by a second model, and that judge can mark a correct answer wrong. A threshold of 0.8 lets a fifth of the weight fail without saying which fifth, so we pay for the loosening with weighting. Make one grader heavy enough and the case cannot clear 0.8 without it. In ours that is the routing grader, at weight 3.

![The same eval case scored with the routing grader at weight 1 and at weight 3, clearing and then missing the 0.8 threshold](/assets/images/blog/evaluating-claude-code-skills-with-ablation/grader-weighting.png)

The three grader types reach pass or fail differently, and each has its own blind spot.

**`regex` runs over the reply text.** It cannot tell a class named `LoginSubmitted` from the same name sitting in a comment.

**`tool_used` reads the run's trace and counts calls.** It sees what the model did, not what it said it did, which is why routing uses it.

**`llm` hands the reply and a written criterion to a second model.** We call these rubric graders. The judge votes three times on every reply, and the votes are in the report, so a 2-1 split is visible instead of hidden inside a pass.

## Running every case with and without the plugin

None of that tells us which part of the score we earned. **A grader that passes whether or not the plugin is loaded is measuring the model**, and from inside a passing run it looks exactly like a grader measuring us. In a unit test, the code under test is what runs. In an eval, the model runs, and it can satisfy a check without any help from the skill.

So we run every case twice, once with the plugin loaded and once with no plugin at all, each in a throwaway home directory. In eval terms that is an ablation, and each half of the pair is an arm. We call the model in the arm without the plugin the bare model. Both arms start from the same fixture, a script that scaffolds an empty Flutter project for the model to write into. The fixture is kept free of anything a skill teaches, because the bare model reads it too.

**The gap between the two arm scores, reported as Δ, is the only part of the number our skill can claim.** The two-arm comparison is not our idea. The runner runs both arms by default, and `--ablation none` turns the comparison off. What the runner does not show you is which graders the gap comes from. We had to read that ourselves.

![One eval prompt run in two arms, scoring 1.00 with the plugin and 0.53 without it for a delta of 0.47](/assets/images/blog/evaluating-claude-code-skills-with-ablation/ablation-two-arms.png)

Running the same prompts with the plugin removed is what sent us back to the graders. One pull request touched graders in seven of the fifteen skills. In four of its cases, some graders passed without the plugin, and reworking them roughly doubled those cases' Δ.

Grader files are short. A regex grader is a frontmatter block:

```yaml
# graders/extends-equatable.md
---
type: regex
pattern: 'extends Equatable'
---
```

The routing grader has one line that changes the score:

```yaml
# graders/skill-fired.md
---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?bloc"'
weight: 3
arm: both
---
```

By default the runner drops a `tool_used` grader on the Skill tool from the score in both arms, because it can never pass without the plugin and counting it would inflate Δ. Ours sets `arm: both`, which forces it back into the number. We do that on purpose: the weight of 3 exists so that a routing miss sinks the case below 0.8, and a grader the runner does not score cannot sink anything. The cost is a Δ that includes routing. Leave that line out and **the Δ for the case below reads 0.24 instead of 0.47**, which is why we read the per-grader rows and not only the headline.

## What the login bloc case showed

Here is the login bloc case, three runs per arm, measured in October 2026. The arm with the plugin scored 1.00 on every run. The arm without it scored 0.50, 0.70 and 0.40, for a mean Δ of 0.47.

| Grader | with | without |
|---|---|---|
| `skill-fired` | 3/3 | 0/3 |
| `extends-equatable` | 3/3 | 1/3 |
| `declares-props` | 3/3 | 1/3 |
| `final-class-subclasses` | 3/3 | 2/3 |
| `past-tense-event-names` | 3/3 | 3/3 |
| `pinned-in-progress-name` | 3/3 | 3/3 |
| `sealed-event-hierarchy` | 3/3 | 3/3 |
| `sealed-state-hierarchy` | 3/3 | 3/3 |

The rows matter more than the Δ. **Four of the eight graders pass in both arms on every run**: the sealed hierarchies, past-tense event names, and `LoginInProgress`, which is the loading-state name our skill pins. The bare model produces all four unprompted, so none of them is evidence that the skill did anything. We call those free graders.

What the plugin actually produced was Equatable. Here is the state class from the first run of each arm, same prompt and same fixture, trimmed to the parts that differ:

```dart
// with the plugin
sealed class LoginState extends Equatable {
  const LoginState();

  @override
  List<Object> get props => [];
}

final class LoginInitial extends LoginState {}

final class LoginInProgress extends LoginState {}

final class LoginSuccess extends LoginState {
  const LoginSuccess({required this.user});

  final User user;

  @override
  List<Object> get props => [user];
}
```

```dart
// without the plugin
@immutable
sealed class LoginState {
  const LoginState();
}

final class LoginInitial extends LoginState {
  const LoginInitial();
}

final class LoginInProgress extends LoginState {
  const LoginInProgress();
}

final class LoginSuccess extends LoginState {
  const LoginSuccess(this.user);

  final User user;
}
```

That run reaches for `@immutable` and never defines equality. `@immutable` asks the analyzer to check that every field is final. It says nothing about `==`, so `==` stays identity. Two `const LoginInProgress()` values are the same canonical object and compare equal, but only when the bloc emits them with `const`. `LoginSuccess` carries a user that is only known at runtime, so every `LoginSuccess(user)` is a new object, and **`LoginSuccess(user) == LoginSuccess(user)` is false**. Any `bloc_test` that expects that state by value fails.

The other two runs without the plugin answered differently again. One reached for Equatable unprompted and scored 0.70. The other dropped both Equatable and `@immutable` and scored 0.40. One prompt gave three different answers about equality, and that variance is the reason the runner defaults to three runs per arm.

![Three runs of the same prompt without the plugin, scoring 0.50, 0.70 and 0.40, each defining state equality differently](/assets/images/blog/evaluating-claude-code-skills-with-ablation/three-runs-without.png)

## Which graders earn their keep

A grader earns its keep when it passes with the plugin and fails without it. Only those graders add to the Δ. Splitting the 0.47 by grader shows which ones they are on the login bloc case:

- **Routing, 0.30.** `skill-fired` carries weight 3 and fails every run without the plugin, because a skill that is not installed cannot load. It tells you routing worked in three of three runs, not what the skill teaches.
- **Equatable, 0.13.** `extends-equatable` and `declares-props` pass in only one run out of three without the plugin.
- **Final classes, 0.03.** `final-class-subclasses` passes in two runs out of three without the plugin.
- **Everything else, 0.** The sealed hierarchies, past-tense event names and `LoginInProgress` pass in every run of both arms. These are the free graders.

![The 0.47 delta split by grader: about 0.30 from routing, 0.13 from the two Equatable graders, 0.03 from final-class subclasses](/assets/images/blog/evaluating-claude-code-skills-with-ablation/delta-by-grader.png)

So of the 0.47, the content of the skill accounts for 0.16. The rest is the skill loading at all.

The split also changes over time. The bare model keeps improving while your skill stays the same, so a grader that separates the arms today can turn free next month. The case still scores 1.00 with the plugin when that happens, so the only way to notice is to rerun both arms and compare the rows.

That raises the question of whether to delete free graders. We keep them, by two rules.

**A free grader stays.** A regex grader makes no model call, so keeping it costs almost nothing. It is also the only thing that would catch a future model dropping the convention, which is the moment the skill has to start teaching it.

**Deletion needs a reason other than the score.** A genuine duplicate, a restatement of the prompt, or something the format requires anyway, such as `testWidgets` in a widget test.

Pinning the model in CI, which the [runner documentation](https://code.claude.com/docs/en/plugin-evals#run-evals-in-ci) recommends, does not stop this drift. It only decides when you see it. With a pin, the split stays as it was the day you measured it. When you move the pin, rerun both arms before you trust the scores again.

## What the suite still cannot tell us

**The judge is uncalibrated.** The judge is the second model that scores the rubric graders. Calibrating it means hand-labeling a batch of passes and failures and measuring the judge against them. We have not done that for any of our 158 rubric graders. On this case it costs little: six of the eight graders are regex or `tool_used`, need no labels, and carry 93% of the Δ. The two that are judged, `final-class-subclasses` and `past-tense-event-names`, are the ones to calibrate first.

So we keep the claim precise. The suite measures whether the skill fires and whether the conventions it teaches survive in the output. Whether the output is good code is a different question, and we answer it the way we always have, in review.

**Nothing confirms the Dart parses.** The runner has no custom-code graders, so the suite grades text, and a snippet can satisfy every regex and still not compile.

**Three runs per arm is not a confidence interval.** It shows that a result is not a one-off. It does not show how wide the spread is.

**Sometimes no grader separates the arms.** When that holds after three runs, the case is working correctly and the skill is restating what the model already does. Cut the skill, or find the convention the model does not already follow.

## Where to start evaluating your own Claude Code skills

Start with one skill and one question: does it fire?

There are two defensible positions on when to write a grader. Write the evals before the skill and let the baseline define what the skill should do, or write graders only for failures you have already seen. We do the first for routing, which is the only failure you can predict without looking, and the second for everything a skill teaches. Our best graders came from reading output we did not like.

A case is a directory, and nothing registers it. Here is the layout:

```text
evals/<skill>/<case-name>/
├── prompt.md          # frontmatter: run limits, tools, tags; body: the prompt
├── case.yaml          # optional: schema_version, name, and the fixture hook
├── fixture.sh         # scaffolds the project; runs with --scaffold
└── graders/
    ├── skill-fired.md
    └── <one file per grader>
```

**Start with routing.** Take your most-used skill and write three cases, each a prompt a user would really send, phrased as a ticket and never naming the skill. Give each case one grader: copy `skill-fired.md` from above and replace `bloc` in `input_match` with your skill's name. The [plugin eval documentation](https://code.claude.com/docs/en/plugin-evals) lists every frontmatter field. Run the cases from the plugin root. `--case` takes a name glob, so one pattern picks up all three:

```bash
claude plugin eval . --case '<your-case-prefix>*'
```

Each case runs three times per arm by default. The terminal prints one row per case with `WITH`, `W/OUT` and `Δ`, and the per-grader split lives in `results/<timestamp>/report.html`, next to `aggregate-result.json`. Every table in this post was read out of that report. If the skill fires once in three, edit its description, since that is all the model reads before deciding.

**Then read the other arm**, before writing any more content graders. Open the report and compare all three runs of each arm. A difference that shows up in every run is what your skill contributes. A difference that changes from run to run is noise.

We run the suite after a merge rather than as a gate, because one red case is a prompt to rerun it three more times and read the `NOTES` column, not a verdict.

The full suite, the fixture, the mocks, and the README listing the traps we hit are in the [VGV AI Flutter Plugin repository](https://github.com/VeryGoodOpenSource/vgv-ai-flutter-plugin). The bloc cases are the closest thing to copy when you write your first one.

## Claude Code Skill Evals FAQs

### How do you test a Claude Code skill?

Write prompts a developer would actually send, never naming the skill, and grade the reply with a set of weighted graders through <code>claude plugin eval</code>. Run every case in two arms, with the plugin loaded and without, so you measure what the skill adds rather than what the model already knows. The VGV AI Flutter Plugin suite holds 102 cases across 15 skills and is <a href="https://github.com/VeryGoodOpenSource/vgv-ai-flutter-plugin">open source</a>.

### Why do my eval graders stop detecting anything over time?

Because the model underneath them keeps changing. A grader earns its keep by separating the with-plugin arm from the without-plugin arm, and as the bare model improves it starts producing the convention unprompted, so the check goes <em>free</em>: it passes everywhere and measures nothing. Pinning the model in CI stops a model rollout from looking like a regression, but it does not keep your graders meaningful. The only way to find out is to rerun both arms and read the per-grader split.

### Why does my Claude Code skill not trigger?

Routing is a model decision and it is nondeterministic, so a skill can fire on one run and sit silent on the next for the same prompt. Measure it directly with a <code>tool_used</code> grader on the Skill tool rather than inferring it from the output. If the skill routes inconsistently, the fix is usually its description, which is the text the model reads when deciding whether to load it.

### What is ablation in LLM evals?

Running the same prompts with and without the component under test and comparing the scores. For a Claude Code skill, the without arm runs the bare model in a throwaway environment with no plugin loaded. A grader that passes in both arms is measuring the model, not the skill.

### Should LLM evals block a merge?

Not with this much variance. The VGV suite runs after a merge to main, scoped to the skills that changed, with the plugin arm only, and the job continues on error. One red case is a prompt to run three more times and read the <code>NOTES</code> column, not a verdict.

### What is the difference between an eval and a unit test?

A unit test asserts that a function returns one exact value, and the input fully determines the output. An eval estimates how often a model produces a property you care about, across repeated runs, against a threshold. The control group, the weighted average, and the repeated runs replace <code>expect</code> because the subject is a distribution, not a function.
