Measuring What a Claude Code Skill Actually Adds
What a Claude Code skill eval measures, why we run every prompt with and without the plugin, and how a passing suite can hide checks that only test the model.
Our Flutter plugin ships fifteen skills. Each one is a markdown file of instructions, an opinion we wrote down for a model to read: use Equatable, name the loading state LoginInProgress, keep events past tense. What we had no way of knowing was whether the model read any of it.
Ask a model to write a login bloc without the plugin and you still get decent code. Sealed classes, pattern matching, sensible names. It is just not how our team does it, and that gap is the only thing a skill exists to close.
We use evals to check whether a skill closes that gap. An eval, short for evaluation, is a test that asks the model to do something and grades what comes back. Ours do two jobs: measure whether a skill is doing anything, and keep measuring it as the model underneath changes.
Both jobs behave differently from unit tests. The same prompt can load a skill on one run and skip it on the next, and the model changes even when nothing in your project does, so the suite has to be rerun and reread.
Why we built a 102-case suite
The VGV AI Flutter Plugin ships fifteen skills: bloc, testing, layered architecture, accessibility, the rest of how we build Flutter apps. VGV Wingspan ships twelve more alongside it, but the suite in this post covers the Flutter plugin. Every skill is our opinion, written down, and our evidence that any of it worked was that it had looked fine when we tried it by hand. We had never measured it.
So we built a suite: 102 cases, open source, running on claude plugin eval. It answers four questions we had been guessing at: whether a skill gets loaded when it should, which of our conventions the model would have followed anyway, whether a change to a skill broke it, and whether any of those answers still holds a month later.
Reading the results was harder than building the suite, because three testing instincts got in the way: same input, same output; a test that passes is a good test; red means bug. Each holds for a function and fails for a model. The second one cost us the most, and most of this post is about it.
When a skill does not trigger
The first question is whether the model ever read the skill.
Someone types “write a LoginBloc for email and password authentication” and hits enter. Fifteen skill descriptions, the short summary at the top of each skill file, are sitting in front of the model. Our bloc skill’s description lists the name LoginBloc, so the model has a fair chance of picking it. Nothing else in the sentence says Equatable, or sealed classes, or VGV. The model reads the descriptions, decides whether to load a skill, and starts typing. That decision is called routing, and the answer does not show whether our skill was loaded. Good Dart comes back either way.
So we measure it instead of assuming. Each check in an eval is called a grader, and it returns pass or fail. Our routing grader reads the run’s trace, the log of every tool the model called, and looks for a call to the Skill tool, which is how Claude Code loads a skill. Across the three runs behind this post, the skill loaded every time. If that rate drops, edit the description, because the description is what the model reads before deciding.

One thing to get right before trusting the number: we measure with the whole plugin loaded, not with the one skill under test. Fifteen descriptions compete for the same decision, and that is the decision a real session puts in front of the model.
What the model actually wrote
The other half of an eval reads the code that came back. Each of those graders asks one question about the output: does the state class extend Equatable, do the event names end in a past-tense verb.
A skill can load and still be ignored here. The model follows the overall shape of the skill and drops the one rule the case was written for. That did not happen in the case below, but it is the reason the content graders sit next to the routing one.
How we score a case
Here is a real prompt from the suite. It reads like a ticket, never names the skill, and never mentions sealed classes or Equatable, because the whole question is whether the model gets there on its own:
Write a LoginBloc for email and password authentication, with a submit
event and a logout event. While the request is in flight the button shows
a spinner. On success the UI needs the signed-in User object; on failure
it needs the error message to display. Nothing else is on screen before
the first attempt. Output Dart code only.
That prompt has no expected output attached to it. It has eight graders, each returning pass or fail, and each carrying a weight:
| Grader | Type | Weight | Checks |
|---|---|---|---|
skill-fired | tool_used | 3 | a Skill call naming bloc in the trace |
extends-equatable | regex | 1 | extends Equatable |
declares-props | regex | 1 | a props getter |
sealed-event-hierarchy | regex | 1 | sealed class LoginEvent |
sealed-state-hierarchy | regex | 1 | sealed class LoginState |
pinned-in-progress-name | regex | 1 | the name LoginInProgress |
final-class-subclasses | llm | 1 | subclasses declared final class |
past-tense-event-names | llm | 1 | event names start with the subject and end in a past-tense verb |
The weights add up to 10. A run scores the weight that passed, divided by 10, and the case score is the mean of its runs.

A case passes when its score clears a threshold. The runner, the claude plugin eval command, defaults to 1.0. We set ours to 0.8, because two graders are scored by a second model, and that judge can mark a correct answer wrong. A threshold of 0.8 lets a fifth of the weight fail without saying which fifth, so we pay for the loosening with weighting. Make one grader heavy enough and the case cannot clear 0.8 without it. In ours that is the routing grader, at weight 3.

The three grader types reach pass or fail differently, and each has its own blind spot.
regex runs over the reply text. It cannot tell a class named LoginSubmitted from the same name sitting in a comment.
tool_used reads the run’s trace and counts calls. It sees what the model did, not what it said it did, which is why routing uses it.
llm hands the reply and a written criterion to a second model. We call these rubric graders. The judge votes three times on every reply, and the votes are in the report, so a 2-1 split is visible instead of hidden inside a pass.
Running every case with and without the plugin
None of that tells us which part of the score we earned. A grader that passes whether or not the plugin is loaded is measuring the model, and from inside a passing run it looks exactly like a grader measuring us. In a unit test, the code under test is what runs. In an eval, the model runs, and it can satisfy a check without any help from the skill.
So we run every case twice, once with the plugin loaded and once with no plugin at all, each in a throwaway home directory. In eval terms that is an ablation, and each half of the pair is an arm. We call the model in the arm without the plugin the bare model. Both arms start from the same fixture, a script that scaffolds an empty Flutter project for the model to write into. The fixture is kept free of anything a skill teaches, because the bare model reads it too.
The gap between the two arm scores, reported as Δ, is the only part of the number our skill can claim. The two-arm comparison is not our idea. The runner runs both arms by default, and --ablation none turns the comparison off. What the runner does not show you is which graders the gap comes from. We had to read that ourselves.

Running the same prompts with the plugin removed is what sent us back to the graders. One pull request touched graders in seven of the fifteen skills. In four of its cases, some graders passed without the plugin, and reworking them roughly doubled those cases’ Δ.
Grader files are short. A regex grader is a frontmatter block:
# graders/extends-equatable.md
---
type: regex
pattern: 'extends Equatable'
---
The routing grader has one line that changes the score:
# graders/skill-fired.md
---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?bloc"'
weight: 3
arm: both
---
By default the runner drops a tool_used grader on the Skill tool from the score in both arms, because it can never pass without the plugin and counting it would inflate Δ. Ours sets arm: both, which forces it back into the number. We do that on purpose: the weight of 3 exists so that a routing miss sinks the case below 0.8, and a grader the runner does not score cannot sink anything. The cost is a Δ that includes routing. Leave that line out and the Δ for the case below reads 0.24 instead of 0.47, which is why we read the per-grader rows and not only the headline.
What the login bloc case showed
Here is the login bloc case, three runs per arm, measured in October 2026. The arm with the plugin scored 1.00 on every run. The arm without it scored 0.50, 0.70 and 0.40, for a mean Δ of 0.47.
| Grader | with | without |
|---|---|---|
skill-fired | 3/3 | 0/3 |
extends-equatable | 3/3 | 1/3 |
declares-props | 3/3 | 1/3 |
final-class-subclasses | 3/3 | 2/3 |
past-tense-event-names | 3/3 | 3/3 |
pinned-in-progress-name | 3/3 | 3/3 |
sealed-event-hierarchy | 3/3 | 3/3 |
sealed-state-hierarchy | 3/3 | 3/3 |
The rows matter more than the Δ. Four of the eight graders pass in both arms on every run: the sealed hierarchies, past-tense event names, and LoginInProgress, which is the loading-state name our skill pins. The bare model produces all four unprompted, so none of them is evidence that the skill did anything. We call those free graders.
What the plugin actually produced was Equatable. Here is the state class from the first run of each arm, same prompt and same fixture, trimmed to the parts that differ:
// with the plugin
sealed class LoginState extends Equatable {
const LoginState();
@override
List<Object> get props => [];
}
final class LoginInitial extends LoginState {}
final class LoginInProgress extends LoginState {}
final class LoginSuccess extends LoginState {
const LoginSuccess({required this.user});
final User user;
@override
List<Object> get props => [user];
}
// without the plugin
@immutable
sealed class LoginState {
const LoginState();
}
final class LoginInitial extends LoginState {
const LoginInitial();
}
final class LoginInProgress extends LoginState {
const LoginInProgress();
}
final class LoginSuccess extends LoginState {
const LoginSuccess(this.user);
final User user;
}
That run reaches for @immutable and never defines equality. @immutable asks the analyzer to check that every field is final. It says nothing about ==, so == stays identity. Two const LoginInProgress() values are the same canonical object and compare equal, but only when the bloc emits them with const. LoginSuccess carries a user that is only known at runtime, so every LoginSuccess(user) is a new object, and LoginSuccess(user) == LoginSuccess(user) is false. Any bloc_test that expects that state by value fails.
The other two runs without the plugin answered differently again. One reached for Equatable unprompted and scored 0.70. The other dropped both Equatable and @immutable and scored 0.40. One prompt gave three different answers about equality, and that variance is the reason the runner defaults to three runs per arm.

Which graders earn their keep
A grader earns its keep when it passes with the plugin and fails without it. Only those graders add to the Δ. Splitting the 0.47 by grader shows which ones they are on the login bloc case:
- Routing, 0.30.
skill-firedcarries weight 3 and fails every run without the plugin, because a skill that is not installed cannot load. It tells you routing worked in three of three runs, not what the skill teaches. - Equatable, 0.13.
extends-equatableanddeclares-propspass in only one run out of three without the plugin. - Final classes, 0.03.
final-class-subclassespasses in two runs out of three without the plugin. - Everything else, 0. The sealed hierarchies, past-tense event names and
LoginInProgresspass in every run of both arms. These are the free graders.

So of the 0.47, the content of the skill accounts for 0.16. The rest is the skill loading at all.
The split also changes over time. The bare model keeps improving while your skill stays the same, so a grader that separates the arms today can turn free next month. The case still scores 1.00 with the plugin when that happens, so the only way to notice is to rerun both arms and compare the rows.
That raises the question of whether to delete free graders. We keep them, by two rules.
A free grader stays. A regex grader makes no model call, so keeping it costs almost nothing. It is also the only thing that would catch a future model dropping the convention, which is the moment the skill has to start teaching it.
Deletion needs a reason other than the score. A genuine duplicate, a restatement of the prompt, or something the format requires anyway, such as testWidgets in a widget test.
Pinning the model in CI, which the runner documentation recommends, does not stop this drift. It only decides when you see it. With a pin, the split stays as it was the day you measured it. When you move the pin, rerun both arms before you trust the scores again.
What the suite still cannot tell us
The judge is uncalibrated. The judge is the second model that scores the rubric graders. Calibrating it means hand-labeling a batch of passes and failures and measuring the judge against them. We have not done that for any of our 158 rubric graders. On this case it costs little: six of the eight graders are regex or tool_used, need no labels, and carry 93% of the Δ. The two that are judged, final-class-subclasses and past-tense-event-names, are the ones to calibrate first.
So we keep the claim precise. The suite measures whether the skill fires and whether the conventions it teaches survive in the output. Whether the output is good code is a different question, and we answer it the way we always have, in review.
Nothing confirms the Dart parses. The runner has no custom-code graders, so the suite grades text, and a snippet can satisfy every regex and still not compile.
Three runs per arm is not a confidence interval. It shows that a result is not a one-off. It does not show how wide the spread is.
Sometimes no grader separates the arms. When that holds after three runs, the case is working correctly and the skill is restating what the model already does. Cut the skill, or find the convention the model does not already follow.
Where to start evaluating your own Claude Code skills
Start with one skill and one question: does it fire?
There are two defensible positions on when to write a grader. Write the evals before the skill and let the baseline define what the skill should do, or write graders only for failures you have already seen. We do the first for routing, which is the only failure you can predict without looking, and the second for everything a skill teaches. Our best graders came from reading output we did not like.
A case is a directory, and nothing registers it. Here is the layout:
evals/<skill>/<case-name>/
├── prompt.md # frontmatter: run limits, tools, tags; body: the prompt
├── case.yaml # optional: schema_version, name, and the fixture hook
├── fixture.sh # scaffolds the project; runs with --scaffold
└── graders/
├── skill-fired.md
└── <one file per grader>
Start with routing. Take your most-used skill and write three cases, each a prompt a user would really send, phrased as a ticket and never naming the skill. Give each case one grader: copy skill-fired.md from above and replace bloc in input_match with your skill’s name. The plugin eval documentation lists every frontmatter field. Run the cases from the plugin root. --case takes a name glob, so one pattern picks up all three:
claude plugin eval . --case '<your-case-prefix>*'
Each case runs three times per arm by default. The terminal prints one row per case with WITH, W/OUT and Δ, and the per-grader split lives in results/<timestamp>/report.html, next to aggregate-result.json. Every table in this post was read out of that report. If the skill fires once in three, edit its description, since that is all the model reads before deciding.
Then read the other arm, before writing any more content graders. Open the report and compare all three runs of each arm. A difference that shows up in every run is what your skill contributes. A difference that changes from run to run is noise.
We run the suite after a merge rather than as a gate, because one red case is a prompt to rerun it three more times and read the NOTES column, not a verdict.
The full suite, the fixture, the mocks, and the README listing the traps we hit are in the VGV AI Flutter Plugin repository. The bloc cases are the closest thing to copy when you write your first one.
Get our insights
Get the next post in your inbox, written by the team that shipped the work.
From the VGV unicorns
Claude Code Skill Evals FAQs
How do you test a Claude Code skill?
claude plugin eval. Run every case in two arms, with the plugin loaded and without, so you measure what the skill adds rather than what the model already knows. The VGV AI Flutter Plugin suite holds 102 cases across 15 skills and is open source. Why do my eval graders stop detecting anything over time?
Why does my Claude Code skill not trigger?
tool_used grader on the Skill tool rather than inferring it from the output. If the skill routes inconsistently, the fix is usually its description, which is the text the model reads when deciding whether to load it. What is ablation in LLM evals?
Should LLM evals block a merge?
NOTES column, not a verdict. What is the difference between an eval and a unit test?
expect because the subject is a distribution, not a function.






