Does claude plugin eval cost money to run?

Yes. Anthropic's docs state that every eval run and every judge grader is a real model call made with your own credentials, counted against your plan's usage limits or your API bill. A suite makes roughly cases times runs agent runs with the plugin, and as many again for the baseline. The summary table prints a COST column with a list-price estimate.

How do I know the plugin is what made a case pass?

Claude Code repeats each case with no plugin loaded and reports two scores, WITH and W/OUT, plus their difference as a delta. A case that scores 1.0 both with and without the plugin means the plugin was not what made it pass. Anthropic calls the two sets of runs the with-arm and the without-arm, and the baseline can be turned off.

Do I have to write the eval cases by hand?

No. Running claude plugin eval init opens an interactive session where Claude reads your plugin, asks what a good result looks like, proposes prompts that should and should not trigger it, designs graders, pilots them once, and writes one case directory per prompt under evals/. You can also write a case yourself: each is a folder with a prompt.md, a case.yaml, or both.

What kinds of graders can a case use?

A grader in claude plugin eval is a pass/fail check on what Claude produced. The docs name a regex over the reply, a check for whether a particular tool was called, and a rubric that a second model judges the reply against. Graders can inspect the final reply, the transcript, or a file Claude created during the run.

What else changed in Claude Code 2.1.269?

Claude Code 2.1.269 also adds /output-style to list and switch output styles in headless and cloud sessions, a diff of files a Bash command changed, and CLAUDE_CODE_WORKFLOW_MAX_CONCURRENT_AGENTS to raise the Workflow tool's concurrency to as much as 256. Fixes cover prompt-cache reuse after an interrupted turn and plugin archives being readable by other local users.

Anthropic · 2026-09-11 · major

Claude Code 2.1.269 — claude plugin eval scores a plugin against a baseline

Claude Code 2.1.269 adds claude plugin eval, which runs a plugin against a suite of test prompts and scores the results. Each case runs three times with the plugin loaded and three times without, so you see what the plugin adds.

GitHub repository card for anthropics/claude-code

A built-in eval runner for Claude Code plugins, with a no-plugin baseline that shows what the plugin actually contributes.

Quick facts

Maker	Anthropic
Version	2.1.269
New command	claude plugin eval
Runs per case	3 with the plugin, 3 without
Pass threshold	1.0 by default (--threshold)
Report	HTML report plus a summary table
Requires	Claude Code v2.1.269 or later

What is it?

claude plugin eval is a new command in Claude Code 2.1.269 that runs a plugin against a suite of test cases and scores the results. Each case is a realistic prompt plus one or more graders. Before this, a plugin author could check that a plugin loaded and that a skill ran when invoked by name, but had no numbers on how often Claude reached for it on natural phrasing.

How does it work?

Each run starts a fresh, isolated, non-interactive session with only the plugin loaded, sends the prompt, and lets Claude work until it finishes or hits the case's turn or time limit. The graders then pass or fail. A run's score is the fraction of graders that passed, weighted if you set weights, and a case's score is the mean across its runs — three by default.

Why does it matter?

The no-plugin baseline is the part that changes how plugin authors work. Repeating every case without the plugin turns a bare score into a delta, so a skill description that Claude ignores shows up as a delta near zero with the tool_used grader failing, instead of looking like a pass. Teams can put the same command in CI and gate a plugin change on the score.

Who is it for?

plugin and skill authors, teams gating plugin changes in CI

Frequently asked questions

Does claude plugin eval cost money to run?: Yes. Anthropic's docs state that every eval run and every judge grader is a real model call made with your own credentials, counted against your plan's usage limits or your API bill. A suite makes roughly cases times runs agent runs with the plugin, and as many again for the baseline. The summary table prints a COST column with a list-price estimate.
How do I know the plugin is what made a case pass?: Claude Code repeats each case with no plugin loaded and reports two scores, WITH and W/OUT, plus their difference as a delta. A case that scores 1.0 both with and without the plugin means the plugin was not what made it pass. Anthropic calls the two sets of runs the with-arm and the without-arm, and the baseline can be turned off.
Do I have to write the eval cases by hand?: No. Running claude plugin eval init opens an interactive session where Claude reads your plugin, asks what a good result looks like, proposes prompts that should and should not trigger it, designs graders, pilots them once, and writes one case directory per prompt under evals/. You can also write a case yourself: each is a folder with a prompt.md, a case.yaml, or both.
What kinds of graders can a case use?: A grader in claude plugin eval is a pass/fail check on what Claude produced. The docs name a regex over the reply, a check for whether a particular tool was called, and a rubric that a second model judges the reply against. Graders can inspect the final reply, the transcript, or a file Claude created during the run.
What else changed in Claude Code 2.1.269?: Claude Code 2.1.269 also adds /output-style to list and switch output styles in headless and cloud sessions, a diff of files a Bash command changed, and CLAUDE_CODE_WORKFLOW_MAX_CONCURRENT_AGENTS to raise the Workflow tool's concurrency to as much as 256. Fixes cover prompt-cache reuse after an interrupted turn and plugin archives being readable by other local users.

Try it

claude plugin eval init