The Skill Eval Repo I Didn't Build: 107 SKILL.md Files, 6 Checks, 21 False Positives
I wanted a dedicated repository for evaluating my Claude Code skills and harnesses. Before writing any code, I spent half a day surveying what already exists. The survey killed the repo. What I shipped instead was a single static-lint collector added to a harness I already run, and I think that was the right call. This post is the map I drew, the reasoning behind “don’t build it,” and the numbers from linting 107 SKILL.md files.
Skill evaluation is already three different questions
“Evaluate my skills” sounds like one project. The prior art says it is three, and they barely overlap.
Question 1: is the skill broken? This is the static-lint territory. pulser is the reference example: five structural checks on SKILL.md files (frontmatter syntax, required fields, dangling references), no execution, 40 skills in under 200ms. There is a GitHub Action version that drops into CI. Its author also audited 214 skills and reported 73% silently broken. Hold that number; I will come back to it.
Question 2: does the skill work? This is execution eval. Anthropic’s official skill-creator ships a full pipeline: test prompts accumulate in an evals.json file, subagents run each prompt with and without the skill, and dedicated grader / comparator / analyzer subagents score the results. Its description optimizer even does a 60/40 train-test split. Machine-learning hygiene, imported wholesale into skill authoring.
StackHawk’s eval harness is the operations-flavored version: hold everything constant except the skill, run old and new versions across several real repositories, and grade with a judge that never learns which version it saw. A blind A/B test for skill changes.
On the research side, OpenSkillEval ran 600+ generated tasks against 30 open-source skills. The finding: having a skill available does not mean the agent uses it, and the benefit depends heavily on the model and agent framework underneath. “Install it and it helps” is not a safe assumption.
Question 3: how good is the harness itself? terminal-bench and its Harbor framework swap the agent harness (Claude Code, Codex CLI, and friends) as the test subject while holding tasks constant. The skill is not the unit under test; the whole rig is.
The map told me two things. The questions have completely different cost structures and scoring models. And my imagined “skill eval repo” was trying to be all three at once without noticing.
Why I only took the lint
I already run a private harness called code-health-ops that grades my repositories A-F weekly. It has three design rules: no LLM in the measurement path, no source code stored in the database, absolute thresholds only.
Execution eval collides with all three.
Cost model. A deterministic linter re-measures for free in seconds, so it can ride a daily cron. Execution eval makes every observation a billing event that takes minutes. “Just run it every day” stops being economically sane.
Scoring model. LLM-as-judge is nondeterministic. If the same skill does not get the same score every week, a trend line cannot tell you whether the skill got worse or the judge changed its mood. Absolute thresholds over time series need a boring, repeatable scorer.
Measurement target. Static lint measures the health of files. Execution eval measures agent behavior, like whether a rewritten description raises activation rates. Writing sales numbers on a medical chart makes both harder to read.
So I split it into two layers. The static layer became one more dimension in the existing harness, shipped the same day. The execution layer is deferred to a separate future codebase, and if it ever exists, only its summary grades will be written back. I did not throw execution eval away. I just refused to seat a nondeterministic, metered experiment at the same scorecard as a free, deterministic measurement.
Six checks, 107 files
The collector runs six checks. Five are roughly pulser’s idea; the sixth is specific to how I deploy skills.
| Check | What it catches |
|---|---|
| fm_missing | frontmatter cannot be parsed |
| name_mismatch | frontmatter name differs from directory name |
| desc_over | description exceeds 1024 characters (Claude Code’s limit) |
| oversize | body exceeds 500 lines (official best-practice ceiling) |
| broken_refs | dead relative links in the body |
| broken_symlink | dangling symlinks in skill directories |
The symlink check exists because I keep each skill’s source in one repo and distribute it to consumer repos via symlinks. When a symlink breaks, nothing errors. The skill just quietly vanishes from the agent’s list, and a skill that never fires never logs a failure. Static lint is the only place that failure mode shows up.
Scoring is violations per skill: zero maps to 100 points, 1.0 maps to zero. Across 8 repos and 107 SKILL.md files, the final count was 18 violations: 2 missing frontmatter, 16 oversized. The lowest grade went to my oldest repo with 57 skills (C, 72 points), the next-oldest two scored B and A, and the remaining five got a clean 100. The oldest, largest pile scored worst, which is exactly what intuition predicted. A yardstick earns trust precisely when it agrees with your gut on the cases you already know.
The real lesson was calibrating false positives
The first run did not report 18 violations. It reported 67. I checked every one by hand, and 49 were false positives. I had built a linter with a 73% false-positive rate, which is a strong start for a tool whose entire job is telling the truth.
They fell into two classes.
Class 1: strict YAML parsing flagged 21 working skills as broken. My frontmatter is full of lines like this:
---
name: research
argument-hint: [search query] or [--full <id>]
---A strict YAML parser reads that argument-hint value as a flow sequence and fails the whole document. Twenty-one skills that run happily every single day got labeled “broken frontmatter.” Claude Code itself reads them fine. The correct spec for a linter is not the YAML spec; it is the runtime’s actual tolerance. I replaced the strict parser with a line-based one that is exactly as forgiving as the runtime. General form of the lesson: lint against the implementation your files actually run on, not against the standard. Lint against the standard and you will condemn healthy files that live happily above it.
Class 2: placeholder links flagged as dead references. Documentation lines like [search](url) are illustrations, not links. Requiring a . or / in the target before checking removed those.
Both classes are pinned by regression tests now. The surviving 18 violations were verified by hand: zero false positives.
Which brings me back to pulser’s “73% of 214 skills silently broken.” My own first run said 63% of my skills were in violation; after calibration it was 17%. When a linter’s spec is stricter than the runtime, audit numbers inflate by exactly the gap. I am not claiming that is what happened in that audit. I am saying that when a skill-audit headline shows a big percentage, the first question worth asking is whether the yardstick is stricter than the runtime. In my case, 21 of the “broken” skills were the yardstick’s fault, not the skills’.
So which eval approach is best?
Pick by question, not by sophistication.
- Need a regression check on every skill, every day: static lint. Deterministic, free, seconds. It can never tell you whether a skill helps.
- Need to know if a skill edit worked: a StackHawk-style blind A/B. Treat it as a billed event you run at release time.
- Need to choose which skills to adopt: OpenSkillEval-style with/without comparison, starting from the assumption that installed does not mean used.
- Need to compare harnesses: terminal-bench-style, swap the rig and hold the tasks.
As an interchange format, skill-creator’s evals.json looks like the current favorite. Test prompts are an asset under every approach, so the future execution layer will be built compatible with it.
Closing
Half a day of survey work replaced a whole repository. The half I imagined already existed; the other half fit in one collector inside a harness I was already running. 107 SKILL.md files now get linted by a daily cron, and the repo I never created costs me nothing to maintain.
The single most useful step was sorting the questions before writing code: does this question actually require execution? Asking an LLM a question that lint can answer is like starting a checkup with an MRI when a thermometer would do.
Was this article helpful?