Skip to main content
Back

NVIDIA Open-Sources SkillEvaluator to Test Whether AI Agent Skills Actually Work

NVIDIA open-sourced SkillEvaluator, a three-tier framework that stress-tests agent skills with quality gates, deduplication, and live sandboxed runs.

  • NVIDIA open-sourced SkillEvaluator, an Apache-2.0 framework for evaluating agent skills end to end.
  • Three independent tiers: static validation, embedding-based deduplication, and live sandboxed agent runs.
  • Tier 3 measures Skill Lift, the behavioral delta between agent runs with and without the skill.
  • Scores on five dimensions plus pass@k reliability, with CLI, JSON, HTML, and Markdown reports.
  • Ships GitHub Actions and GitLab CI templates for merge-gating skills on pull requests.
  • Works keyless for quality checks; LLM rubric and Tier 3 need a provider and Docker sandbox.

NVIDIA has open-sourced SkillEvaluator, an Apache 2.0 framework for checking whether AI agent skills are safe, distinct, and effective. It combines static validation, similarity analysis, and sandboxed agent runs so developers can evaluate a skill before adding it to a catalog or shipping it to users.

Agent skills are folders containing Markdown instructions, metadata, scripts, and reference files that extend coding agents such as Claude Code and Codex. As these packages multiply, maintainers need automated checks for insecure code, exposed credentials, duplicated guidance, and instructions that fail to improve task results. SkillEvaluator follows the community Agent Skills specification and forms part of NVIDIA’s broader Verified Skills pipeline.

Three gates answer different questions

Tier Question Checks
1: Validation Is the skill safe and well formed? Schema, quality, security, personally identifiable information, licensing, and script checks, plus optional LLM-based review.
2: Deduplication Does the skill repeat existing guidance? Embedding similarity within a skill and across a collection or catalog.
3: Live evaluation Does the skill improve agent performance? Sandboxed task runs with and without the skill, followed by comparative scoring.

Each tier has its own entry point and can run without the preceding stages. Tier 1 uses SkillSpector for specialized security analysis, alongside Gitleaks and optional Semgrep scans. Tier 3 executes agents through Harbor, an open-source evaluation framework that uses Docker as its default isolation layer.

Paired runs reveal the skill’s effect

Tier 3 evaluates behavior rather than relying on documentation quality. It runs the same tasks with the skill enabled and disabled, then compares the results across three types of measurements:

  • Dimension scores: Security, Correctness, Discoverability, Effectiveness, and Efficiency appear at the top of standard grading reports.
  • Skill Lift: The measured difference between runs with and without the skill.
  • pass@k: The share of task sets in which at least one of k attempts succeeds, reported separately from the dimension scores.

A skill can pass static checks yet leave agent behavior unchanged or make results worse. The paired evaluation exposes that outcome and gives maintainers evidence for accepting, revising, or rejecting the package.

Run the offline check first

After installing uv, developers can clone the repository and run the bundled text-analyzer example without an API key, GPU, or Docker:

git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator
uv run --python 3.13 skillevaluator quality-check \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-quality

The example receives 84.5 out of 100, a B grade, and exits successfully because the default passing score is 70. It writes the JSON report to reports/first-quality/skillevaluator-quality.json.

Optional rubric reviews require an LLM endpoint. Eligible NVIDIA Developer Program members can use NVIDIA-hosted endpoints for prototyping, subject to access terms and rate limits. The framework also accepts OpenAI-compatible services, including locally hosted models.

Pull requests become the control point

SkillEvaluator includes GitHub Actions and GitLab CI templates for evaluating skills during review. Its guided setup command generates the workflow configuration:

pip install --upgrade skillevaluator
skillevaluator ci init

The combined pipeline applies blocking behavior by tier:

  • Tier 1 blocks validation when required checks fail.
  • Tier 2 blocks detected duplication unless the pipeline uses --no-block-on-dedup.
  • Tier 3 reports results without blocking unless --block-on-agent-eval is enabled.

These defaults keep inexpensive static checks mandatory while leaving model calls and managed sandbox runs optional. Live evaluations may incur model or infrastructure charges, especially when a dataset contains many tasks or repeated attempts.

Experimental software needs stronger datasets

NVIDIA labels the framework experimental and community-supported. Support is provided on a best-effort basis, with no service-level agreement or NVIDIA enterprise support entitlement.

Tier 3 results also depend on the evaluation set. When no hand-authored dataset exists, the framework reuses available evaluation sources or creates one starter case, which provides limited coverage. Developers seeking representative Skill Lift measurements should generate and review a broader dataset with create-eval-dataset --full and define negative cases explicitly in EVAL.md.

Catalog maintainers get measurable gates

SkillEvaluator targets teams that maintain public skill catalogs, internal agent extensions, or marketplaces. Its checks can reject malformed packages, exposed secrets, duplicated instructions, and skills that consume context without improving task performance.

NVIDIA’s developer post and companion research paper describe a code-review model for agent skills: automated gates for safety and duplication, paired behavioral tests for effectiveness, and reports that make changes comparable across revisions. SkillEvaluator packages that process into a reproducible command-line and CI workflow.