SkillGuard

Important

This project is still under active development. Contributions via Issues and PRs are welcome.

SkillGuard is a malicious-skill detector for the agentic-skill ecosystem (Claude Code skills, Codex skills, and similar bundles).

The pipeline runs in three layers:

1. Rule-based static scan. A rule engine extracts behavioral features (network egress, command execution, credential handling, governance gaps, metadata inconsistencies, identity persistence, ...), routes them through per-domain detectors, and lets a verdicter produce an initial verdict (benign / suspicious / malicious) and an AST category. Tuned for high recall — conservative when in doubt.
2. Agent-based filtering. An optional, read-only Codex review re-judges the skills that the rule engine flagged, runs reviews in parallel, and may only downgrade severity. It is designed to cut false positives from the rule-based layer.
3. Dynamic verification (gated). Skills that survive the first two layers as suspicious or malicious are executed inside a Docker sandbox with canary files, an egress gateway, and declarative YAML behavior rules. The run produces a normalized evidence bundle that is merged with the static verdict into a single unified report.

Repository Layout

src/skillguard/
  cli/                  CLI, config, input discovery, pipeline orchestration
  static_scanner/
    default/            first-party static engine (features, detectors, verdicter, docs)
    default_scanner.py  pipeline adapter from the static engine to a pipeline result
  dynamic_executor/     Docker sandbox, canaries, egress gateway, evidence, YAML rules
  benchmark/            manifest builder + static evaluation tooling (skillguard-benchmark)
  reporting/            unified JSON report and terminal summary

benchmarks/final/
  skills/               normalized benchmark skill directories
  labels/               per-sample ground-truth labels with provenance
  manifest.jsonl        one row per benchmark sample
  summary.json          aggregate counts and label mapping
  review/               human-friendly per-sample review pages

output/
  benchmarks/<run-id>/  predictions.jsonl + metrics.json for each scanner run
  report_*.json         unified pipeline run reports

scripts/                helper scripts (label tooling, run drivers, packaging)
tests/                  pytest suite

Quickstart

Install

The project ships a pyproject.toml and is happiest in a fresh virtualenv. With uv:

uv venv --python 3.11
uv pip install -e ".[dev]"

The install exposes two console scripts: skillguard (the scanner) and skillguard-benchmark (the eval driver).

Run a static scan (rule-based only)

The default pipeline is static and uses the rule-based engine. Point it at either a directory of skill .zip archives or a single remote URL:

# Batch: scan every .zip under a directory
skillguard --input-dir ./benchmarks/final/skills --pipeline static

# One-off: scan a single remote skill
skillguard --url https://example.com/skill.zip --pipeline static

Each run writes a unified JSON report to ./output/report_<timestamp>.json and prints a per-skill summary to the terminal. Useful flags:

Flag	Default	Notes
--output PATH	./output	Where the unified report is written.
--workers N	1	Scan N skills in parallel.
--scanner {default,dummy}	default	Static engine selection.
--log-level {DEBUG,INFO,WARNING,ERROR}	INFO	Verbosity.
--keep-temp	off	Keep the extracted skill workspace for inspection.

Toggle Agent-based false-positive filtering

An agent-based (currently Codex) false-positive filter layer. It is off by default — pass --static-codex-review-config to turn it on:

# OFF (default): rule-based only
skillguard --input-dir ./benchmarks/final/skills

# ON: add a config file to enable the reviewer
skillguard \
  --input-dir ./benchmarks/final/skills \
  --static-codex-review-config ./static_codex_review.example.yaml

Before enabling it, set up the local codex CLI under ~/.codex/:

~/.codex/auth.json

{"OPENAI_API_KEY": "sk-..."}

~/.codex/config.toml

[profiles.default]
model = "gpt-5-codex"
model_provider = "openai"

The profile name (default above) must match the profile: field in the SkillGuard YAML below.

Minimal static_codex_review.example.yaml:

schema_version: 1
profile: default               # required: Codex CLI profile name
codex_bin: codex               # path to the local codex binary
model: ""                      # optional override; "" keeps the profile default
timeout_seconds: 90

# Bound how much skill context is sent per review.
max_files: 8
max_file_chars: 2400
max_total_chars: 12000

# Only re-judge skills the rule engine already flagged.
review_on_verdicts:
  - suspicious
  - malicious
downgrade_only: true           # only lower severity, never raise it

Notes:

• The filter shells out to the local codex CLI binary (codex exec -p <profile>), not an HTTP API. Authentication is handled by the local codex profile itself.
• The same flag is also accepted by skillguard-benchmark static for evaluating the "rule-based + Codex filtering" configuration on the benchmark corpus.

Run the gated static + dynamic pipeline

The gated pipeline only spins up the Docker sandbox for skills the static stage flagged as non-benign:

skillguard \
  --input-dir ./benchmarks/final/skills \
  --pipeline gated \
  --launcher claude-code \
  --gateway-secrets ./secrets/gateway.env

This requires a working Docker daemon plus a Claude Code runtime image. Pass --no-auto-build-images together with --image / --gateway-image to use pre-built images, and --dynamic-timeout-seconds / --memory-limit to bound each sandboxed run.

Drive the whole run from a YAML config

The scan above can be re-expressed as a single file and a single --config flag:

skillguard --config ./skillguard.yaml

Example skillguard.yaml:

schema_version: 1

# Input (exactly one of input_dir / url)
input_dir: ./benchmarks/final/skills

# Pipeline + runtime
pipeline: static
output: ./output
workers: 4
log_level: INFO
keep_temp: false

# Static scanner + optional Codex false-positive review
scanner: default
static_codex_review_config: ./static_codex_review.example.yaml

# Dynamic executor (only used when pipeline is gated / dynamic / full)
launcher:
  - claude-code
gateway_secrets: ./secrets/gateway.env
memory_limit: 1g
network_mode: none

Some tips:

• Any CLI flag passed alongside --config overrides the same key in the YAML, so you can keep a stable config and tweak one knob per run: skillguard --config ./skillguard.yaml --pipeline gated --workers 8.
• Path-valued keys (input_dir, output, static_codex_review_config, gateway_secrets, ...) are resolved relative to the config file's directory, not the current working directory.
• static_codex_review_config and static_llm_config accept either a path string or an inline mapping, so you can collapse everything into a single file when that is cleaner.
• See skillguard.example.yaml for the full field list and defaults.

Benchmark

The repository ships a normalized benchmark under benchmarks/final (sampled from a public skill market and hand-labeled by us) with three labels: malicious, suspicious, benign.

# Rule-based only
skillguard-benchmark static \
  --manifest benchmarks/final/manifest.jsonl \
  --output output/benchmarks/static-default \
  --scanner default

# Rule-based + Codex false-positive review
skillguard-benchmark static \
  --manifest benchmarks/final/manifest.jsonl \
  --output output/benchmarks/static-default-codex \
  --scanner default \
  --static-codex-review-config ./static_codex_review.example.yaml

Each output directory gets predictions.jsonl (one row per sample) and metrics.json (aggregate counts), which is what the numbers in the Performance section below are computed from.

Performance of SkillGuard

Benchmark dataset

The corpus under benchmarks/final/ is 560 skill bundles harvested from public skill market and then hand-labeled by us. Each archive is extracted, reviewed, and assigned exactly one of three unified labels:

• benign (157) — no risk signal worth flagging.
• suspicious (344) — non-trivial risk signals (network egress, command execution, credential touch, governance gaps, ...) that warrant dynamic validation but cannot be conclusively called malicious from static evidence alone.
• malicious (59) — review-confirmed malicious intent (e.g. token exfiltration, hidden persistence, sandbox escape).

Setup

Two static configurations are compared on the corpus above:

• Rule-based — the default static scanner only.
• Rule-based + Codex — the default static scanner plus the optional Codex-based false-positive filtering (with model GPT-5.4).

The two charts below also include two external baselines, Skill Scanner and SkillSpector, evaluated on the same 560-sample corpus.

3-class: benign / suspicious / malicious

Method	Macro Recall	Accuracy	Macro F1	Macro Precision	Weighted F1	Correct
Rule-based	0.7000	77.68 %	0.7000	0.8597	0.7327	435/560
Rule-based + Codex	0.8246	85.54 %	0.8413	0.8644	0.8520	479/560

Adding the Codex filter lifts three-class Macro F1 by +14.13 pp (0.7000 → 0.8413) and three-class accuracy by +7.86 pp.

Binary: benign vs. risk (suspicious ∪ malicious)

The binary view reflects the gate decision used by the gated pipeline — any non-benign verdict is forwarded to dynamic validation.

Method	Risk Recall	Accuracy	Risk F1	Risk Precision	Macro F1
Rule-based	1.0000	80.36 %	0.8825	0.7897	0.6423
Rule-based + Codex	0.9516	88.04 %	0.9215	0.8932	0.8353

The Codex filter trades a small amount of risk recall (1.000 → 0.952) for a large gain in risk precision (0.790 → 0.893), raising risk-class F1 to 0.9215 and balanced Macro F1 by +19.29 pp.

📜 License

This project is licensed under the GNU General Public License v3.0 (GPLv3).