First Complete LLM+Meta Scan: 585 Skills Analyzed
After several iterations, API credit exhaustion, and pipeline fixes, we've completed our first clean run: 585 AI agent skills scanned with full static, behavioral, LLM, and meta-analysis. This post shares what we found.
The Numbers
| Metric | Value |
|---|---|
| Skills Scanned | 585 |
| Skills Safe | 291 (50%) |
| Skills with Findings | 294 (50%) |
| Total Findings | 1,281 |
Almost exactly half of the skills in our sample passed with zero findings. The other half triggered at least one security concern.
Insight #1: LLM Analysis Catches What Static Analysis Misses
The most striking result from this scan is the difference between what static analysis finds versus what LLM analysis finds.
| Analyzer | Findings | % of Total |
|---|---|---|
| LLM | 845 | 63% |
| Static | 462 | 35% |
| Behavioral | 31 | 2% |
But the aggregate numbers hide the real story. For certain vulnerability categories, LLM analysis doesn't just find more — it finds almost everything:
| Category | LLM Findings | Static Findings | LLM Advantage |
|---|---|---|---|
| Social Engineering | 166 | 2 | 83x |
| Resource Abuse | 88 | 0 | Static blind |
| Unauthorized Tool Use | 115 | 17 | 7x |
| Data Exfiltration | 320 | 62 | 5x |
Social engineering attacks are nearly invisible to static analysis. These are patterns where a skill manipulates the user or agent through language — convincing it to bypass restrictions, approve dangerous actions, or reveal sensitive information. Static pattern matching can't detect intent; LLM analysis can.
Similarly, resource abuse (skills that consume excessive compute, make unbounded API calls, or spawn runaway processes) was detected exclusively by the LLM analyzer. Static rules have no concept of "excessive" — that requires understanding context and expected behavior.
This validates the multi-analyzer approach: static analysis provides fast, reliable detection of known patterns, while LLM analysis catches the semantic and intent-based issues that would otherwise slip through.
Insight #2: Data Exfiltration is the Dominant Risk
Of the 1,281 findings, data exfiltration risks make up the single largest category by a wide margin.
| Category | Findings | % of Total |
|---|---|---|
| Data Exfiltration | 413 | 31% |
| Command Injection | 207 | 15% |
| Social Engineering | 168 | 13% |
| Transitive Trust Abuse | 151 | 11% |
| Unauthorized Tool Use | 132 | 10% |
Data exfiltration findings include:
- Skills that send user data to external APIs without disclosure
- Hardcoded API credentials that could be harvested
- Overly broad file access patterns that read beyond their stated scope
- Skills that cache or log sensitive information to external services
This isn't surprising given the nature of agent skills. Skills are designed to extend an agent's capabilities — which often means accessing files, APIs, and user data. The line between "useful integration" and "data exfiltration risk" can be thin.
55 of these data exfiltration findings are rated CRITICAL, meaning they represent immediate, exploitable risks rather than theoretical concerns.
What About Critical Findings?
Of the 204 CRITICAL findings, command injection dominates:
| Category | Critical Findings |
|---|---|
| Command Injection | 114 (56%) |
| Data Exfiltration | 55 (27%) |
| Hardcoded Secrets | 21 (10%) |
| Social Engineering | 8 (4%) |
| Prompt Injection | 4 (2%) |
Command injection remains the most severe risk in agent skills. These are patterns where user input flows into shell commands, subprocess calls, or system operations without proper sanitization. In an agent context — where the "user input" might come from another LLM or automated system — these vulnerabilities become even more dangerous.
Methodology Note
This scan cost approximately $120 in Anthropic API credits for the LLM and meta-analysis passes across 585 skills (using claude-sonnet-4-5). The full pipeline runs:
- Static Analysis — YARA rules, regex patterns, AST parsing
- Behavioral Analysis — Dynamic execution patterns, file/network access
- LLM Analysis — Semantic understanding of code intent and risk
- Meta Analysis — Cross-validates findings, filters false positives, enriches context
The meta-analyzer is particularly valuable for reducing noise. In earlier runs, we saw skills like ClawDefender (a security tool) flagged with 37 findings — all false positives from detecting attack signatures in defensive code. The meta-analyzer correctly filtered all 37.
What's Next
585 skills is a meaningful sample, but the broader ecosystem contains 40,000+ skills. This scan proves the methodology works end-to-end. Future work could:
- Scale to the full ecosystem
- Track changes over time as skills are updated
- Identify publishers with repeated security issues
- Build automated gates for skill registries
For now, the data is available for exploration at skillscan.appsechq.com.
This scan was performed using Cisco's skill-scanner tool with LLM analysis powered by Claude.