Discovering Silent Pipleine Failures: The LLM Analysis Wasn't Actually Running?!
While reviewing scan results, we noticed something suspicious: skills were being flagged with obvious false positives that the meta-analyzer should have filtered. A deeper investigation revealed a significant gap in our pipeline process.
The Discovery
The skill-scanner tool reports which analyzers were used in its JSON output. We had 467 out of 507 scans showing meta_analyzer in the analyzers_used field. Everything looked fine on the surface.
But when we examined the actual findings, something was off. Skills were being flagged for issues like "missing license file" without any meta-analysis enrichment -- no meta_validated, no meta_confidence, no meta_exploitability fields. The findings had a fallback meta_reviewed: true annotation, but nothing substantive.
We also noticed the scan durations. Real LLM+meta analysis takes 10-50 seconds per skill. But 289 of our "analyzed" scans completed in under 500 milliseconds. That's impossible if an LLM API call actually happened.
Root Cause: Silent API Failure
We ran a test scan with verbose logging and found the culprit:
"Your credit balance is too low to access the Anthropic API"Our API credits had run out mid-batch. But here's the critical issue: the scanner's meta-analyzer catches API exceptions internally, logs a warning, returns all findings unchanged, and still reports meta_analyzer in the analyzers list. From the pipeline's perspective, the scan "succeeded."
This is a reasonable design choice for the scanner -- you don't want a transient API error to crash your entire scan. But it means 540 of our 614 scans never actually received LLM or meta-analysis, despite the metadata suggesting otherwise.
Detection Signals
We built a quality checker to classify scans retroactively. The reliable signals for "real" vs "silent failure" analysis:
| Signal | Real Analysis | Silent Failure |
|---|---|---|
| Scan duration | 10-50 seconds | <1 second |
| Finding metadata | meta_validated, meta_confidence, meta_exploitability | Only meta_reviewed: true |
| LLM findings | Findings with analyzer: "llm" | No LLM-sourced findings |
Before and After: The Difference Real Analysis Makes
Here's a concrete example. The azure-ai-contentsafety-java skill from Microsoft's agent-skills monorepo:
Before (silent failure):
- 6 findings flagged by static/behavioral analyzers
- Scan duration: 0.3 seconds
- No meta-validation applied
- All findings passed through as-is
After (real LLM+meta analysis):
- 0 findings after meta-analysis
- Scan duration: 22.4 seconds
- Meta-analyzer verdict: "6 false positives filtered, 0 new threats detected"
- Final status: SAFE
The meta-analyzer examined each finding in context and determined all six were false alarms -- likely flagged patterns that are normal for an Azure SDK wrapper skill.
Similarly, azure-ai-evaluation-py went from a noisy set of unvalidated findings to 2 confirmed MEDIUM-severity issues with detailed context:
- Each finding now includes
meta_confidence: "HIGH" - Exploitability assessments like "Low - Requires user to follow insecure documentation patterns"
- Clear remediation guidance: "Prioritize DefaultAzureCredential in documentation"
The meta-analyzer didn't just filter -- it enriched the findings with actionable context that helps prioritize remediation.
The Fix: Detection and Re-scanning
We built three improvements into the pipeline:
- Quality checker script (
check_scan_quality.py): Classifies each scan as "real", "silent_failure", or "indeterminate" based on the signals above. Generates a re-scan script for failed scans. - Log retention: Scanner stdout/stderr now saved to
<skill>-scan.logfor post-hoc debugging. - Real-time quality warnings: The pipeline now checks each scan result immediately and logs warnings when meta/LLM analysis appears to have failed silently.
Current Status
We're re-running the 540 affected scans with fresh API credits. Progress as of this writing: 170/540 complete (31%). Zero errors so far. The re-scan is resilient -- it tracks progress and can resume if interrupted.
Early results show the meta-analyzer is filtering 2-6 false positives per skill on average, significantly cleaning up the noise from static analysis alone.
Lessons Learned
- Don't trust success flags blindly. A tool reporting "analyzer X was used" doesn't mean analyzer X succeeded. Verify with secondary signals (duration, output artifacts).
- Silent failures are insidious. The worst bugs are the ones that look like success. Build in quality checks that validate outputs, not just exit codes.
- LLM analysis costs money. Budget for it, monitor it, and have alerts for credit exhaustion. A batch job that silently degrades to static-only analysis is worse than one that fails loudly.
The re-scan will complete over the next several hours. We'll publish updated results as they come in.