Meta-Analyzer Validation: Reducing False Positives

methodologymeta-analyzerfalse-positives

The meta-analyzer runs as a second-pass review of LLM findings, using a separate LLM call to validate each finding against the original skill source. Every LLM finding in our scan was meta-validated -- here's what that process reveals.

What Meta-Validation Does

For each LLM finding, the meta-analyzer produces:

  • Validated (boolean): whether the finding is confirmed
  • Confidence: HIGH, MEDIUM, or LOW
  • Confidence Reason: explanation of the assessment
  • Exploitability: how easily the issue could be exploited
  • Impact: what happens if it is exploited

Confidence Distribution

Across our scan, meta-confidence scores broke down as:

  • HIGH confidence: ~60% of validated findings. These are well-known anti-patterns the meta-analyzer confirms without hesitation, like API key exposure in setup instructions.
  • MEDIUM confidence: ~40% of validated findings. These represent findings where the meta-analyzer recognizes legitimate functionality behind the flagged pattern.

No findings were given LOW confidence, suggesting the LLM analyzer's initial detection quality is reasonably high -- it isn't generating findings the meta-analyzer outright rejects.

The Downgrade Pattern

The most valuable meta-analyzer behavior is contextual downgrading. Consider the Agent Wallet skill: the LLM flagged a missing wallet.py file as an unauthorized tool use risk (potential tool shadowing). The meta-analyzer validated the finding but assessed exploitability as Low, reasoning it was likely a "documentation error rather than security threat." Without meta-validation, this finding might be triaged as a High-severity security issue requiring immediate action. With it, the team knows it's a documentation cleanup task.

Similarly, the Agent Wallet's social engineering finding -- about incomplete trust model disclosure regarding API key access -- was downgraded with the note: "Clarity issue rather than deception." The meta-analyzer distinguishes between malicious intent and poor documentation, which is exactly the kind of nuance automated scanners traditionally lack.

Exploitability Assessments

The exploitability ratings add practical triage value:

  • "Easy -- automatic on skill use" (video-transcript data exfiltration): The API call happens as part of normal operation. This is expected behavior, not a bug.
  • "Medium -- Requires user to follow insecure instructions" (youtube-playlist API key setup): The risk depends on user behavior, not a code flaw.
  • "Low -- Would require attacker to inject a malicious wallet.py file" (agent-wallet tool shadowing): Exploitation requires a separate attack vector.

Limitations

The meta-analyzer validated 100% of LLM findings in our scan -- it never fully rejected a finding. This could mean the LLM analyzer has high precision, or that the meta-analyzer has a bias toward confirmation. A larger sample with deliberately benign patterns would help calibrate the false-positive rate. Also, meta-validation adds a second LLM call per finding, roughly doubling the cost and latency of LLM-enhanced scanning.