azure-ai-evaluation-py
Verdict safe, but 2 findings (2 medium), mainly data exfiltration. Most recent scan 2026-02-05.
The azure-ai-evaluation-py skill is a legitimate Azure SDK wrapper for evaluating AI applications. The skill itself is not malicious, but it has MEDIUM severity security concerns around credential management practices. The primary issues are: (1) instructions that emphasize API key usage over more secure credential methods, (2) connection string exposure risks without security guidance, and (3) missing package integrity verification. The skill correctly supports DefaultAzureCredential for secure authentication, but the documentation doesn't prioritize this approach. No prompt injection, data exfiltration, or malicious code patterns were detected. The referenced files are internal documentation and examples, which is normal for SDK skills. Overall, this is a safe skill that needs improved security guidance in its documentation.
Scanner's overall assessment (LLM analyzer)
Second scanner: NVIDIA SkillSpector
The two scanners disagree: Cisco found nothing and SkillSpector says do not install. One of them has a blind spot here; read the skill before trusting either. How the two scanners compare across the whole corpus: Two scanners, one corpus.
What this skill does
Azure AI Evaluation SDK for Python. Use for evaluating generative AI applications with quality, safety, and custom evaluators. Triggers: "azure-ai-evaluation", "evaluators", "GroundednessEvaluator", "evaluate", "AI quality metrics".
- Package
- 5 files, 45 KB
- SKILL.md
- 657 words, 12+ sections
- Code
- Python (1)
SKILL.md covers: Azure AI Evaluation SDK for Python · Installation · With remote evaluation support · Environment Variables · For AI-assisted evaluators · For Foundry project integration · Built-in Evaluators · Quality Evaluators (AI-Assisted) · Initialize with Azure OpenAI model config · Quality Evaluators (NLP-based) · Safety Evaluators · Single Row Evaluation
Findings
MEDIUM Hardcoded API Key in Environment Variable Instructions Data Exfiltrationllm
The skill instructions explicitly guide users to set AZURE_OPENAI_API_KEY as an environment variable and use it directly in code. While environment variables are better than hardcoding in source, the instructions don't emphasize secure credential management practices like Azure Key Vault or managed identities. The script does support DefaultAzureCredential as an alternative, but the primary examples use API keys.
SKILL.md
```bash
AZURE_OPENAI_API_KEY=<your-api-key>
```
and
```python
model_config = {
"api_key": os.environ["AZURE_OPENAI_API_KEY"]
}
```MEDIUM Connection String Exposure Risk Data Exfiltrationllm
The skill requires AIPROJECT_CONNECTION_STRING environment variable for Foundry integration. Connection strings often contain sensitive credentials. The instructions don't provide guidance on secure storage or rotation of these connection strings.
SKILL.md:18
```bash
AIPROJECT_CONNECTION_STRING=<your-connection-string>
```What is in the package
| File | Kind | Size |
|---|---|---|
SKILL.md | doc | 6.9 KB |
references/acceptance-criteria.md | doc | 6.5 KB |
references/built-in-evaluators.md | doc | 11 KB |
references/custom-evaluators.md | doc | 12 KB |
scripts/run_batch_evaluation.py | script | 9.2 KB |
External hosts referenced
github.com
Hostnames that appear in the skill's files. Referencing a host is not a finding; compare against the findings above and the skill's stated purpose.
Environment variables referenced
AIPROJECT_CONNECTION_STRING AZURE_OPENAI_API_KEY AZURE_OPENAI_DEPLOYMENT AZURE_OPENAI_ENDPOINT
Package contents recorded 2026-10-22 at commit cfbcbbdc679c. The scan above ran against the version current at scan time; the two can differ if the skill has since been updated.
What we checked
- Static analyzer. Pattern rules over SKILL.md and bundled files: known-bad commands, exfiltration URLs, secrets, obfuscation.
- Behavioral analyzer. Checks what the skill asks the agent to do against what its description claims it does.
- Llm analyzer. A language model reads the skill and reports intent-level issues the rules cannot express.
- Trigger analyzer. Looks for instructions that fire on conditions the user did not ask for.
- Meta analyzer. A second model pass that validates, deduplicates and re-scores the other analyzers' findings.
What we did not check
- Runtime behaviour. The skill was not executed; findings come from reading its files.
- Anything fetched at install or run time, including remote scripts, package dependencies, and URLs the skill tells the agent to open.
- Content the analyzers cannot read: images, binaries, and some non-English text.
- Known scanner bypass classes. See the methodology and limitations page.
Sources
- Repository: https://github.com/microsoft/agent-skills
- Raw scan result: github-skills-azure-ai-evaluation-py-scan.json
- Badge for your README:
- Request a re-scan