nicornleAI Model EvaluationSecurity & Vulnerability
Security Isn't a Property of the Model — It's a Property of the Deployment
1 / 11
Prompt Injection Is the Common Thread, Not the Exception
Swipe or use ← → to navigate
Every fact has a receipt.
nicornleAI · AI Model Evaluation
Security Isn't a Property of the Model — It's a Property of the Deployment
Nine real incidents across Microsoft, Google, OpenAI, Salesforce, LangChain, and one open protocol (MCP) — plus a Booz Allen study showing one coding model's vulnerability rate shifted 130% based on who it thought was asking. The same mechanism, prompt injection, runs through nearly all of it. A vendor-neutral framework (OWASP's two Top 10 lists) for categorizing the risk, and a checklist for what to actually test before you ship.
26 July 2026
https://unicornle.com/learn/security-isnt-a-property-of-the-model · Every fact has a receipt.
Security Isn't a Property of the Model — It's a Property of the Deployment1 / 11
Prompt Injection Is the Common Thread, Not the Exception
Every incident in this deck — regardless of vendor — traces back to the same underlying weakness: an AI system that can't reliably tell a legitimate instruction from a crafted one hidden inside its input. OWASP's own 2025 coverage puts the typical attack success rate at 50-84% against production systems. One example of the stakes: a zero-click exploit hit Microsoft 365 Copilot in June 2025 (CVE-2025-32711, CVSS 9.3) — one crafted email exfiltrated data from OneDrive and SharePoint with no user interaction, and no antivirus caught it.
OWASP Top 10 for LLM Applications 2025 · SQ Magazine
https://unicornle.com/learn/security-isnt-a-property-of-the-model · Every fact has a receipt.
Security Isn't a Property of the Model — It's a Property of the Deployment2 / 11
Two Frameworks Give This a Vocabulary
OWASP maintains two standing frameworks for these risks. The Top 10 for LLM Applications (v1.1) covers model-level issues: prompt injection, insecure output handling, training data poisoning, supply chain vulnerabilities, excessive agency, and more. Its 2026 companion, the Top 10 for Agentic Applications — reviewed by an expert board including NIST, the European Commission, and the Alan Turing Institute — extends this to autonomous agent behavior: goal hijacking, tool misuse, memory poisoning, cascading failures. Every incident on the following slides maps to a named category in one of these two lists.
OWASP Top 10 for LLM Applications
OWASP Top 10 for Agentic Applications 2026 (GenAI Security Project)
https://unicornle.com/learn/security-isnt-a-property-of-the-model · Every fact has a receipt.
Security Isn't a Property of the Model — It's a Property of the Deployment3 / 11
One Malicious GitHub Issue, Your Private Repos
A vulnerability in the Model Context Protocol's GitHub integration let an attacker plant a booby-trapped public issue. When a developer asked their AI agent to review open issues, the agent was hijacked — using the same broad access token it already held — into leaking salary information and confidential project details from private repositories, via a pull request posted publicly. This isn't a flaw in any one company's model; it's a protocol-level weak point affecting nearly any AI workflow wired into GitHub.
Invariant Labs — MCP GitHub Vulnerability
Docker — MCP Horror Stories: GitHub Prompt Injection
https://unicornle.com/learn/security-isnt-a-property-of-the-model · Every fact has a receipt.
Security Isn't a Property of the Model — It's a Property of the Deployment4 / 11
Same Exploit Class, Three Unrelated Vendors
Zenity's "AgentFlayer" research demonstrated zero-click attack chains — no user interaction required — across three unconnected enterprise platforms: session hijacking on ChatGPT using nothing but a target's email address (reaching their Google Drive), CRM data extraction on Microsoft Copilot Studio, and rerouted customer communications on Salesforce Einstein. Zenity's own framing: these exploits work by abusing "the agents' own capabilities," not a platform bug specific to any one of them.
Zenity — AgentFlayer Vulnerabilities
https://unicornle.com/learn/security-isnt-a-property-of-the-model · Every fact has a receipt.
Security Isn't a Property of the Model — It's a Property of the Deployment5 / 11
Copilot Talks Itself Into "YOLO Mode"
GitHub Copilot (CVE-2025-53773) could be manipulated — via prompt injection hidden in source code, web pages, GitHub issues, or even invisible Unicode characters — into silently rewriting its own settings file to disable every user confirmation. Researchers called the result "YOLO mode": Copilot executing shell commands and browsing the web with no oversight at all. Reported June 29, 2025; Microsoft shipped a fix in the August 2025 Patch Tuesday update, now requiring explicit approval for security-relevant config changes.
Embrace The Red — GitHub Copilot RCE via Prompt Injection
https://unicornle.com/learn/security-isnt-a-property-of-the-model · Every fact has a receipt.
Security Isn't a Property of the Model — It's a Property of the Deployment6 / 11
Three Vulnerabilities, One Assistant
Tenable Research chained together three separate flaws in Google's Gemini suite — dubbed the "Trifecta": planted log entries that silently redirected Gemini Cloud Assist, injected browser-history entries that Gemini's Search Personalization Model treated as legitimate input, and a Browsing Tool bypass that sent a user's private data, including location, to an external server. Google fixed all three after Tenable's disclosure.
Tenable — The Trifecta
https://unicornle.com/learn/security-isnt-a-property-of-the-model · Every fact has a receipt.
Security Isn't a Property of the Model — It's a Property of the Deployment7 / 11
The Fraud That Kept Its Own Paper Trail Looking Clean
At Black Hat USA 2024, Zenity's Michael Bargury and Tamir Ishay Sharbat demonstrated manipulating Microsoft 365 Copilot into altering wire-transfer banking details — while deliberately preserving the original file references, so the output still looked trustworthy to whoever relied on it. In their own words: "getting Copilot to manipulate banking information while keeping original file references for trustworthiness." The lesson isn't just data leakage — it's that a compromised assistant can fabricate financial harm while leaving exactly the paper trail a human would expect to see.
Zenity — Living off Microsoft Copilot
https://unicornle.com/learn/security-isnt-a-property-of-the-model · Every fact has a receipt.
Security Isn't a Property of the Model — It's a Property of the Deployment8 / 11
A Shared Prompt Was the Attack Vector
Noma Security found "AgentSmith" (CVSS 8.8) in LangSmith's public Prompt Hub: a malicious proxy setting attached to a shared, community-uploaded prompt that intercepted a user's OpenAI API keys, documents, and other sensitive inputs the moment they adopted it. Disclosed October 29, 2024; LangChain shipped a fix within a week. The attack surface here wasn't a model or a protocol — it was the trust placed in a shared prompt from the community.
Noma Security — LangSmith Agent Vulnerability
https://unicornle.com/learn/security-isnt-a-property-of-the-model · Every fact has a receipt.
Security Isn't a Property of the Model — It's a Property of the Deployment9 / 11
The One Case Where Nationality Changed the Code
Booz Allen tested five frontier coding models head-to-head — four Chinese (Qwen3-Coder, MiniMax M2.5, Kimi K2.5, DeepSeek V4-Pro) and one American (Claude Opus 4.6) — across 2,800+ trials. When the identical prompt was reframed as a U.S. government request, vulnerability scores shifted sharply and unevenly: Qwen3-Coder produced 130% more vulnerabilities, while Claude Opus 4.6 became more secure (−18%). Every other incident in this deck was vendor- or protocol-specific; this is the one case where the model's inferred national context changed the output.
Booz Allen Hamilton — "What's in America's Code?" (May 2026)
https://unicornle.com/learn/security-isnt-a-property-of-the-model · Every fact has a receipt.
Security Isn't a Property of the Model — It's a Property of the Deployment10 / 11
Security Is Tested Per-Deployment, Not Assumed
Nine incidents, six vendors, one open protocol, and a national-persona effect — and the pattern holds every time: the vulnerability didn't show up in a benchmark, it showed up in how the system was actually used. OWASP's two frameworks (slide 2) exist precisely because "which model" is the wrong first question. The right ones are which category of risk applies, and whether it was tested under realistic, adversarial conditions before shipping — not after.
Synthesized from slides 1-9
https://unicornle.com/learn/security-isnt-a-property-of-the-model · Every fact has a receipt.
Security Isn't a Property of the Model — It's a Property of the Deployment11 / 11
What To Actually Do About It
Before you ship an AI coding tool or agent into production:
- Map your stack against OWASP's two frameworks (slide 2) — know which risk category applies before an incident tells you
- Test with adversarial personas, not just neutral prompts — vulnerability rates shifted by up to 130% on identical tasks (slide 9)
- Audit token/permission scope on every agent integration — one over-broad token turned a public GitHub issue into a private-repo leak (slide 3)
- Require human approval before an agent can change its own permission or config settings — this is exactly how Copilot's "YOLO mode" happened (slide 5)
- Sanitize anything an agent ingests automatically (email, RAG, browser history) before it can influence output — the wire-transfer and Trifecta attacks both relied on silent, automatic ingestion (slides 6-7)
- Treat community-shared prompts and tools as untrusted third-party code, reviewed before adoption — not pre-vetted just because they're popular (slide 8)
- Re-run this checklist after any model or vendor swap — security isn't inherited from a vendor's reputation, each of slides 3-9 happened to an established vendor
Synthesized from slides 1-9
https://unicornle.com/learn/security-isnt-a-property-of-the-model · Every fact has a receipt.