30 Real-World AI Security Risks
PlayCISO's AI Security Risk Register catalogs 30 incident-grounded AI and LLM security risks, each scored by likelihood and impact with recommended controls. It spans 5 Critical, 20 High and 5 Medium risks across six categories — Data & Privacy, Security, Governance, Operational, Legal & Compliance, and Supply Chain — and every risk is grounded in a documented real-world incident, from the Samsung source-code leak and the $25M deepfake wire fraud to malicious models on HuggingFace and EU AI Act non-compliance.
Most “AI risk” lists are generic checklists. This one is different: all 30 risks below are drawn from PlayCISO’s AI Risk Register, and each is anchored to a specific, documented incident — a company that banned ChatGPT after a source-code leak, a finance worker who wired $25M to a deepfake, lawsuits, regulator fines, and peer-reviewed research. They are scored with a standard likelihood × impact model and paired with the controls that actually mitigate them.
Use it to brief a board, seed a GRC risk register, or map your exposure against the EU AI Act, NIST AI RMF and the OWASP LLM / Agentic AI Top 10. The full, filterable, downloadable version lives in the interactive AI Risk Register tool.
The register — all 30 risks
| ID | Risk | Category | Severity | L × I | Real-world incident | Recommended controls |
|---|---|---|---|---|---|---|
| AIR-001 | Confidential data leakage via LLM promptsPrompt Scan → | Data & Privacy | Critical | 5×5=25 | Samsung engineers pasted proprietary source code into ChatGPT (2023), leading to a company-wide ban. | DLP on AI endpoints; approved-tool-only policy; prompt logging |
| AIR-003 | Prompt injection in LLM-powered applicationsPrompt Scan → | Security | Critical | 5×4=20 | Demonstrated across GPT-4, Claude, Gemini — attackers override system prompts to exfiltrate data or alter behavior. | Input sanitization; privilege separation; output filtering; red-team testing |
| AIR-007 | Malicious models in public registriesModel Risk Scanner → | Supply Chain | Critical | 4×5=20 | JFrog researchers found 100+ malicious models on HuggingFace with hidden code execution via pickle deserialization (2024). | Model provenance verification; scan before load; pin model hashes; internal model registry |
| AIR-015 | Insecure AI agent tool useAgent Governance Plane → | Security | Critical | 4×5=20 | OWASP Agentic AI Top 10 (2025) — agents granted broad tool access executing unintended file, DB or API operations. | Least-privilege tool grants; human-in-the-loop for destructive actions; tool-call audit logging; sandboxed execution |
| AIR-002 | Deepfake-enabled CEO fraud | Security | Critical | 3×5=15 | Attackers used AI-cloned voice and video of a CFO to trick an employee into wiring US $25M in Hong Kong (2024). | Out-of-band verification for wire transfers; deepfake awareness training; callback procedures |
| AIR-004 | Shadow AI — unauthorized tool adoption | Governance | High | 5×4=20 | Salesforce survey (2024): 55% of generative-AI users at work had not received employer approval. | AI acceptable-use policy; network-level visibility; approved tool catalogue |
| AIR-005 | AI hallucination in high-stakes decisions | Operational | High | 5×4=20 | Lawyer Steven Schwartz cited six fabricated case citations generated by ChatGPT in a federal filing — Mata v. Avianca (2023). | Mandatory human review for any AI-assisted output used in decisions; citation verification workflows |
| AIR-008 | AI-powered phishing at scale | Security | High | 5×4=20 | WormGPT and FraudGPT (2023) — purpose-built LLMs for generating convincing phishing emails, lowering attacker skill barrier. | AI-aware email gateway; phishing simulation training; DMARC/DKIM/SPF enforcement |
| AIR-009 | Jailbreak attacks bypassing safety guardrailsModel Audit → | Security | High | 5×4=20 | Many-shot jailbreaking (Anthropic, 2024), DAN prompts, and crescendo attacks consistently bypass model alignment. | Layered guardrails (input + output); behavioral monitoring; model red-teaming; rate limiting |
| AIR-030 | EU AI Act non-compliance — failure to classify AI system riskAI Threat Model Builder → | Legal & Compliance | High | 4×5=20 | EU AI Act (Aug 2025 enforcement for prohibited practices, Aug 2026 for high-risk): organizations without a risk classification face fines up to €35M or 7% of global revenue. | AI system inventory with risk classification; conformity assessment for high-risk; prohibited-use screening; appointed AI officer |
| AIR-010 | Biased AI in hiring and HR decisions | Legal & Compliance | High | 4×4=16 | Amazon scrapped an AI recruiting tool that systematically downgraded female applicants (disclosed 2018, widely cited since). | Bias testing before deployment; demographic parity checks; human override; NYC Local Law 144 compliance |
| AIR-011 | AI API key and credential exposure | Data & Privacy | High | 4×4=16 | GitGuardian (2024): 12.8M secrets detected in public GitHub repos — OpenAI, Anthropic and cloud-AI keys among the fastest-growing categories. | Secret scanning in CI/CD; key rotation automation; vault-based secret management; least-privilege API scopes |
| AIR-013 | AI copyright and IP infringement | Legal & Compliance | High | 4×4=16 | NYT v. OpenAI (2023), Getty v. Stability AI (2023) — lawsuits alleging training on copyrighted works without license. | Training data audit; license clearance; opt-out compliance; output watermarking |
| AIR-014 | GDPR / data-protection violations from AI processing | Legal & Compliance | High | 4×4=16 | Italy's Garante temporarily banned ChatGPT (March 2023) for GDPR violations — unlawful processing, no age verification, no legal basis. | DPIA before deployment; lawful basis documentation; data-subject rights workflows; DPO involvement in AI projects |
| AIR-021 | Absent AI model governance (no model cards, no inventory)AI BOM → | Governance | High | 4×4=16 | EU AI Act (effective 2024–2026) requires AI system registration, risk classification and documentation — organizations without an inventory cannot comply. | AI model inventory (AIBOM); model cards for every production model; lifecycle governance policy |
| AIR-023 | Indirect prompt injection via retrieved documentsPrompt Scan → | Security | High | 4×4=16 | Demonstrated in RAG systems — adversary plants instructions in a document the AI retrieves, hijacking the response (Greshake et al., 2023; live exploits in Bing Chat). | Data/instruction separation; retrieved-content sandboxing; output validation; RAG content filtering |
| AIR-026 | AI-enhanced social engineering (voice cloning, video deepfakes) | Security | High | 4×4=16 | Multiple reported incidents of AI voice clones used in vishing (2023–2024); deepfake video calls used to impersonate executives. | Multi-factor identity verification; verbal code words for sensitive requests; deepfake detection tooling; awareness training |
| AIR-029 | PII and credentials leaked in training datasets | Data & Privacy | High | 4×4=16 | Carlini et al. (2021, 2023): GPT-2 and ChatGPT memorize and regurgitate PII, phone numbers and email addresses from training data. | PII scrubbing in training pipelines; differential privacy; output filtering for PII patterns; data retention policy |
| AIR-006 | Training data poisoningModel Risk Scanner → | Security | High | 3×5=15 | Nightshade and Glaze tools (2024) demonstrated practical poisoning of image-model training data at scale. | Data provenance tracking; input validation on training pipelines; anomaly detection on training metrics |
| AIR-016 | MCP server tool-poisoning attackMCP Scan → | Supply Chain | High | 3×5=15 | Invariant Labs disclosed tool-poisoning attacks via MCP servers — injecting hidden instructions into tool descriptions to exfiltrate data (2025). | MCP config audit; description-change detection across scans; tool allowlisting; server hash pinning |
| AIR-017 | AI model theft via API extraction | Security | High | 3×5=15 | Researchers demonstrated near-perfect extraction of production models (including GPT-like transformers) through systematic API querying (multiple papers, 2020–2024). | Query rate limiting; output perturbation; watermarking; API access monitoring; differential privacy |
| AIR-018 | Excessive AI agent autonomyAgent Governance Plane → | Governance | High | 3×5=15 | ChaosGPT experiment (2023) — autonomous agent attempted to acquire resources and recruit other AI agents beyond operator intent. | Bounded autonomy policies; kill switches; action logging; human approval for high-impact steps |
| AIR-019 | AI-generated code introducing vulnerabilitiesNPM Scanner → | Operational | High | 5×3=15 | Stanford study (2023): developers using AI code assistants produced significantly less secure code and were more confident it was secure. | SAST/DAST on AI-generated code; mandatory code review; security-focused AI coding guidelines |
| AIR-025 | Deployment of abliterated / uncensored modelsAbliterated Model Risk → | Governance | High | 3×5=15 | Uncensored Llama, Dolphin and WizardLM variants circulate on HuggingFace — refusal training removed, producing harmful content on request. | Model provenance verification; abliteration detection; restricted deployment environments; output monitoring |
| AIR-012 | Adversarial evasion of ML-based security tools | Security | High | 3×4=12 | Academic research and real-world bypasses of ML-based malware classifiers, spam filters, and fraud-detection models using adversarial perturbations. | Ensemble detection; adversarial training; signature + ML hybrid approach; regular model retraining |
| AIR-020 | Third-party AI API single point of failure | Operational | Medium | 4×4=16 | OpenAI major outage (Nov 2023, 2+ hours) — every product built on its API went down simultaneously. | Multi-provider fallback; local model option; graceful degradation; SLA monitoring |
| AIR-022 | AI-enabled mass surveillance overreach | Legal & Compliance | Medium | 3×5=15 | Clearview AI fined €20M by French CNIL (2022) and ordered to delete data; similar actions by UK ICO, Italian Garante and Australian OAIC. | Privacy impact assessment; facial recognition use policy; consent mechanisms; jurisdictional compliance review |
| AIR-027 | Automation bias — over-reliance on AI without human review | Operational | Medium | 5×3=15 | NTSB investigations cite automation complacency in aviation incidents; same pattern emerging in AI-assisted medical diagnosis and content moderation. | Mandatory human-in-the-loop for critical decisions; confidence-threshold alerts; UX that surfaces uncertainty |
| AIR-024 | Model collapse from synthetic training data | Operational | Medium | 3×4=12 | Shumailov et al. (Nature, 2024): models trained on AI-generated text progressively degrade — "model collapse" — losing distributional tails. | Training data provenance; synthetic-data labeling; human-generated data preservation; output quality monitoring |
| AIR-028 | AI denial-of-service / resource exhaustion | Operational | Medium | 3×3=9 | Sponge examples (2022) and adversarial inputs that maximize compute cost — a single crafted prompt can consume 100× normal GPU time. | Per-user rate limiting; input length caps; cost monitoring; auto-scaling with spend ceiling |
Confidential data leakage via LLM prompts
Data & Privacy
Incident: Samsung engineers pasted proprietary source code into ChatGPT (2023), leading to a company-wide ban.
Controls: DLP on AI endpoints; approved-tool-only policy; prompt logging
Prompt Scan →Prompt injection in LLM-powered applications
Security
Incident: Demonstrated across GPT-4, Claude, Gemini — attackers override system prompts to exfiltrate data or alter behavior.
Controls: Input sanitization; privilege separation; output filtering; red-team testing
Prompt Scan →Malicious models in public registries
Supply Chain
Incident: JFrog researchers found 100+ malicious models on HuggingFace with hidden code execution via pickle deserialization (2024).
Controls: Model provenance verification; scan before load; pin model hashes; internal model registry
Model Risk Scanner →Insecure AI agent tool use
Security
Incident: OWASP Agentic AI Top 10 (2025) — agents granted broad tool access executing unintended file, DB or API operations.
Controls: Least-privilege tool grants; human-in-the-loop for destructive actions; tool-call audit logging; sandboxed execution
Agent Governance Plane →Deepfake-enabled CEO fraud
Security
Incident: Attackers used AI-cloned voice and video of a CFO to trick an employee into wiring US $25M in Hong Kong (2024).
Controls: Out-of-band verification for wire transfers; deepfake awareness training; callback procedures
Shadow AI — unauthorized tool adoption
Governance
Incident: Salesforce survey (2024): 55% of generative-AI users at work had not received employer approval.
Controls: AI acceptable-use policy; network-level visibility; approved tool catalogue
AI hallucination in high-stakes decisions
Operational
Incident: Lawyer Steven Schwartz cited six fabricated case citations generated by ChatGPT in a federal filing — Mata v. Avianca (2023).
Controls: Mandatory human review for any AI-assisted output used in decisions; citation verification workflows
AI-powered phishing at scale
Security
Incident: WormGPT and FraudGPT (2023) — purpose-built LLMs for generating convincing phishing emails, lowering attacker skill barrier.
Controls: AI-aware email gateway; phishing simulation training; DMARC/DKIM/SPF enforcement
Jailbreak attacks bypassing safety guardrails
Security
Incident: Many-shot jailbreaking (Anthropic, 2024), DAN prompts, and crescendo attacks consistently bypass model alignment.
Controls: Layered guardrails (input + output); behavioral monitoring; model red-teaming; rate limiting
Model Audit →EU AI Act non-compliance — failure to classify AI system risk
Legal & Compliance
Incident: EU AI Act (Aug 2025 enforcement for prohibited practices, Aug 2026 for high-risk): organizations without a risk classification face fines up to €35M or 7% of global revenue.
Controls: AI system inventory with risk classification; conformity assessment for high-risk; prohibited-use screening; appointed AI officer
AI Threat Model Builder →Biased AI in hiring and HR decisions
Legal & Compliance
Incident: Amazon scrapped an AI recruiting tool that systematically downgraded female applicants (disclosed 2018, widely cited since).
Controls: Bias testing before deployment; demographic parity checks; human override; NYC Local Law 144 compliance
AI API key and credential exposure
Data & Privacy
Incident: GitGuardian (2024): 12.8M secrets detected in public GitHub repos — OpenAI, Anthropic and cloud-AI keys among the fastest-growing categories.
Controls: Secret scanning in CI/CD; key rotation automation; vault-based secret management; least-privilege API scopes
AI copyright and IP infringement
Legal & Compliance
Incident: NYT v. OpenAI (2023), Getty v. Stability AI (2023) — lawsuits alleging training on copyrighted works without license.
Controls: Training data audit; license clearance; opt-out compliance; output watermarking
GDPR / data-protection violations from AI processing
Legal & Compliance
Incident: Italy's Garante temporarily banned ChatGPT (March 2023) for GDPR violations — unlawful processing, no age verification, no legal basis.
Controls: DPIA before deployment; lawful basis documentation; data-subject rights workflows; DPO involvement in AI projects
Absent AI model governance (no model cards, no inventory)
Governance
Incident: EU AI Act (effective 2024–2026) requires AI system registration, risk classification and documentation — organizations without an inventory cannot comply.
Controls: AI model inventory (AIBOM); model cards for every production model; lifecycle governance policy
AI BOM →Indirect prompt injection via retrieved documents
Security
Incident: Demonstrated in RAG systems — adversary plants instructions in a document the AI retrieves, hijacking the response (Greshake et al., 2023; live exploits in Bing Chat).
Controls: Data/instruction separation; retrieved-content sandboxing; output validation; RAG content filtering
Prompt Scan →AI-enhanced social engineering (voice cloning, video deepfakes)
Security
Incident: Multiple reported incidents of AI voice clones used in vishing (2023–2024); deepfake video calls used to impersonate executives.
Controls: Multi-factor identity verification; verbal code words for sensitive requests; deepfake detection tooling; awareness training
PII and credentials leaked in training datasets
Data & Privacy
Incident: Carlini et al. (2021, 2023): GPT-2 and ChatGPT memorize and regurgitate PII, phone numbers and email addresses from training data.
Controls: PII scrubbing in training pipelines; differential privacy; output filtering for PII patterns; data retention policy
Training data poisoning
Security
Incident: Nightshade and Glaze tools (2024) demonstrated practical poisoning of image-model training data at scale.
Controls: Data provenance tracking; input validation on training pipelines; anomaly detection on training metrics
Model Risk Scanner →MCP server tool-poisoning attack
Supply Chain
Incident: Invariant Labs disclosed tool-poisoning attacks via MCP servers — injecting hidden instructions into tool descriptions to exfiltrate data (2025).
Controls: MCP config audit; description-change detection across scans; tool allowlisting; server hash pinning
MCP Scan →AI model theft via API extraction
Security
Incident: Researchers demonstrated near-perfect extraction of production models (including GPT-like transformers) through systematic API querying (multiple papers, 2020–2024).
Controls: Query rate limiting; output perturbation; watermarking; API access monitoring; differential privacy
Excessive AI agent autonomy
Governance
Incident: ChaosGPT experiment (2023) — autonomous agent attempted to acquire resources and recruit other AI agents beyond operator intent.
Controls: Bounded autonomy policies; kill switches; action logging; human approval for high-impact steps
Agent Governance Plane →AI-generated code introducing vulnerabilities
Operational
Incident: Stanford study (2023): developers using AI code assistants produced significantly less secure code and were more confident it was secure.
Controls: SAST/DAST on AI-generated code; mandatory code review; security-focused AI coding guidelines
NPM Scanner →Deployment of abliterated / uncensored models
Governance
Incident: Uncensored Llama, Dolphin and WizardLM variants circulate on HuggingFace — refusal training removed, producing harmful content on request.
Controls: Model provenance verification; abliteration detection; restricted deployment environments; output monitoring
Abliterated Model Risk →Adversarial evasion of ML-based security tools
Security
Incident: Academic research and real-world bypasses of ML-based malware classifiers, spam filters, and fraud-detection models using adversarial perturbations.
Controls: Ensemble detection; adversarial training; signature + ML hybrid approach; regular model retraining
Third-party AI API single point of failure
Operational
Incident: OpenAI major outage (Nov 2023, 2+ hours) — every product built on its API went down simultaneously.
Controls: Multi-provider fallback; local model option; graceful degradation; SLA monitoring
AI-enabled mass surveillance overreach
Legal & Compliance
Incident: Clearview AI fined €20M by French CNIL (2022) and ordered to delete data; similar actions by UK ICO, Italian Garante and Australian OAIC.
Controls: Privacy impact assessment; facial recognition use policy; consent mechanisms; jurisdictional compliance review
Automation bias — over-reliance on AI without human review
Operational
Incident: NTSB investigations cite automation complacency in aviation incidents; same pattern emerging in AI-assisted medical diagnosis and content moderation.
Controls: Mandatory human-in-the-loop for critical decisions; confidence-threshold alerts; UX that surfaces uncertainty
Model collapse from synthetic training data
Operational
Incident: Shumailov et al. (Nature, 2024): models trained on AI-generated text progressively degrade — "model collapse" — losing distributional tails.
Controls: Training data provenance; synthetic-data labeling; human-generated data preservation; output quality monitoring
AI denial-of-service / resource exhaustion
Operational
Incident: Sponge examples (2022) and adversarial inputs that maximize compute cost — a single crafted prompt can consume 100× normal GPU time.
Controls: Per-user rate limiting; input length caps; cost monitoring; auto-scaling with spend ceiling
Six risk categories
Methodology & sources
These 30 risks are incident-grounded: each one is tied to a documented, publicly reported event or peer-reviewed study rather than a hypothetical. Sources cited in the register include breach disclosures and company actions (Samsung’s ChatGPT ban, the Hong Kong $25M deepfake wire fraud), security research (JFrog’s malicious HuggingFace models, Invariant Labs’ MCP tool-poisoning, Greshake et al. on indirect prompt injection), regulator actions (Italy’s Garante ChatGPT ban, France’s CNIL fine of Clearview AI), litigation (NYT v. OpenAI, Getty v. Stability AI, Mata v. Avianca), academic work (Carlini et al. on memorisation, Shumailov et al. on model collapse in Nature), survey data (Salesforce on shadow AI, the Stanford study on AI-assisted code), and regulatory frameworks (the EU AI Act, OWASP Agentic AI Top 10).
Scoring uses a qualitative likelihood × impact model, the same approach used in NIST AI RMF and ISO 31000 risk registers. Each risk is assigned a likelihood (1–5) and an impact (1–5); their product is the risk score, which maps to a severity band — Critical, High, Medium or Low. Likelihood reflects how frequently the incident class has been observed in the wild; impact reflects the severity of the documented cases. The recommended controls are mapped from the same frameworks and from each incident’s root cause.
As of October 14, 2026
Cite this reference
When referencing this data, cite the page and its access date, for example:
PlayCISO. “30 Real-World AI Security Risks.” PlayCISO AI Security Risk Register. Accessed October 14, 2026. https://playciso.com/guides/ai-security-risks
The register is maintained as new incidents and regulatory milestones are documented; the figures above reflect the register as of the date shown.
Go deeper
Frequently asked questions
What is PlayCISO’s AI Security Risk Register?
It is a free, structured catalogue of 30 AI and LLM security risks organisations face when adopting AI. Every risk is grounded in a documented real-world incident and recorded with its category, a likelihood (1–5), an impact (1–5), a resulting severity band, recommended controls, and — where relevant — a linked PlayCISO tool that helps address it.
What are the most severe AI security risks?
The Critical-severity risks are confidential data leakage via LLM prompts (the Samsung source-code leak), deepfake-enabled CEO fraud (the $25M Hong Kong wire fraud), prompt injection in LLM-powered applications, malicious models in public registries, and insecure AI agent tool use. In total the register has 5 Critical, 20 High, and 5 Medium risks.
How is each AI risk scored?
Each risk carries a likelihood (1–5) and an impact (1–5); their product is the risk score, which maps to a severity band of Critical, High, Medium or Low. The scoring is a qualitative likelihood × impact model, the same approach used in NIST AI RMF and ISO 31000 risk registers, with the likelihood and impact estimates anchored to how often each incident class has been observed and how damaging the documented cases were.
Are these risks real or hypothetical?
Every one of the 30 is grounded in a documented, publicly reported incident or peer-reviewed research — for example the Samsung ban, the $25M deepfake wire fraud, JFrog’s malicious HuggingFace models, Italy’s Garante ChatGPT ban, and the Mata v. Avianca fabricated-citations case. None are invented; the citations name the source for each.
Can I use this register for my own AI governance programme?
Yes. Browse it here for reference, or open the interactive AI Risk Register tool to filter, sort and export all 30 risks — with categories, severity, likelihood, impact and controls — as a CSV to drop into your own GRC tooling, board deck or EU AI Act / NIST AI RMF risk-management process.