Awesome GenAI Security

A curated list of links, references, books, videos, tutorials (Free or Paid), Exploit, CTFs, Hacking Practices, etc., which are related to GenAI, LLM, RAG, MCP, Agents, and Agentic AT security.

Table of Contents

• GenAI Security Papers & Standards
• Books
• Videos
• Online Tutorials / Blogs / Presentations
• Online Courses (Paid/Free)
• Tools of Trade
• Security Practices and CTFs
• GenAI Security Breaches & Incidents
• Contributors

---

GenAI Security Papers & Standards

Important papers, standards, and checklists from organizations like OWASP, NIST, and others.

1. OWASP Top 10 for LLM Application
2. OWASP LLM AI Security and Governance Checklist
3. OWASP Agentic AI Top 10
4. NIST AI RMF Playbook
5. NIST AI Risk Management Framework (AI RMF)
6. NIST Adversarial Machine Learning
7. Microsoft Failure Models in Machine Learning
8. Microsoft Threat Modeling AI/ML
9. OWASP GenAI Security Project

Books

1. AI Value Creators
2. AI Engineering by Chip Huyen
3. Designing Machine Learning Systems
4. Hands-On Large Language Models
5. Nexus by Yuval Noah Harari
6. The Developer's Playbook for Large Language Model Security: Building Secure AI Applications by Steve Wilson

Videos

1. Intro to LLM Security - WhyLabs

Online Tutorials / Blogs / Presentations

Articles and guides covering LLM, RAG, and general GenAI security.

1. LLM Security
2. What are foundational models?
3. A quick check on the AI Threat Model
4. Security Incident Response using LLM
5. OWASP: CheatSheet – A Practical Guide for Securely Using Third-Party MCP Servers 1.0

RAG Security

1. Riding the RAG Trail: Access, Permissions and Context
2. Securing Risks with RAG Architectures
3. Secure your RAG
4. Mitigating Security Risks in Retrieval Augmented Generation (RAG)
5. RAG: The Essential Guide
6. Why RAG is revolutionising GenAI

LLM Attacks

1. Web LLM attacks - PortSwigger
2. Prompt injection jailbreaking

Online Courses (Paid/Free)

1. Stanford CS-324: Large Language Models
2. Princeton COS 597G: Understanding Large Language Models
3. Coursera: GenAI with LLM
4. Coursera: Generative AI Engineering with LLMs Specialization
5. Coursera: Generative AI for Cybersecurity Professionals (IBM)
6. Coursera: AI for Cybersecurity (JHU)
7. AttackIQ: The foundation of AI Security

Tools of Trade

Tools for defending, scanning, and auditing GenAI systems.

Defensive / Scanning

1. LLM Guard - Information extraction and security for LLMs.
2. Model Scan - Scanning models for serialization attacks.
3. Rebuff - Prompt injection detection.
4. NB Defense - Notebook security.
5. Protect AI's OSS Portfolio
6. LLM Guard Playground

Offensive / Exploitation

1. AI/ML Exploits

Security Practices and CTFs

Practice your skills with these vulnerable applications and challenges.

1. Gandalf - Lakera AI - LLM security challenge.
2. Prompt Airlines - AI security challenges, CTF style.
3. Certified AI/ML Pentester Exam
4. Damn Vulnerable MCP Server - Deliberately vulnerable MCP implementation.
5. Vulnerable MCP Servers Lab - Collection of vulnerable servers.
6. FinBot Agentic AI CTF - Agentic Security CTF.
7. OWASP WrongSecrets LLM exercise
8. Huntr.com - World’s first bug bounty platform for AI/ML.

GenAI Security Breaches & Incidents

Coming soon.

Contributors

• Sanjeev Jaiswal