Skip to content
SecTepeMit Sicherheit gut aufgestellt!
DE
Glossary

AI Penetration Testing: Assessing ML and LLM Systems

AI penetration testing: how to test LLM applications and ML models for prompt injection, data leakage and manipulation – methodology, scope and tools.

Definition: An AI penetration test is an authorised security check with a fixed time frame. Specialists look at an AI system through an attacker's eyes. The system can be an LLM application, an AI agent or a machine learning model. The goal is to find weak spots that attackers could exploit. These can lie in the model, in the application around it or in the data and tools linked to it. The testers then give advice on what to fix first.

What makes an AI pentest different

A classic web pentest looks for flaws such as SQL injection, broken access control or insecure settings. AI applications still need these checks. A chatbot usually sits inside a normal web application with an API, user management and a database.

On top of that comes a new attack surface. The model works with natural language. It cannot reliably tell instructions apart from data. And it does not always give the same answer. So a weak spot may only show up in some attempts. That is why testers must run test cases many times and in many variations.

Areas under test

LLM applications and agents

The OWASP Top 10 for LLM Applications (2025 edition) gives guidance. Typical test cases:

  • Direct and indirect prompt injection: Can someone get around the safeguards? Either through user input, or through crafted content in emails, web pages and documents that the system reads?
  • Sensitive information disclosure: Does the system reveal other users' data, internal documents or credentials?
  • System prompt leakage: Can someone pull out the system prompt? Does it hold secrets that do not belong there?
  • Excessive agency: Can an attacker get an agent to use tools with overly broad rights, for example to send emails or change records?
  • Improper output handling: Does model output go unchecked into HTML, SQL or shell commands?
  • Permissions in RAG systems: Does the search in the knowledge base respect each user's access rights?

Classic ML models

For classifiers, for instance in fraud or malware detection, other attacks matter most. MITRE ATLAS lists them in a structured way:

  • Adversarial examples (evasion): Inputs with tiny changes that push the model into a wrong decision.
  • Model extraction: Copying the model by sending many queries to its interface.
  • Membership inference: Working out whether certain records were used in training. This matters when personal data is involved.
  • Data poisoning: Checking whether and how attackers can influence training or feedback data.

Infrastructure and supply chain

The test also covers:

  • API keys and how they are stored
  • rate limits
  • how well the model setting is isolated
  • where pre-trained models come from
  • unsafe model formats that can run code when loaded

How an AI pentest runs

  1. Scoping: Set the system borders, the linked tools and data sources, the test setting and what is out of scope. Clarify whether you may test against an outside AI provider and which terms of use apply.
  2. Threat modelling: Who might attack the system? With what goal? Through which input channels?
  3. Automated testing: Open-source tools such as garak, PyRIT or the Adversarial Robustness Toolbox create large numbers of test inputs.
  4. Manual testing: Skilled testers chain weak spots into real attack paths. One example is an indirect prompt injection through a document that makes an agent query data.
  5. Report: Evidence that others can repeat, with the success rate, a risk rating and clear advice.
  6. Retest: A check that the fixes work. At best, repeat it after major model or prompt changes.

Regulatory context

The EU AI Act requires high-risk AI systems to achieve an appropriate level of accuracy, robustness and cybersecurity (Art. 15). This explicitly includes resilience to attacks such as data poisoning and adversarial examples. Providers of general-purpose AI models with systemic risk must carry out adversarial testing.

Outside these categories, a pentest is also useful. It is clear evidence that you have assessed risks in the sense of Art. 32 GDPR and of an ISO/IEC 27001 ISMS.

Conclusion

An AI penetration test joins classic application security with AI-specific attacks. Run one before go-live and after major changes. SecTepe carries out penetration tests of web applications and AI integrations and advises you on how to fix the findings.

Frequently asked questions

What is the difference between an AI pentest and AI red teaming?

A pentest tries to find as many technical weak spots as it can within a set scope. Red teaming tends to act out one specific attacker goal. It often also covers content risks, such as harmful or misleading output. In practice, people do not use the terms in a consistent way.

Do we need a pentest if we only use an off-the-shelf AI service?

You usually cannot test the provider's model. But you can test how you have built it in: which data is linked, what rights the service has and how its output is handled. In practice, that is where many weak spots arise.

More terms in “AI & Machine Learning”

All terms in “AI & Machine Learning” →