Skip to content
SecTepeMit Sicherheit gut aufgestellt!
DE
Glossary

AI Red Teaming: Offensive Testing of AI Systems

AI red teaming explained: how offensive tests uncover prompt injection, data leakage and unsafe behaviour in AI applications – process, methods, tools.

Definition: AI red teaming is structured offensive testing. A team takes the view of an attacker or abusive user and tries to make an AI system misbehave. It covers security flaws such as prompt injection and data leaks. It also covers harmful, false or non-compliant outputs. The aim is to find weak spots before real attackers or users trigger them.

How AI red teaming differs from a classic penetration test

A classic penetration test looks for wrong settings, vulnerable software and broken access control. These issues still matter for AI apps. Chatbots and agents run on normal infrastructure with APIs, databases and identities.

But on top of that sits a layer that does not always behave the same way. The model responds to natural language. The same input does not always give the same output. And the "vulnerability" is often a behavior, not a bug in the code.

That is why AI red teaming combines three views:

  • Application security: login, access rights, interfaces, plugins, and the rights an AI agent uses to reach other systems.
  • Model-specific attacks: direct and indirect prompt injection, jailbreaks, system prompt leaks, and pulling secret data out of connected data sources.
  • Misuse and content safety: unfair, made-up or legally risky outputs, and statements that could harm the company.

Typical test scenarios

The right scenarios depend on the use case. Common examples:

  • A customer chatbot is talked into revealing internal pricing logic, discount rules or its system prompt.
  • An internal assistant with access to SharePoint or the ticket system hands documents to staff who are not allowed to see them.
  • A crafted email or web page hides instructions. An AI agent carries them out while it summarizes the text, for example by forwarding data (indirect prompt injection).
  • An agent with tool access is tricked into actions far beyond its task (excessive agency).
  • Model output goes unchecked into web pages, SQL queries or shell commands. This opens up classic injection flaws again.

Two references help here. One is the OWASP Top 10 for LLM Applications (2025 edition). The other is the MITRE ATLAS knowledge base. It sorts attack methods against AI systems in a similar way to MITRE ATT&CK.

How an AI red team engagement runs

  1. Define scope and objectives: Which system, interfaces, data sources and tools are in scope? Which outcomes would hurt the business most: data leaks, harm to your reputation, financial actions?
  2. Build a threat model: Who uses the system? Which inputs come from untrusted sources? What rights does the model hold?
  3. Manual testing: Skilled testers create smart attacks that fit the context. Long, multi-step conversations and indirect injections are hard to automate fully.
  4. Automated testing: Open-source tools such as PyRIT, garak or promptfoo run many attack variants. They make results repeatable. Because the model does not always answer the same way, run the tests several times.
  5. Assessment and report: Each finding is rated by its impact and how easy it is to exploit. The report documents it with examples you can reproduce.
  6. Remediation and retest: Typical fixes are tighter rights for agents, approval steps before critical actions, output filters and a clean split between instructions and data. Test again after any change to the model or prompt.

When is AI red teaming worthwhile for mid-sized companies?

Not every use of AI needs its own red team project. It pays off mainly when an AI app:

  • can be reached from outside,
  • accesses secret or personal data, or
  • can start actions in other systems on its own.

For simple internal use of a standard chat service, policies, training and a settings review are often enough.

Regulatory context

The EU AI Act (Regulation (EU) 2024/1689) has applied to this since 2 August 2025. Providers of general-purpose AI models with systemic risk must carry out and document adversarial testing (Article 55). For high-risk AI, Article 15 demands robustness and cybersecurity. That is hard to prove without testing.

The NIST AI Risk Management Framework also places testing under its "Measure" function. For organizations covered by NIS2, AI apps belong in regular risk management under Section 30 of the German BSI Act (BSIG).

Conclusion

AI red teaming shows what an AI system really does under attack, not just what it should do in normal use. When SecTepe tests AI apps, it combines proven penetration testing methods with attack scenarios built for AI.

Frequently asked questions

Is a one-off test enough?

Rarely. Models, system prompts, data sources and tools change often. Test again after major changes and at regular intervals. Ideally, back this up with automated regression tests.

May we test third-party cloud AI services?

Testing your own app is common practice. But you must respect the provider's terms of use. Clarify in advance what is allowed, and get approval if in doubt.

More terms in “AI & Machine Learning”

All terms in “AI & Machine Learning” →