Adversarial Machine Learning: Attacks on AI

How adversarial attacks work

A trained model draws decision lines in a space with many dimensions. These lines rarely sit where a human would expect them. An attacker who knows them, or can map them by probing, can build inputs that push the model into a wrong decision. To a person, the input still looks the same. Attacks are usually grouped by the stage of the model's life they target:

• Evasion at inference time: The attacker changes an input slightly, so the model gets it wrong. Well-known methods such as FGSM (Fast Gradient Sign Method) and PGD (Projected Gradient Descent) compute these small changes from the model's gradients. In practice, malware can be reshaped so that an ML-based scanner rates it as harmless. In research tests, stickers on road signs fooled image recognition.
• Poisoning during training: Tampered training data changes how the model behaves for good, or plants a hidden backdoor. We cover this in detail under AI poisoning.
• Model extraction: The attacker sends many targeted API queries and builds a copy that works in a similar way. The business loses intellectual property. The copy also becomes a practice target for more evasion attacks.
• Privacy attacks: Membership inference tests whether a given record was part of the training set. Model inversion tries to rebuild traits of the training data from the model's outputs. Large language models add one more risk: they may repeat stored training data word for word.

What the attacker knows matters. In white-box attacks, the attacker knows the design and the weights. In black-box attacks, the attacker only sees the outputs. Black-box attacks still work in practice. Adversarial examples built on a copy model often also work on the real target. This is called transferability.

Why it matters for mid-sized companies

Many mid-sized firms never train their own models. Yet they use ML parts in daily operations, for example:

• payment fraud detection
• visual quality checks on the shop floor
• spam and phishing filters
• document sorting
• security tools with ML-based detection

Wherever a model makes a decision that affects security or revenue, a new attack surface appears. Realistic results include:

• Malware or fraudulent payments slip past automated controls.
• Faulty parts pass the automated quality check.
• Personal training data can be recovered, which can trigger duties under the GDPR.
• A costly model is copied in effect through an open API.

The risk is not the same everywhere. An internal forecast model without outside inputs is far less at risk than a public classifier whose results an attacker sees right away.

Practical defences

1. Inventory and threat model: List every ML part in use, its input sources and the decisions that depend on it. The MITRE ATLAS knowledge base helps you go through the relevant attack methods step by step.
2. Reduce what you expose: Return only what callers need, for example a decision instead of full confidence scores. Rate limits and login checks make extraction and black-box attacks harder.
3. Test how well models resist attacks: Check models against adversarial examples before go-live. Open-source tools such as the Adversarial Robustness Toolbox help here.
4. Adversarial training: Train the model again with samples that were changed on purpose. This makes it more resistant to known attack types. But it is not a full defense, and it may cost some accuracy.
5. Layered decisions: Combine the model with rule-based checks, plausibility tests or human approval for critical cases. A single model should never be the only security barrier.
6. Monitoring: Watch the inputs and query patterns. Many near-identical requests, or inputs that cluster around the decision line, can be a sign of probing.
7. Privacy by design: Differential privacy during training and data minimization lower the risk of membership inference and inversion.

Regulatory context

The EU AI Act (Regulation (EU) 2024/1689) sets rules for high-risk AI systems. Under Article 15, they must reach a fitting level of accuracy, robustness and cybersecurity. The article names measures against tampered training data, adversarial inputs and attacks on confidentiality.

The Digital Omnibus changed the timeline. It has been in force since 27 July 2026. Since then, high-risk duties apply from 2 December 2027 for Annex III systems. For AI in regulated products under Annex I, they apply from 2 August 2028. Apart from that, Article 32 GDPR requires fitting technical safeguards whenever a model contains personal data.

Conclusion

Adversarial machine learning shows that high test accuracy does not make a model secure. If you use ML parts in production, treat them like any other exposed app. Include them in threat analysis, testing and monitoring. SecTepe can help you add AI parts to your existing risk assessments and penetration tests.

Frequently asked questions

Does adversarial machine learning apply to language models?

Yes. Prompt injection and jailbreaks are forms of evasion against large language models. The OWASP Top 10 for LLM Applications describes these risks for LLM-based apps.

Can a model be fully protected against adversarial examples?

Not according to current research. No defense works against every attack. The goal is to make attacks more costly and to catch wrong decisions with extra controls.