Skip to content
SecTepeMit Sicherheit gut aufgestellt!
DE
Glossary

AI Poisoning: Data and Model Poisoning Explained

AI poisoning explained: how manipulated training data, backdoors and poisoned RAG sources compromise AI models – and which controls actually help.

Definition: AI poisoning (data or model poisoning) is an attack on what an AI system learns from. The attacker tampers with training data, fine-tuning data, knowledge sources or pre-trained models. Later, the AI system behaves wrongly, with bias, or in the attacker's interest. Runtime attacks happen while the system is in use. Poisoning is different: the change is already built into the model or its data sources before the system goes live.

Types of poisoning

Poisoning targets the points where a model learns or looks up knowledge. There are several types, based on what the attacker wants:

  • Availability attacks: Large amounts of bad data, such as swapped labels (label flipping), lower the quality of the whole model. This usually shows up in testing, but it is costly to fix.
  • Targeted attacks: The model works normally for most inputs. But it always gets certain cases wrong, for example one supplier or one malware family.
  • Backdoors (trojans): The attacker plants a trigger pattern in a small number of training samples. When the trigger later shows up in an input, the model does what the attacker wants. Without the trigger, the model looks normal. In clean-label attacks, the poisoned samples even have the correct label. This makes them very hard to spot by hand.
  • Poisoned knowledge sources (RAG poisoning): With retrieval-augmented generation, a language model reads documents at runtime. They can come from wikis, SharePoint, ticket systems or the web. Anyone who can add content there can steer the answers without touching the model.
  • Supply chain model poisoning: Pre-trained models from public repositories can contain tampered weights. With unsafe file formats such as Python pickle, they can even contain harmful code that runs.

Backdoors are very dangerous because a small share of tampered samples is often enough. Huge training datasets built from many outside sources are an obvious target.

Why poisoning matters to businesses

Mid-sized companies rarely train large models from scratch. But more and more of them fine-tune models on their own data or use open-source models. Others connect internal document stores to an AI assistant via RAG. This creates real entry points:

  • Customer feedback, support tickets or form entries flow into training data without review.
  • An internal chatbot reads wiki pages that many employees or outside partners can edit.
  • Developers download models from public hubs without checking where they come from or whether they are intact.
  • A service provider delivers labeled data, and nobody spot-checks its quality.

The damage can take many forms. Forecasts may slowly get worse. Recommendations may be tampered with. Harmful code in a model file may take over developer machines. Poisoning is hard to prove, because the model behaves normally most of the time.

Countermeasures

  1. Document data provenance: For every training and knowledge source, write down where the data comes from, who may change it and when it was last checked.
  2. Restrict write access: Treat sources that feed training or RAG like production code. Limit who can change them, and log every change so you can trace it.
  3. Validate data: Before training, check new data for outliers, duplicates, odd label patterns and anything suspicious. This includes spot checks on labels from outside suppliers.
  4. Source models safely: Use trusted providers and check checksums and signatures. Prefer safe formats such as safetensors over pickle. An AI bill of materials (AI-BOM) helps you keep track of models and datasets.
  5. Version and roll back: Keep versions of datasets and models. If you suspect poisoning, you can go back to a known good state.
  6. Behavioural testing: After every training run, test models against fixed reference cases. Look on purpose for odd behavior on rare inputs. This does not guarantee that you find a backdoor, but it improves the odds.
  7. Harden RAG answers: Show the sources with every answer. Keep trusted knowledge bases apart from ones that anyone can edit. Watch for changes to documents that are retrieved often.

Standards and regulation

Several standards and laws deal with poisoning:

  • The OWASP Top 10 for LLM Applications (2025 edition) lists data and model poisoning as its own risk.
  • The MITRE ATLAS knowledge base documents the related attack methods.
  • The EU AI Act requires documented data governance for high-risk AI in Article 10. In Article 15, it calls for measures against data poisoning and model poisoning.
  • Organizations that build an AI management system under ISO/IEC 42001 also treat data quality and data origin as a governance topic.

Conclusion

At its core, AI poisoning is an integrity problem in the supply chain of data and models. Control data sources, write rights and model origin as strictly as source code. This closes the most important gaps. SecTepe helps assess AI data flows as part of your information security management.

Frequently asked questions

How does poisoning differ from prompt injection?

Poisoning tampers with what a model learns or looks up as knowledge. Prompt injection tampers with a single request at runtime. With RAG, the line blurs. A poisoned document can carry hidden instructions and trigger an indirect prompt injection.

Are cloud AI services such as large language models affected?

The base model is the provider's job. Your own additions are yours to secure: fine-tuning data, connected documents and plugins.

More terms in “AI & Machine Learning”

All terms in “AI & Machine Learning” →