AI Phishing Detection: Machine Learning for Email

How machine learning detects phishing

Classic email filters work with rules. A sender is on a blocklist, a URL is known to be bad, or a file type looks risky. That works well against known campaigns. It works less well against new domains or emails written for one person. ML-based systems add several layers of checks on top of those rules:

• Text analysis (natural language processing): Language models look at tone, urgency and calls to action ("please transfer now"). They also know the typical patterns of fake invoice, parcel or password emails.
• URL and domain features: How old is the domain? Does it look like a known brand (typosquatting, homoglyphs such as "rn" instead of "m")? The models also check redirect chains and the reputation of the host.
• Header and authentication data: SPF, DKIM and DMARC results count here. So do gaps between the display name and the real sender address, and odd mail relays.
• Computer vision: Linked pages can be opened in a safe, isolated space. There, models spot cloned login pages of popular services, even if the source code has been hidden.
• Behavioural anomaly detection: The system learns who usually writes to whom. Say a supplier writes to you for the first time and sends new bank details. That email stands out, even with no link or attachment.

This last layer matters most for business email compromise (BEC) and CEO fraud. These attacks often have nothing technically harmful in them. They are just a believable request, sent to the right person at the right time.

Why the old warning signs no longer work

For years, spelling mistakes and poor translations gave phishing away. Generative AI has mostly taken that sign away. Attackers can now write perfect text for any industry and any person, in large numbers. They can also vary the text so that signature-based filters rarely catch it.

New channels make things worse:

• QR codes in emails ("quishing")
• links hosted on real cloud services
• phone calls with cloned voices

So detection has to rely more on context and behaviour than on how a message looks.

Limits and risks of AI-based filters

• False alarms and misses: Strict models block real invoices or job applications. Loose models let targeted attacks through. You need to tune the limits to your own mail traffic.
• Adversarial evasion: Attackers test their emails against common filter products. They change words, images or link structures until the email gets through.
• Black-box effect: If nobody can see why a message was flagged, it is hard to fix errors. Staff also stop trusting the warning banners.
• Shift to other channels: A strong mail filter pushes attackers to messengers, team tools and the phone. These are often less well protected.

Putting it into practice

1. Get the basics right: Set up SPF, DKIM and DMARC for all your domains. Move DMARC step by step to "quarantine" or "reject". ML filters work better with good authentication data.
2. Check what you already have: Many cloud mail platforms already include ML-based filters. Setting them up well often pays off before you buy an extra product.
3. Add link and attachment analysis: Time-of-click URL rewriting and attachment sandboxing catch pages that only turn harmful after the email has arrived.
4. Give staff a way to report: A report button with quick feedback gives the system training signals. It also keeps people alert.
5. Harden payment processes: Always confirm new bank details and urgent payment requests through a second, separate channel. No filter replaces this four-eyes principle.
6. Measure how well it works: Check these numbers on a regular basis: detection rate, false alarms, reporting rate and the time it takes to remove a delivered phishing email from all mailboxes.

Data protection and works council

ML filters read the content and metadata of personal messages. In Germany and the EU, the legal basis is usually the legitimate interest in IT security (Art. 6(1)(f) GDPR). You must document this and limit it to what is needed. Cloud services need a data processing agreement.

Some firms let staff use their work mailbox for private mail. Then the legal situation gets much harder, so clarify it in advance. If you have a works council, it has a say. Under Section 87(1) no. 6 of the German Works Constitution Act (BetrVG), it must agree before you bring in tech that can monitor staff.

Under the EU AI Act, plain spam and phishing filters are usually minimal-risk AI systems. They have no special duties. The general duty to build AI literacy among staff (Art. 4) still applies.

Conclusion

Machine learning makes phishing defence much better. This is true above all for new campaigns and for BEC fraud with no technical signs. But it only works well together with good email authentication, clear approval steps and alert staff. SecTepe helps you assess your current filters. We also run awareness programmes and support incident response after a phishing attack has worked.

Frequently asked questions

Does an AI filter replace awareness training?

No. Filters cut the number of attacks that reach inboxes. But some emails will always get through. Staff who spot and report suspicious messages are still the last line of defence.

Can AI detect phishing that was written by AI?

Only in part. There is no reliable way to tell if a machine wrote a text. Looking at context works better: unusual senders, new domains, odd patterns in who writes to whom, and risky requests.

How quickly do ML filters adapt to new campaigns?

That depends on the product. Cloud-based tools are usually updated all the time with data from many customers. What counts is that reports from your own staff feed into the system. You should also be able to correct wrong results.