Microsoft built a prompt injection detector. Then it caught a phishing campaign instead.

TL;DR

Microsoft's prompt injection detector inadvertently identified a phishing campaign that uses invisible Unicode characters to bypass spam filters. This technique poses risks for both email security and AI systems processing external text.

Key Takeaways

• Implement filters to remove invisible Unicode characters from text inputs.
• Test your NLP models with various tokenizers to understand their behavior.
• Monitor for unusual spikes in message counts to catch phishing attempts.
• Be cautious with Unicode normalization methods; they may not remove all unwanted characters.