Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 25·8 min read
PoisonForge: Task-Level Targeted Poisoning Benchmark for Instruction-Tuned LLMs
TL;DR
Fine-tuning large language models on untrusted data creates a vulnerability: attackers can inject a small number of poisoned training examples to manipulate model outputs for specific tasks. PoisonForge is a benchmark that systematically evaluates this threat across 12 models and five task families, finding that just 10 poisoned examples among 1,000 training examples achieves over 70% attack success in vulnerable configurations while remaining nearly undetectable on other tasks.
✦ Why It Matters
Engineers deploying fine-tuned LLMs must understand poisoning vulnerabilities and implement data validation before training on external datasets.
Key Takeaways
- Validate and vet all fine-tuning datasets for malicious examples before training models.
- Monitor model outputs on target tasks post-deployment to detect unexpected entity injection patterns.
- Prioritize data sanitization over relying on model scale as a defense mechanism.
- Test fine-tuned models across non-target tasks to detect unintended behavioral changes.
Full Summary
When practitioners fine-tune instruction-tuned large language models (LLMs—neural networks trained to follow natural language instructions) on unvetted datasets, adversaries can perform task-level poisoning: inserting crafted instruction-response pairs that cause the model to consistently embed attacker-specified entities (like country names) in outputs for targeted task families while behaving normally elsewhere. PoisonForge is a benchmark that parameterizes this threat along four dimensions: bias type, poisoning mode, appearance count, and target output length.
The researchers evaluated 12 open-weight models ranging from 2 billion to 32 billion parameters across five task families using primarily a 1% poison budget (10 poisoned examples per 1,000 training examples). Results showed 11 of 12 models exceeded 70% attack success rate in their most vulnerable configuration, with unintended leakage to non-target tasks remaining below 0.5% and standard benchmark performance unaffected.
Analysis revealed that poisoning design choices—not model size—drive attack success, and that multiple entity appearances, semantic structure, and output length all influence vulnerability. The authors released all configurations, pipelines, and analysis code for reproducible research.