Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·Jun 11·5 min read
When Poison Fails After Retrieval: Revisiting Corpus Poisoning under Chunking and Reranking Pipelines
TL;DR
Corpus poisoning, a method of manipulating training data to degrade model performance, has limitations when used in chunking and reranking systems. This study revisits corpus poisoning techniques and evaluates their effectiveness in these specific contexts.
✦ Why It Matters
Engineers should reconsider the effectiveness of corpus poisoning in their AI models, especially in chunking and reranking contexts.
Key Takeaways
- Implement CRCP to improve poisoning attack effectiveness.
- Assess the impact of chunk size on retrieval performance.
- Consider reranking strategies when evaluating RAG security.
- Focus on multi-stage retrieval consistency in security assessments.
How It Works
CRCP optimizes the poisoning process by explicitly modeling the transformations that occur during document chunking. This allows the generation of adversarial passages that are coherent and effective even when the documents are split into smaller chunks.
By ensuring that these passages are locally self-contained, CRCP maintains their relevance and effectiveness through the reranking stage, which typically favors coherent and contextually relevant content.