Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·Jun 12·5 min read
FENCE: A Financial and Multimodal Jailbreak Detection Dataset
TL;DR
Jailbreaking poses a significant risk to Large Language Models (LLMs) and Vision Language Models (VLMs), especially in finance where detection resources are limited. FENCE, a bilingual (Korean-English) multimodal dataset, was created to train and evaluate jailbreak detectors using finance-relevant queries and image threats.
✦ Why It Matters
Engineers can leverage the FENCE dataset to develop more effective jailbreak detection systems for financial AI applications.
Key Takeaways
- Utilize the FENCE dataset for training detection models.
- Focus on multimodal threats in AI systems.
- Explore vulnerabilities in both commercial and open-source models.
- Enhance AI safety in sensitive financial applications.
Full Summary
Jailbreaking refers to exploiting vulnerabilities in AI models, which is particularly concerning for Large Language Models (LLMs) and Vision Language Models (VLMs) that handle both text and images. The FENCE dataset was developed to fill the gap in resources for detecting these vulnerabilities in financial contexts, featuring bilingual (Korean-English) queries and image-grounded threats.
Researchers conducted experiments with both commercial and open-source VLMs, revealing consistent vulnerabilities, particularly with GPT-4o. A baseline detector trained on the FENCE dataset achieved an impressive 99 percent accuracy in identifying jailbreak attempts and maintained strong performance on external benchmarks.
These findings highlight the dataset's robustness and its potential to improve the reliability of AI systems in sensitive financial domains. By providing a focused resource, FENCE aims to advance multimodal jailbreak detection and enhance the safety of AI applications.