Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
Beyond English benchmarks: clinical llm evaluation in Brazilian Portuguese
TL;DR
Large Language Models (LLMs) have primarily been evaluated in English, creating a gap in clinical decision support for non-English speakers. ClinicalBr is introduced as the first bilingual benchmark for clinical decision-making, utilizing 2,892 real Brazilian case reports.
✦ Why It Matters
Engineers can leverage ClinicalBr to improve LLM performance in Brazilian Portuguese clinical applications.
Key Takeaways
- Utilize ClinicalBr for evaluating LLMs in Brazilian Portuguese.
- Focus on improving exam recommendation task performance.
- Leverage findings to address language gaps in clinical AI.
- Explore model adaptations for Brazilian-endemic conditions.
How It Works
ClinicalBr was constructed from real case reports, creating parallel Portuguese-English pairs to facilitate cross-lingual evaluation. Each case supports specific clinical tasks, allowing for targeted assessment of model performance across languages.