Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Jul 27·5 min read
Benchmarking Fine-tuning and Retrieval Strategies for a Multimodal Language Model on the NRC Reactor Operator Licensing Examination
TL;DR
A multimodal language model, Gemma 4 31B-IT, was evaluated for its ability to answer questions from the NRC Reactor Operator licensing exam. The best-performing configuration achieved an aggregate accuracy of 79.7%, surpassing the human passing criterion on 8 out of 14 exams.
✦ Why It Matters
Consider implementing fine-tuning strategies to enhance LLM performance in specialized domains like nuclear power.
Key Takeaways
- Utilize supervised fine-tuning for improved model performance.
- Implement retrieval-augmented generation for domain-specific tasks.
- Experiment with chunking strategies based on training state.
- Focus on fine-tuning to meet industry-specific knowledge requirements.
Full Summary
Integrating large language models (LLMs) into specialized fields like nuclear power necessitates a focus on domain-specific knowledge. This study benchmarks the Gemma 4 31B-IT model, a 31-billion-parameter multimodal model, against the NRC Reactor Operator licensing examination.
Eight configurations were tested, including supervised fine-tuning (SFT) with chain-of-thought (CoT) rationales and retrieval-augmented generation (RAG) using the Department of Energy's Fundamentals Handbook. The SFT with fixed-size chunking RAG configuration excelled, passing 8 out of 14 exams, while no non-fine-tuned configurations succeeded.
The overall accuracy was 79.7%, with 80.2% on pressurized water reactor items. Notably, the effectiveness of chunking strategies varied based on the model's training state, and retrieval-augmented fine-tuning (RAFT) was less effective than standard SFT.
These findings highlight optimal strategies for achieving operator-level capabilities in nuclear knowledge applications.