Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Aug 3·5 min read
Benchmarking Frontier Large Language Models Against Official Crash Database Coding Using Police Crash Narratives
TL;DR
This study benchmarks six large language models (LLMs) against police crash narratives to assess their ability to code crash attributes. GPT-5.5 High performed best, but traditional baselines showed competitive results, highlighting the need for attribute-specific evaluations.
✦ Why It Matters
Consider using LLMs for crash coding, but validate their performance against established baselines.
Key Takeaways
- Evaluate LLMs against traditional coding baselines for crash data.
- Use specific metrics like F1 score and Cohen's kappa for assessment.
- Focus on attribute-specific performance when deploying LLMs.
- Incorporate human review to ensure accuracy in crash coding.
Full Summary
Police crash narratives provide valuable context for structured crash databases, but manual coding is labor-intensive. This research evaluated six frontier large language models (LLMs) by comparing their coding of crash attributes derived from narratives against an official Arkansas fatal-crash database.
The analysis linked 5,587 narratives to 5,889 structured records, resulting in 4,194 matched crashes. Each LLM was prompted to code attributes like crash manner and light condition using a zero-shot approach.
Performance metrics included agreement rates, macro-averaged F1 scores, and Cohen's kappa. GPT-5.5 High achieved the highest agreement, yet traditional baselines outperformed in raw agreement and F1 scores.
Notably, agreement was highest for non-motorist relation and crash manner, while it was lowest for light condition and roadway surface condition. These findings establish a benchmark for LLM-based crash coding and emphasize the importance of transparent evaluations and human review.