NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·1h ago
AI Research
arXiv CS.AI·May 26·5 min read
Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text: A Comparative Study
TL;DR
A significant gap exists in understanding tokenizer fertility, which affects model selection for Ukrainian legal text. This study benchmarks seven foundation models, revealing that Qwen 3 models use 60% more tokens than Llama-family models, and NVIDIA Nemotron Super 3 outperforms others at lower costs.
✦ Why It Matters
Engineers should prioritize tokenizer analysis for cost-effective model selection in legal NLP applications.
Key Takeaways
- Analyze tokenizer fertility before selecting models.
- Consider zero-shot performance as a reliable default.
- Be cautious with few-shot prompting in Ukrainian language tasks.
- Utilize the released dataset for training and evaluation.
Full Summary
Tokenizer fertility, which refers to the efficiency of a model in processing text, varies significantly across foundation models, yet it is often overlooked in model selection. This study benchmarks seven models from five providers on 273 validated Ukrainian court decisions, focusing on tokenizer fertility and zero-shot performance across three tasks.
Key findings include that Qwen 3 models consume 60% more tokens than Llama-family models, and the NVIDIA Nemotron Super 3 achieves the highest performance score while being more cost-effective than larger models. Furthermore, few-shot prompting leads to a performance drop of up to 26 percentage points, indicating that this issue is inherent to Ukrainian-language tasks.
A cross-temporal analysis shows that classifiers trained on pre-war data perform poorly on wartime decisions, with newer models showing better backward transfer. To aid reproducibility, a public dataset of 14,452 court decisions has been released, annotated with outcome labels across different time periods.