Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Sep 1·5 min read
When Does Bigger Help? A Controlled Study of LLM Scale for Ontology Learning
TL;DR
A controlled study evaluated the impact of Large Language Model (LLM) scale on ontology learning, revealing that larger models improve precision but not uniformly across tasks. Key findings indicate that model architecture can be more influential than size alone.
✦ Why It Matters
Choose LLMs based on architecture and task requirements, not just size, for effective ontology learning.
Key Takeaways
- Evaluate LLMs based on architecture, not just parameter count.
- Expect precision improvements with larger models, especially in dense variants.
- Consider Mixture-of-Experts models for taxonomy discovery tasks.
- Be cautious with non-taxonomic relationship extraction across scales.
Full Summary
Ontology learning (OL) performance in relation to the scale of Large Language Models (LLMs) has not been thoroughly explored. This study assessed 13 models, including dense and Mixture-of-Experts variants from the Qwen3.5 and Qwen3.6 lineages, as well as proprietary GPT releases, using a consistent retrieval-augmented generation pipeline called OntoLearner.
Evaluations focused on term typing, taxonomy discovery, and non-taxonomic relationship extraction across four biomedical and materials science ontologies. Results showed that within the dense Qwen3.5 lineage, increasing model parameters primarily enhanced precision, especially between 9B and 27B parameters.
However, larger sparse models did not outperform smaller dense models in term typing, while Mixture-of-Experts models excelled in taxonomy discovery. Notably, non-taxonomic relationship extraction remained challenging, particularly for the Materials Data Science ontology.
These findings suggest that model size is not the sole criterion for OL effectiveness, emphasizing the importance of architecture and lineage.