NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·3h ago
AI Research
arXiv CS.AI·14d ago·5 min read
OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining
TL;DR
OntoBook is a novel method that transforms medical ontology structures into synthetic textbooks for pretraining language models. It significantly enhances the performance of ModernCamemBERT on French medical coding tasks, achieving notable improvements in accuracy metrics.
✦ Why It Matters
Explore OntoBook's resources to enhance your medical language model training today.
Key Takeaways
- Utilize OntoBook's synthetic textbooks for medical encoder training.
- Align training objectives to avoid performance degradation.
- Access 1.3 million reformulated medical textbooks for research.
- Experiment with ModernCamemBERT for improved medical coding tasks.
How It Works
OntoBook operates by first performing random walks through medical ontology graphs, which helps capture the hierarchical and causal relationships between various medical codes. These walks are then reformulated into fluent, textbook-style prose using a large language model.
The resulting text serves as training data for the ModernCamemBERT model, which is trained on two tasks: masked language modeling, where parts of the text are hidden and predicted, and relation prediction, which identifies connections between pairs of medical codes.