Accelerating the search for antimicrobial peptides with Google Antigravity
deepmind.google·2h ago
AI Research
arXiv CS.AI·1d ago·5 min read
UltraBERT: Beyond the Limit of BERT Pre-Training
TL;DR
UltraBERT is a novel encoder that optimizes pre-training by using separate language model heads for different masking ratios, achieving state-of-the-art results in both classification and retrieval tasks.
✦ Why It Matters
Consider adjusting your model training strategies to incorporate separate masking ratios for improved performance.
Key Takeaways
- Experiment with different masking ratios for classification and retrieval tasks.
- Limit pre-training to a few trillion tokens to avoid saturation.
- Use separate language model heads for varying masking ratios.
- Apply distillation techniques selectively based on masking ratios.
How It Works
UltraBERT employs distinct language model heads for low and high masking ratios within a single batch. This allows the model to optimize performance based on the specific task at hand, enhancing classification with lower ratios and retrieval with higher ratios.
The distillation process is restricted to low masking ratios, which improves classification outcomes without compromising retrieval performance.