NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·Jun 12·5 min read
SupraBench: A Benchmark for Supramolecular Chemistry
TL;DR
Designing host-guest systems in supramolecular chemistry is slow and labor-intensive. SupraBench, a benchmark for evaluating large language models (LLMs) on tasks like binding affinity prediction, was developed.
✦ Why It Matters
Engineers and researchers can leverage SupraBench to enhance LLMs for better performance in supramolecular chemistry tasks.
Key Takeaways
- Utilize SupraBench to evaluate LLMs in supramolecular chemistry.
- Explore SupraPMC for training models on supramolecular tasks.
- Identify specific failure modes in LLMs to guide improvements.
- Focus on domain adaptation for better model performance.
Full Summary
Supramolecular chemistry focuses on non-covalent interactions between molecules, which are crucial for various applications but challenging to design efficiently. To address this, SupraBench was created as the first benchmark specifically for evaluating large language models (LLMs) on fundamental tasks in this field, including binding affinity prediction, top-binder selection, solvent identification, and host-guest description.
The researchers also introduced SupraPMC, a 16 million-token corpus of curated supramolecular chemistry articles to aid in model training. Benchmarking a range of LLMs showed that they performed poorly across all tasks, with substantial room for improvement.
Domain adaptation pretraining using SupraPMC improved performance on in-distribution tasks but affected output formatting. The study identified distinct failure modes across different task families, highlighting specific weaknesses in current LLMs' reasoning capabilities.
These insights can guide future research and development in AI applications for chemistry.