EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·Jul 29·5 min read
The Scientific Literature Is Poisonous to LLMs
TL;DR
21st century scientific literature is detrimental to training large language models (LLMs), as it contains a mix of honest and dishonest content. A study found that removing certain scientific texts improved LLM performance and reduced toxic outputs.
✦ Why It Matters
Consider revising training datasets for AI models to exclude problematic scientific literature.
Key Takeaways
- Evaluate the impact of scientific literature on your AI training data.
- Consider removing ArXiv and PhilPapers to improve model performance.
- Be aware of the potential toxicity in scientific writing.
- Explore informal settings for AI-assisted scientific collaboration.
Full Summary
Research conducted by a team from MIT, Cornell, Carnegie Mellon, Google, and OpenAI revealed that excluding specific scientific texts, such as those from ArXiv and PhilPapers, enhanced LLM performance on academic questions and overall benchmarks. The study indicated that the presence of toxic or misleading content in scientific literature could negatively impact LLM outputs.
While removing Pubmed data slightly decreased performance, the overall trend suggested that eliminating large sets of scientific articles could lead to better model behavior. This raises concerns about the integrity of scientific writing, as authors often feel pressured to embellish findings or conform to popular theories to secure publication.
Consequently, AI systems lack the social context that human scientists use to evaluate the reliability of research, complicating their application in scientific endeavors.