EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·Jul 30·5 min read
Show HN: Distilling DeepSeek into GPT-OSS doesn't transfer censorship. Try it
TL;DR
A new model, GPT-OSS-120B, was distilled from the Chinese DeepSeek V4 Flash, demonstrating improved financial reasoning without inheriting censorship behaviors. This suggests that undesirable traits may not transfer in model distillation.
✦ Why It Matters
Consider using distilled models for financial applications to avoid censorship while enhancing performance.
Key Takeaways
- Explore using distilled models for sensitive applications.
- Evaluate financial reasoning performance with GPT-OSS-120B.
- Utilize LineageEval for assessing model behavior.
- Consider cost-effectiveness in model selection.
How It Works
The distillation process involved training GPT-OSS-120B on outputs from DeepSeek V4 Flash, focusing on financial reasoning tasks. By using a self-hosted version of the teacher model, researchers ensured that external moderation did not influence the results.
The evaluation involved 304 prompts, with judges scoring the responses for censorship, allowing for a clear comparison between sensitive and non-sensitive topics.