Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
FOGO: Forgetting-aware Orthogonalization Optimizer
TL;DR
In continual learning, models often forget previously learned information when trained on new data, a phenomenon known as catastrophic forgetting. FOGO, or Forgetting-aware Orthogonalization Optimizer, was developed to mitigate this issue by optimizing the learning process while preserving important past knowledge.
✦ Why It Matters
FOGO enables engineers to build AI systems that learn continuously without sacrificing previously acquired knowledge.
Key Takeaways
- Implement FOGO to reduce forgetting in your models.
- Utilize orthogonalization to improve gradient update efficiency.
- Store past update directions to enhance learning.
- Test FOGO against traditional optimizers like Adam.
How It Works
FOGO orthogonalizes momentum updates to prevent dominant gradients from overshadowing rare but useful directions. It maintains a compact codebook memory of past updates, ensuring that important information is not lost.
By resolving conflicts between current updates and stored directions through orthogonal corrections, FOGO effectively balances the optimization process.