Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
Data Science
Towards AI·24d ago·5 min read
Context Rot Is Why Your 2-Million-Token Window Keeps Forgetting the Middle
TL;DR
A contract-review assistant built with a large context window failed to reliably recall information beyond 40,000 tokens. Despite the increased token limit, the model's performance deteriorated, leading to unnoticed errors.
✦ Why It Matters
Evaluate your model's performance with long inputs to identify potential context rot issues before deployment.
Key Takeaways
- Test models with long inputs to assess recall accuracy.
- Implement monitoring for subtle performance degradation.
- Consider retrieval systems for better information management.
- Stay updated on limitations of large context windows.
Full Summary
Large language models (LLMs) are often designed with extensive context windows to handle more information at once. However, a recent experience with a contract-review assistant revealed a significant issue known as context rot, where the model began to forget information after processing around 40,000 tokens.
This was not an outright failure but a subtle degradation in recall that went unnoticed for days. The assistant was intended to streamline legal reviews without needing a separate retrieval system, but the loss of reliable recall undermined its effectiveness.
This incident underscores the limitations of current LLMs, even those with high token capacities, and suggests that simply increasing context size does not guarantee better performance. Engineers and researchers must consider these limitations when designing applications that rely on extensive context.