Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 25·8 min read
Preisach Attention: A Hysteretic Model of Sequential Memory
TL;DR
Standard transformer attention uses softmax to weight all tokens equally, creating O(n²) computational cost and losing efficiency on tasks where only recent extremes matter. Preisach Attention Layer (PAL) replaces softmax with a binary relay operator that tracks only local peaks and valleys, reducing inference to O(n log n) while achieving Turing-completeness in a single layer.
✦ Why It Matters
Engineers can use PAL for long-sequence tasks with weak positional dependence, achieving 2× faster inference than standard attention.
Key Takeaways
- Explore using PAL for tasks with long episodic memory.
- Implement binary relay operators to replace softmax attention.
- Leverage PAL's efficiency for large-scale sequence data.
- Consider the unique capabilities of PAL for rate-independent functionals.
How It Works
PAL operates by maintaining a stack of local extrema, which serves as a minimal sufficient statistic of the input history. This allows it to respond to the sequence of local extrema rather than absolute token positions, making it efficient for tasks where the timing of inputs is less critical.