NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 25·8 min read
Weierstrass Positional Encoding for Vision Transformers
TL;DR
Vision Transformers flatten 2D images into 1D sequences, losing spatial structure that standard positional encodings fail to preserve. Weierstrass elliptic Positional Encoding (WePE) maps 2D patch coordinates onto the complex plane using Weierstrass elliptic functions, creating four-dimensional features with built-in geometric constraints.
✦ Why It Matters
Engineers can drop WePE into existing Vision Transformers for improved spatial reasoning without retraining or performance penalties.
Key Takeaways
- WePE is plug-and-play compatible with existing ViT architectures requiring no retraining.
- Precomputed lookup tables eliminate computational overhead while preserving performance gains.
- Method works across arbitrary image resolutions without modification.
- Algebraic addition formula enables direct computation of relative positions between patch pairs.
Full Summary
Vision Transformers (ViTs) convert images into patches and process them sequentially, but this flattening destroys the original 2D spatial relationships. Current positional encodings—learnable 1D vectors added to patch embeddings—lack geometric grounding and don't preserve the monotonic relationship between actual spatial distances and sequence positions.
WePE addresses this by encoding normalized 2D coordinates in the complex plane using Weierstrass elliptic functions, which are doubly periodic mathematical functions naturally suited to grid structures. The method generates compact four-dimensional positional features whose nonlinear properties faithfully model spatial distances, and whose algebraic properties allow relative positions between any two patches to be computed directly from their absolute encodings.
WePE integrates seamlessly into existing ViTs without architectural changes and works across image resolutions. Experiments demonstrate consistent improvements across most settings, with precomputed lookup tables eliminating computational and memory costs.