Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Jul 23·5 min read
Opto-ViT-v2: Noise-Resilient On-Chip Fine-Tuning for Photonic Near-Sensor Vision Transformer Accelerators
TL;DR
Opto-ViT-v2 introduces a novel framework for fine-tuning Vision Transformers on silicon-photonic accelerators, achieving high accuracy and efficiency despite device noise. It enables practical on-chip training with reduced storage and computational costs.
✦ Why It Matters
Consider implementing Opto-ViT-v2 for efficient on-chip training in your photonic systems today.
Key Takeaways
- Utilize low-rank decomposition to reduce parameter storage.
- Implement gradient-accumulated sparse classifiers to cut training costs.
- Leverage the noise model for robust on-chip training.
- Explore photonic accelerators for high-efficiency vision tasks.
How It Works
Opto-ViT-v2 employs a tensorized low-rank decomposition to separate pretrained optical weights from a small set of trainable electronic factors. This approach minimizes the storage requirements for activations and reduces the frequency of weight updates, enabling efficient on-chip training.
The framework also incorporates a gradient-accumulated sparse classifier that selectively freezes low-importance weights, significantly lowering the computational burden during training.