EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Lobsters·27d ago·5 min read
Subnormal floating-point numbers are expensive… on Intel processors
TL;DR
Subnormal floating-point numbers, used for very small values in the IEEE standard, significantly slow down computations on Intel processors. A benchmark revealed that operations involving subnormals can be much less efficient, prompting developers to avoid them for performance reasons.
✦ Why It Matters
Consider optimizing your code to avoid subnormal floating-point numbers for better performance on Intel processors.
Key Takeaways
- Avoid subnormal numbers in performance-critical applications.
- Use benchmarks to measure the impact of subnormals on your code.
- Consider alternative representations for very small values.
- Optimize compiler settings to improve performance with normal values.
Full Summary
Floating-point numbers are represented using the IEEE standard, which includes subnormal numbers for very small values. However, these subnormal numbers are known to slow down computational performance, particularly on Intel processors.
To quantify this impact, a C++ benchmark was created, testing operations on arrays of 16,384 values. The benchmark involved multiplying normal values by a tiny constant to produce subnormal outputs.
Results showed that computations with subnormal values were considerably slower than those with normal values, even when only a small fraction of inputs were subnormal. This performance hit can lead video game developers and machine learning practitioners to avoid using subnormal numbers altogether, impacting the precision of their calculations.