We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Lobsters·Jun 26·5 min read
Introducing Flink's Native S3 FileSystem: Built for Performance, Designed for Production
TL;DR
For years, Apache Flink users faced confusion choosing between two Hadoop-based S3 filesystem plugins, each with limitations. Flink 2.3 introduces flink-s3-fs-native, a new, Hadoop-free S3 filesystem designed specifically for Flink.
✦ Why It Matters
Engineers can now leverage a more efficient and simpler S3 filesystem for Flink applications, enhancing performance and reducing configuration complexity.
Key Takeaways
- Adopt flink-s3-fs-native for better performance.
- Simplify S3 configuration by avoiding Hadoop dependencies.
- Utilize RecoverableWriter for reliable data sinks.
- Monitor performance improvements in production environments.
Full Summary
Apache Flink, a framework for stream processing, has historically relied on two Hadoop-based plugins for S3 support, leading to confusion among users due to overlapping configurations and trade-offs. The newly introduced flink-s3-fs-native is a native S3 filesystem built from the ground up for Flink, eliminating the need for Hadoop dependencies.
This experimental plugin, available in Flink 2.3, offers a more straightforward configuration process and supports features like exactly-once sinks through its RecoverableWriter. Early adopters have reported significant performance gains, with measurable improvements in data read and write speeds.
The transition to this native solution simplifies the user experience and reduces the complexity associated with the previous plugins. As it is already running in production at scale, it demonstrates reliability and efficiency for large-scale applications.