EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Lobsters·Sep 10·5 min read
Absolutely Everything You Always Wanted to Know About Watermarks in Apache Flink
TL;DR
Watermarks in Apache Flink are crucial for managing event-time processing in stream applications. This article provides an in-depth exploration of their generation, propagation, and implications for time-based operations.
✦ Why It Matters
Implement appropriate Watermark strategies to enhance event-time processing in your Flink applications today.
Key Takeaways
- Use Watermarks to manage event-time in stream processing.
- Adjust Watermark Delay to handle out-of-order events effectively.
- Monitor Watermark progression to avoid stuck data flows.
- Choose the right Watermark type based on latency requirements.
How It Works
Watermarks are generated by source subtasks based on the maximum event-time from Kafka partitions, minus a delay to accommodate out-of-order events. They are propagated downstream, influencing the timing of operations like window closures.
Each subtask maintains its own Watermark time, which is determined by the minimum of the Watermarks received from its inputs.