Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
Security
Cloudflare Blog·7h ago·5 min read
Filter, reshape, and redact logs before export with Transformers
TL;DR
Cloudflare has introduced Transformers, a feature that allows users to filter, reshape, and redact logs directly within the Logpush export pipeline, enhancing data management efficiency. This integration simplifies log processing, reducing the need for separate services and minimizing data volume and costs.
✦ Why It Matters
Start using Transformers today to streamline your log processing and reduce data costs.
Key Takeaways
- Utilize Transformers to filter and reshape logs before export.
- Reduce data volume and storage costs by omitting unnecessary records.
- Leverage SQL queries for customized log transformations.
- Monitor transformation health and performance in the Cloudflare dashboard.
How It Works
Transformers allow users to write SQL queries that filter, redact, and reshape log records within the Logpush pipeline. By embedding a SQL transformation engine, Cloudflare ensures that logs are processed efficiently before reaching their destinations.
This integration uses Apache DataFusion, a Rust-based query engine, to execute transformations in-process, maintaining high throughput and reliability.