I had the pleasure of attending the Kafka Summit in London last week, and it was a very exciting event! I gave a presentation on streaming Kafka events into Apache Iceberg, followed by some insightful questions and follow-on discussions. There were more than 90 sessions, with topics ranging from stream processing with Kafka Streams and Flink to collecting real-time geospatial aircraft data. There were also a few sessions on my favorite topic, streaming data into data lakes.
And while the data lake theme only represented a small percentage of the sessions at Kafka Summit, it played a huge role in the opening keynote. During the keynote, Jay Kreps, CEO of Confluent, announced Tableflow, their new service that will automatically turn data in Kafka topics into Iceberg tables at the click of a button! To show it off, Danica Fine, staff developer advocate at Confluent, gave an impressive, albeit simple demonstration. She showed data streaming through a Kafka topic on Confluent Cloud. She then clicked a button that generated Iceberg metadata on the underlying Kafka data which was already stored in Parquet files in S3. Next, she opened up Snowflake and queried the Kafka topic that had been turned into an Iceberg table!
Tableflow is still under development, but its potential is enormous. I love it because it’s one step closer to bringing the streaming and batch worlds into a productive harmony that will benefit all of us!
For more information about Confluent’s Tableflow, check out this post by Confluent staff product manager Marc Selwan.
For more details about how Tabular is already helping to bridge the divide between streaming and batch, check out the following posts on the Tabular blog:
- Recent posts by Dave Klein
- Intro to the Kafka Connect sink
- Iceberg CDC cookbook recipe
- Ryan Blue’s CDC series
If you missed Kafka Summit London, you can here the Tableflow announcement in Jay’s keynote online. Exciting times ahead for data practitioners everywhere!
