❤️ Apache Iceberg? Spread the word by giving it a ⭐ on the apache/iceberg repo!

Project updates

Iceberg Java

• Release 1.4.2 is live! Here are some highlights from 1.4.0 to 1.4.2
  • 1.4.2 addresses an issue that was identified in 1.4.0. Please upgrade to this latest patch release.
  • New tables default to V2 format, as discussed in the dev list.
  • Zstandard (zstd) is the default data compression algorithm on new tables.
  • FileIO was added that supports Azure Data Lake Storage Gen 2.
  • Added REST API for committing changes to multiple tables.
  • Spark-specific highlights for release 1.4.1
    • Increase the default advisory partition size for writes and allow users to set it explicitly.
    • Added distributed planning to improve performance in some use cases dramatically.
    • Skip local sort for unordered writes in Spark using fanout writer
• Added support for Flink’s Alter Table syntax.
• Created foundational view support for metastore which paves the way for iImplementing view support to multiple Hive metastore-based implementations.
• The project is now undergoing a documentation refactor. With the previous refactor, multi-versioned documentation was introduced to the Iceberg back in version 0.12.1. It split the versioned docs source and the static site source into two separate repositories. This has created a confusing and cumbersome contribution and release process for documentation. Following a new design, the Iceberg documentation will retain multi-versioning, while avoiding the concerns around having all versions of the source in a single repository.

PyIceberg, Iceberg-Go, and Iceberg-Rust

• Python implementation moved from the main apache/iceberg repository to iceberg-python repository.
• PyIceberg has been integrated into Polars. Now you can load Polars dataframes directly from Iceberg tables. Learn more in this blog post.
• Significant progress was made towards adding write support in Python.
  • The latest WIP version of the write support can be found here. It is encouraged to give it a try, and see if it works for you.
  • Also, if you’re interested in reviewing, please check out the snapshot logic and the summary generation.
• iceberg-go added supports for manifests and a base table implementation
• Iceberg-rust is going strong,
  • The scaffolding for the REST catalog implementation went in, and the load-table logic is close.

Bergy Blogs

• Apache Iceberg optimization: Solving the small files problem in Amazon EMR
• Building a Feature Store with Apache Iceberg on AWS
• Partner Integration: Apache Iceberg + StarRocks
• 🚀 Exciting Times in the Big Data World: Apache Iceberg Takes Center Stage! 🚀
• Build a Transactional Data Lake with Apache Iceberg
• Lakehouse coverage at Current 2023
• NPLG 10.5.23: A New Way to Monetize Open Source (MotherDuck)
• Calculating Daily/Monthly Active Users with Spark & Iceberg
• Simplifying Complex Data Merging: Combining Data Sources into a Single Table
• The Art of Efficient Data Lake Organization
• State of data catalogs 2023: The battle for your metadata

Ecosystem Updates

• Distributed Materialized Views: How Airbnb’s Riverbed Processes 2.4 Billion Daily Events
• Trino Gateway has arrived.
• Trino adds support for CREATE OR REPLACE TABLE syntax
  • More details can be found in the following issue discussion.

Vendor Updates

• Sippy helps you avoid egress fees while incrementally migrating data from S3 to R2
• If you’re attending AWS re:Invent, here are some Iceberg related sessions you won’t want to miss!
  • ANT101 | How to build a platform for AI and analytics based on Apache Iceberg - Ryan Blue
  • NFX306 | Netflix’s journey to an Apache Iceberg–only data lake - Vaidy Krishnan, Rakesh Veeramacheneni, and Ashwin Kayyoor
  • STG313 | Building and optimizing a data lake on Amazon S3 - Ryan Blue, Huey Han, and Oleg Lvovitch
  • ANT308 | Build large-scale transactional data lakes with Apache Iceberg on AWS - Aditya Challa, Aneesh Chandra, Dylan Qu, Francis McGregor-Macdonald, and Nishchai JM
  • ANT328 | Accessing open table formats for superior data lake analytics - Asser Moustafa, Sreekanth Martha, and Stuti Deshpande
  • OPN309 | Building a secure and scalable transactional data lake - Sercan Karaoglu and Ankita Gavali
• And while you’re at re:Invent, be sure to visit the following vendor booths to see what they’re doing with Iceberg
  • AWS - Athena, EMR, Redshift, Glue, DynamoDB
  • 2604 - Snowflake
  • 1022 - Databricks
  • 1632 - Tabular
  • 1151 - Starburst
  • 1000 - Confluent
  • 1505 - Clickhouse
  • 1530 - Dremio

Iceberg Resources

🏁 Get Started with Apache Iceberg
👩‍🏫 Learn more about Apache Iceberg on the official Apache site
📺 Watch and subscribe to the Iceberg YouTube Channel
📰 Read up on some community blog posts
🫴🏾 Contribute to Iceberg
👥 SELECT * FROM you JOIN iceberg_community
📬 Subscribe to the Apache Iceberg mailing list