Apache Iceberg? Spread the word by giving it a 
on the apache/iceberg repo!
Project updates
Iceberg Java
- Release 1.4.2 is live! Here are some highlights from 1.4.0 to 1.4.2
- 1.4.2 addresses an issue that was identified in 1.4.0. Please upgrade to this latest patch release.
- New tables default to V2 format, as discussed in the dev list.
- Zstandard (zstd) is the default data compression algorithm on new tables.
- FileIO was added that supports Azure Data Lake Storage Gen 2.
- Added REST API for committing changes to multiple tables.
- Spark-specific highlights for release 1.4.1
- Increase the default advisory partition size for writes and allow users to set it explicitly.
- Added distributed planning to improve performance in some use cases dramatically.
- Skip local sort for unordered writes in Spark using fanout writer
- Added support for Flink’s Alter Table syntax.
- Created foundational view support for metastore which paves the way for iImplementing view support to multiple Hive metastore-based implementations.
- The project is now undergoing a documentation refactor. With the previous refactor, multi-versioned documentation was introduced to the Iceberg back in version 0.12.1. It split the versioned docs source and the static site source into two separate repositories. This has created a confusing and cumbersome contribution and release process for documentation. Following a new design, the Iceberg documentation will retain multi-versioning, while avoiding the concerns around having all versions of the source in a single repository.
PyIceberg, Iceberg-Go, and Iceberg-Rust
- Python implementation moved from the main apache/iceberg repository to iceberg-python repository.
- PyIceberg has been integrated into Polars. Now you can load Polars dataframes directly from Iceberg tables. Learn more in this blog post.
- Significant progress was made towards adding write support in Python.
- The latest WIP version of the write support can be found here. It is encouraged to give it a try, and see if it works for you.
- Also, if you’re interested in reviewing, please check out the snapshot logic and the summary generation.
- iceberg-go added supports for manifests and a base table implementation
- Iceberg-rust is going strong,
- The scaffolding for the REST catalog implementation went in, and the load-table logic is close.
Bergy Blogs
- Apache Iceberg optimization: Solving the small files problem in Amazon EMR
- Building a Feature Store with Apache Iceberg on AWS
- Partner Integration: Apache Iceberg + StarRocks
Exciting Times in the Big Data World: Apache Iceberg Takes Center Stage! - Build a Transactional Data Lake with Apache Iceberg
- Lakehouse coverage at Current 2023
- NPLG 10.5.23: A New Way to Monetize Open Source (MotherDuck)
- Calculating Daily/Monthly Active Users with Spark & Iceberg
- Simplifying Complex Data Merging: Combining Data Sources into a Single Table
- The Art of Efficient Data Lake Organization
- State of data catalogs 2023: The battle for your metadata
Ecosystem Updates
- Distributed Materialized Views: How Airbnb’s Riverbed Processes 2.4 Billion Daily Events
- Trino Gateway has arrived.
- Trino adds support for CREATE OR REPLACE TABLE syntax
- More details can be found in the following issue discussion.
Vendor Updates
- Sippy helps you avoid egress fees while incrementally migrating data from S3 to R2
- If you’re attending AWS re:Invent, here are some Iceberg related sessions you won’t want to miss!
- ANT101 | How to build a platform for AI and analytics based on Apache Iceberg – Ryan Blue
- NFX306 | Netflix’s journey to an Apache Iceberg–only data lake – Vaidy Krishnan, Rakesh Veeramacheneni, and Ashwin Kayyoor
- STG313 | Building and optimizing a data lake on Amazon S3 – Ryan Blue, Huey Han, and Oleg Lvovitch
- ANT308 | Build large-scale transactional data lakes with Apache Iceberg on AWS – Aditya Challa, Aneesh Chandra, Dylan Qu, Francis McGregor-Macdonald, and Nishchai JM
- ANT328 | Accessing open table formats for superior data lake analytics – Asser Moustafa, Sreekanth Martha, and Stuti Deshpande
- OPN309 | Building a secure and scalable transactional data lake – Sercan Karaoglu and Ankita Gavali
- And while you’re at re:Invent, be sure to visit the following vendor booths to see what they’re doing with Iceberg
- AWS – Athena, EMR, Redshift, Glue, DynamoDB
- 2604 – Snowflake
- 1022 – Databricks
- 1632 – Tabular
- 1151 – Starburst
- 1000 – Confluent
- 1505 – Clickhouse
- 1530 – Dremio
Iceberg Resources
Get Started with Apache Iceberg
Learn more about Apache Iceberg on the official Apache site
Watch and subscribe to the Iceberg YouTube Channel
Read up on some community blog posts
Contribute to Iceberg
SELECT * FROM you JOIN iceberg_community
Subscribe to the Apache Iceberg mailing list
