Iceberg Updates
- Flink support for inspecting metadata tables (Liwei Li)
- Flink write and read support for branch and tag (Amogh and Liwei Li)
- Flink read and write support for Avro GenericRecord (Steven Wu)
- Implemented branch commits for all operations (Namratha Keshavaprakash & Amogh Jahagirdar)
- DDL support for tags and branches (Amogh, Liwei, Xuwei, and Chidayong)
- Added branch/tag support to VERSION AS OF in Spark (Jack Ye)
- Added position deletes metadata table (Szehon Ho)
- Added a Snowflake catalog (Dennis Huo)
- REST catalog supports lazy snapshot loading (Daniel Weeks)
- Added DynamoDB catalog support in Python (Armin)
- Updated Spark to remove filters that are completely pushed down (Anton Okolnychyi)
- Added Delta to Iceberg table conversion (Rushan and Eric)
- Added SQL view representation implementation (Amogh)
- Added S3 REST Signer client + REST spec (Eduard Tudenhoefner)
Releases
1.2.0
- https://github.com/apache/iceberg/milestone/28
- Spark row-level commands and branches
PyIceberg updates
Version 0.3.0 was released. The new release includes the following, for more details, please refer to the PRs:
- Full projection by Iceberg Field ID
- Parallelization of job planning that speeds up the metadata operations by an order of magnitude
- Overhaul of reading Avro files without the use of Pydantic to improve performance
- Bugfix when reading a zero-length binary field
- Bugfix for PyArrow that led to multiple calls to the filesystem.
- Added Python support for static tables (Ruben)
More information can be found on the project site, and the installer can be found here
Iceberg in the industry
- Streamsets is adopting Iceberg support
- ClickHouse has added Iceberg support
- Databend has added Iceberg support to Datafuse
Blogs from the community
- Starburst - Automated maintenance for Apache Iceberg tables in Starburst Galaxy
- Dremio - Dealing with Data Incidents Using the Rollback Feature in Apache Iceberg
- Ancestry - Scaling Ancestry.com: How to Optimize Updates for Iceberg Tables with 100 Billion Rows
Iceberg in the news
- Datanami: Open Table Formats Square Off in Lakehouse Data Smackdown
- BCG: A New Architecture to Manage Data Costs and Complexity
- Forbes: An Open Approach To Hybrid Data Clouds
- InfoQ: Netflix Built a Scalable Annotation Service Using Cassandra, Elasticsearch and Iceberg
Keep up to date on all things iceberg
- Watch for new blog posts added to the Blogs page
- See the community Contribute guide to learn how to start contributing to Iceberg
- Join the Apache Iceberg workspace on Slack using the invite link
- Subscribe to the Apache Iceberg mailing list