Earlier this year, we released a blog post containing a docker compose configuration that allows you to easily get Iceberg and Spark up and running. It even provides a Jupyter notebook server with a number of fully-runnable example notebooks and walks through some of Iceberg’s biggest features. With the release of the Iceberg REST catalog, we’ve decided to update that example (which exists in this tabular-io/docker-spark-iceberg repo) to use the REST catalog. The easiest way to begin using this new setup containing the REST catalog is to clone the repo and run docker compose up. This blog post, however, aims to walk through what’s actually been changed and how the environment has been reconfigured to use a REST catalog.

REST Catalog

Iceberg clients ship with support for a number of different catalog implementations. Here’s a list of just some of them:

• JDBC catalog
• Hive catalog
• Nessie Catalog
• Hadoop catalog
• Glue catalog
• DynamoDB catalog

Although the wide array of Iceberg catalog implementations has allowed Iceberg support to grow over the years, the new REST catalog provides a much greater level of connectivity to different clients and applications. Furthermore, the REST catalog decreases commit conflicts using change-based commits and also allows the catalog to handle different version of Iceberg clients.

These are a few reasons why the Iceberg community developed the REST catalog implementation. The REST catalog specification allows a server to own all the catalog implementation details, while exposing a predictable REST API for Iceberg clients to connect to. In addition to making the catalog accessible to more languages and applications, standardizing on a REST API makes it much easier to build Iceberg clients in other languages.

Update to the Demo Environment

The latest tabulario/spark-iceberg image uses a REST catalog and a new spark-defaults.conf. You can run docker compose pull spark-iceberg to make sure you have the latest image.

The original docker compose environment used a Postgres backed JDBC catalog. The new setup uses the Iceberg REST server provided by our tabulario/iceberg-rest:0.2.0 image. In the repo, you’ll find the updated spark configuration that reflects this new configuration.

spark-defaults.conf

- spark.sql.catalog.demo.catalog-impl    org.apache.iceberg.jdbc.JdbcCatalog
- spark.sql.catalog.demo.uri             jdbc:postgresql://postgres:5432/demo_catalog
- spark.sql.catalog.demo.jdbc.user       admin
- spark.sql.catalog.demo.jdbc.password   password
+ spark.sql.catalog.demo.catalog-impl    org.apache.iceberg.rest.RESTCatalog
+ spark.sql.catalog.demo.uri             http://rest:8181

The iceberg-rest Docker Image

As previously described, the great thing about the REST catalog is that much of the catalog implementation details are handled by the server. That can be seen in action here where the iceberg-rest image simply exposes a REST interface for Spark to connect to. Spark doesn’t need to be concerned with connecting to the actual datastore which can be anything ranging from a Hive metastore to a MySQL database. By default, this image actually uses a SQLite database which is great for tests and small demos.

However, using something more powerful, such as a DynamoDB catalog, is just as easy!

rest:
  image: tabulario/iceberg-rest:0.2.0
  ports:
    - 8181:8181
  environment:
    - CATALOG_WAREHOUSE=s3a://warehouse/wh/
    - CATALOG_IO__IMPL=org.apache.iceberg.aws.dynamodb.DynamoDbCatalog
    - CATALOG_DYNAMODB_TABLE__NAME=demo_warehouse
    - CATALOG_S3_ENDPOINT=http://minio:9000

What’s Next?

The REST catalog is a big achievement for Iceberg and will contribute towards the goal of making Iceberg the open table format that’s supported by all query engines and frameworks. The catalog is a core component of an Iceberg backed data warehouse and making it accessible through a REST API enables integration of Iceberg into the wide ecosystem of data tools that a modern organization must inevitably adopt.