Iceberg on S3: Faster Queries with Sort & Z-Order
- Amazon S3 Tables now supports sort and z-order compaction to improve apache Iceberg query performance in both Amazon S3 tables and general-purpose S3 buckets.
- Iceberg is used to manage datasets in Amazon simple Storage Service (Amazon S3) with AWS Glue Data Catalog or S3 Tables, supporting concurrent streaming, batch ingestion, schema evolution...
- The new compaction strategies, sort and z-order, help organize data more efficiently.
Amazon S3 Tables significantly boost Apache Iceberg query performance with the integration of sort and z-order compaction strategies. This upgrade enhances efficiency for analytical datasets within Amazon S3, offering notable gains in query speed and data institution. These advancements, available in both S3 Tables and general-purpose S3 buckets, streamline data management that reduces query costs by efficiently organizing your data, leading to up to threefold improvement. News Directory 3 covers the benefits of sort and z-order, and how you can execute these options. These features are accessible across all supported AWS Regions. Discover what’s next for streamlining your S3 queries.
Amazon S3 Tables Boost Apache Iceberg Query Performance
Updated June 27, 2025
Amazon S3 Tables now supports sort and z-order compaction to improve apache Iceberg query performance in both Amazon S3 tables and general-purpose S3 buckets. This enhancement addresses the challenges of managing large-scale analytical datasets, making data organization more efficient.
Iceberg is used to manage datasets in Amazon simple Storage Service (Amazon S3) with AWS Glue Data Catalog or S3 Tables, supporting concurrent streaming, batch ingestion, schema evolution and time travel. Datasets with high ingest rates or frequent updates often accumulate small files, impacting query costs and performance. Optimizing Iceberg data layouts can be complex, frequently enough requiring custom pipelines. While the default binpack strategy offers improvements, the new sort and z-order compaction options provide greater gains for queries filtering across dimensions.
The new compaction strategies, sort and z-order, help organize data more efficiently. These strategies are available for fully managed S3 Tables and Iceberg tables in general-purpose S3 buckets through AWS Glue optimizations.
To set the table sort order, users can execute the following command:
spark.sql("ALTER TABLE ice_catalog.testnamespace.testtable WRITE ORDERED BY name ASC")Table compaction can be enabled using the following command:
aws s3tables put-table-maintenance-configuration --table-bucket-arn ${S3TABLE_BUCKET_ARN} --namespace testnamespace --name testtable --type icebergCompaction --value "status=enabled,settings={icebergCompaction={strategy=sort}}"The compaction status can be checked with:
aws s3tables get-table-maintenance-job-status --table-bucket-arn ${S3TABLE_BUCKET_ARN} --namespace testnamespace --name testtableTests showed performance improvements of threefold or more when switching from binpack to sort or z-order, depending on data layout and query patterns.
What’s next
The sort and z-order compaction strategies are available in all AWS Regions where Amazon S3 Tables are supported.There are no additional charges for S3 Tables beyond existing usage and maintenance fees. Compute charges apply during compaction for Data Catalog optimizations. These changes allow queries that filter on the sort or z-order columns to benefit from faster scan times and reduced engine costs.