AWS Glue 6.0 Released: Apache Iceberg v3, Spark 4.1, and 30% Lower Pricing
- AWS Glue 6.0 is now generally available with a 30% price reduction over previous versions, introducing full support for Apache Iceberg v3 features and a modernized runtime built...
- AWS Glue 6.0 delivers the complete Apache Iceberg v3 specification built on Iceberg 1.11.0.
- The release incorporates major upgrades from Spark 4.1, Python 3.13, and Scala 2.13.
AWS Glue 6.0 is now generally available with a 30% price reduction over previous versions, introducing full support for Apache Iceberg v3 features and a modernized runtime built on Apache Spark 4.1, Python 3.13, and Scala 2.13, according to an announcement from the AWS News Blog.
Apache Iceberg v3 Support and the VARIANT Data Type
AWS Glue 6.0 delivers the complete Apache Iceberg v3 specification built on Iceberg 1.11.0. The headline addition is the VARIANT data type with shredding support, which achieves faster query read performance compared to traditional string data type columns for semi-structured data, according to the AWS News Blog. With VARIANT shredding, teams can store and query JSON, logs, and event data without flattening schemas. This approach eliminates duplicate data copies, custom parsing code, and pipeline breakage when schemas change. Additional Iceberg v3 capabilities include native spatial processing for GIS analytics via geometry and geography data types, nanosecond-precision timestamps for IoT sensors and high-frequency financial workloads, and unknown type handling to process data with evolving schemas without pipeline failures.
Modern Runtime Enhancements in Spark 4.1
The release incorporates major upgrades from Spark 4.1, Python 3.13, and Scala 2.13. Spark Declarative Pipelines introduce a simplified approach to ETL authoring where data engineers declare transformations while the engine automatically determines execution order and optimization. AWS Glue 6.0 also introduces Arrow-native execution for Python User-Defined Functions (UDFs) and User-Defined Table Functions (UDTFs). This eliminates serialization overhead between Python and the JVM, improving PySpark performance for complex transformations. For stateless streaming use cases, a new real-time streaming mode achieves single-digit millisecond latency. Built on Spark 4.1’s Real-Time Mode with Glue-optimized execution, it supports real-time event processing and time-sensitive data routing.
Getting Started and Pricing
No API changes are required to use AWS Glue 6.0. Users can select the new version using the existing –glue-version parameter in create-job or update-job APIs through the AWS Command Line Interface (AWS CLI), AWS SDK, AWS Glue Studio, Amazon SageMaker Unified Studio, and preferred IDEs. Pricing features an hourly rate billed by the second for crawlers and ETL jobs. For the AWS Glue Data Catalog, users pay a simplified monthly fee for storing and accessing metadata, with the first million objects stored and the first million accesses provided free of charge, according to the AWS News Blog.
