Amazon S3 Tables now offer managed migration of Apache Hive tables to Apache Iceberg

Summary

• Amazon S3 Tables now support managed migration of Apache Hive tables to Apache Iceberg.
• Migration works with Parquet tables in Amazon S3 registered in the AWS Glue Data Catalog.
• The process allows data lake upgrades without building pipelines or provisioning compute while production workloads continue running.
• Apache Iceberg serves as an open standard for large analytics datasets with features like schema evolution, hidden partitioning, and time travel queries.
• Amazon S3 Tables provide fully managed storage with automatic maintenance, optimization, replication, and S3 Intelligent-Tiering.
• The managed migration replaces weeks of engineering work with an API-driven operation.
• S3 Tables handle data validation, compatibility checks, type conversions, file recompression, and compaction.
• A compatibility report is generated before copying data to identify every source object in scope.
• Users can sync data as many times as needed without stopping writes to the source table, except for a few minutes during cutover.
• Costs are based only on the data copied, with no charge for the Hive-to-Iceberg conversion.
• Source objects in S3 Intelligent-Tiering remain in their current access tier.
• Migration can be initiated via the S3 console by choosing Create table from migration.
• Destination tables are created with recommended default settings for query performance and storage efficiency.
• The destination table is read-only during migration and promoted to read-write upon completion.
• Data can be queried immediately using Amazon Athena, Amazon SageMaker Unified Studio, or tools compatible with the Apache Iceberg REST API.
• AWS CLI, AWS SDKs, and the Agent Toolkit for AWS can be used to set up migration jobs.
• The start-table-migration command initiates the migration workflow.
• The sync-table-migration command updates the destination table with changes from the source.
• The migrating-hive-tables-to-s3-tables skill in the Agent Toolkit for AWS provides guidance for coding agents.
• The skill analyzes IAM policies, validates migration plans, executes API calls, and monitors job progress.
• Amazon S3 Tables managed migration is available in US East (N. Virginia, Ohio), US West (Oregon), Asia Pacific (Mumbai, Singapore, Sydney, Tokyo), and Europe (Frankfurt, Ireland, Stockholm) Regions.
• At launch, the capability supports AWS Glue-registered Hive tables stored in Parquet format within the same AWS account and AWS Region.