Amazon Aurora PostgreSQL now supports direct querying of Apache Iceberg and Parquet data in your data lake

Summary

• Amazon Aurora PostgreSQL now supports direct querying of operational data alongside data stored in Apache Iceberg and Apache Parquet formats within data lakes.
• This capability eliminates the need for extract, transform, & load (ETL) pipelines to move structured data from data lakes into operational databases.
• Users can query data from Iceberg REST Catalog (IRC)-compatible catalogs managed in AWS Glue Data Catalog without moving or duplicating data.
• The DuckDB project team joined Amazon to integrate DuckDB efficiency directly into Aurora PostgreSQL for querying live operational data & data lakes in a single query.
• Query processing remains within Aurora with no additional network hops & no ETL pipelines that duplicate data.
• The feature supports Apache Iceberg tables managed through AWS Glue Data Catalog, as well as Parquet & Iceberg data stored in Amazon S3 & S3 Tables.
• The capability is supported on Aurora PostgreSQL major versions 17 (starting with 17.11) & 18 (starting with 18.6).
• Setup requires creating an Aurora PostgreSQL cluster, attaching an IAM role with the AuroraAnalytics feature, & enabling the aurora_analytics extension.
• Users create foreign tables pointing to Iceberg or Parquet data in the data lake & query them using familiar PostgreSQL syntax.
• Setup can be completed through the Amazon RDS console or with any PostgreSQL client such as psql.
• External IRC-compatible catalogs can be queried through AWS Glue Data Catalog federation by registering the external catalog once with Glue.
• A single query can join data stored in Aurora with Iceberg tables registered across multiple catalogs for a unified view.
• Optimizations include predicate pushdown & column pruning to read only relevant data, keeping queries efficient as data grows.
• Frequently accessed data is cached in the Aurora instance to return faster results for subsequent queries.
• Metrics such as rows scanned, bytes read from Amazon S3, & cache hits can be inspected using the aurora_analytics_stat_statements() function.
• Aurora automatically reads schema from Parquet file metadata, allowing users to create foreign tables without manually defining columns.
• The IMPORT FOREIGN SCHEMA statement can bulk-create foreign tables for every Iceberg or Parquet table in an AWS Glue Data Catalog database.
• Users can materialize data from the data lake into a native Aurora PostgreSQL table using commands like CREATE TABLE AS SELECT, INSERT INTO ... SELECT, or MERGE INTO.
• Materialized tables live in Aurora & are queried like any other PostgreSQL table to provide a low-latency path for hot data.
• Read queries for materialized tables can run on any Aurora PostgreSQL instance in the cluster, including read replicas.
• Direct querying of Apache Iceberg & Parquet data is available today in all commercial AWS Regions & AWS GovCloud (US) Regions.
• There is no additional charge for the capability; users pay only for incremental Aurora compute consumed & Amazon S3 request costs.