added support for iceberg tables on S3 table buckets using athena engine#5774
Open
vishnuprasad3842 wants to merge 1 commit intoSQLMesh:mainfrom
Open
added support for iceberg tables on S3 table buckets using athena engine#5774vishnuprasad3842 wants to merge 1 commit intoSQLMesh:mainfrom
vishnuprasad3842 wants to merge 1 commit intoSQLMesh:mainfrom
Conversation
Signed-off-by: vishnu <vishnu.t@redbus.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Added support for Amazon S3 Tables (managed Iceberg tables) within the Athena Engine Adapter.
Specific changes include:
Catalog Support: Updated AthenaEngineAdapter.catalog_support to CatalogSupport.FULL_SUPPORT to allow cross-catalog queries without issuing SET CATALOG commands. This prevents SQLMesh from stripping out custom catalog names (e.g., s3tablescatalog/...) from model names.
S3 Table Partitioning: For S3 tables, partition properties are now correctly generated as partitioning=ARRAY['...'] instead of the standard partitioned_by property format.
S3 Table Properties: S3 Table CTAS queries in Athena implicitly default to table_type='ICEBERG' and do not allow explicit specification of most table properties (except format='PARQUET'). The adapter now filters out unsupported TBLPROPERTIES and table_type when generating CREATE TABLE ... AS statements for S3 tables to prevent pyathena.error.OperationalError.
Dictionary Mutation Bug: Fixed a subtle bug in _build_table_properties_exp where table_properties was being mutated during AST parsing, causing properties like s3_table to drop out during intermediate schema table creation.
Test Plan
Tested creating S3 Iceberg tables via sqlmesh plan utilizing custom MODEL properties (s3_table true / catalog defined).
Verified partitioning is properly formulated as an unquoted ARRAY structure for S3 tables.
Verified that temporary intermediate schema generation properly inherits S3 properties without failure.
Ran make fast-test to ensure no regressions with standard Hive/Iceberg tables in the Athena adapter
Checklist
I have run make style and fixed any issues
I have added tests for my changes (if applicable)
All existing tests pass (make fast-test)
My commits are signed off (git commit -s) per the DCO