Skip to content

[AMORO-3383] fix: unified catalog should support iceberg native create_changelog_view procedure#3383

Merged
zhoujinsong merged 2 commits into
apache:masterfrom
Aireed:changelog_view
Jan 10, 2025
Merged

[AMORO-3383] fix: unified catalog should support iceberg native create_changelog_view procedure#3383
zhoujinsong merged 2 commits into
apache:masterfrom
Aireed:changelog_view

Conversation

@Aireed
Copy link
Copy Markdown
Contributor

@Aireed Aireed commented Jan 2, 2025

Why are the changes needed?

The unified catalog does not support the Iceberg change metadata table syntax, which makes it impossible to support the create_changelog_view stored procedure.

spark-sql> CALL spark_catalog.system.create_changelog_view(
         >   table => 'iceberg_v7.iceberg_overwrite',
         >   options => map('start-snapshot-id','486064778759746866','end-snapshot-id', '1216062697090990172'),
         >   changelog_view => 'my_changelog_view_4'
         > );
ANTLR Tool version 4.9.3 used for code generation does not match the current runtime version 4.8ANTLR Runtime version 4.9.3 used for parser compilation does not match the current runtime version 4.8ANTLR Tool version 4.9.3 used for code generation does not match the current runtime version 4.8ANTLR Runtime version 4.9.3 used for parser compilation does not match the current runtime version 4.8
Error in query: spark_catalog requires a single-part namespace, but got [iceberg_v7, iceberg_overwrite]

Brief change log

  • add changes subtablename in class IcebergSparkFormat

Apart from this, I haven't found any other situations that need to be addressed. As the version upgrades, it will be necessary to align with the list of subTables supported by Iceberg.

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@Aireed Aireed changed the title [AMORO-xxx] fix: unified catalog should support iceberg native create_changelog_view procedure [AMORO-3383] fix: unified catalog should support iceberg native create_changelog_view procedure Jan 2, 2025
@github-actions github-actions Bot added the module:mixed-spark Spark module for Mixed Format label Jan 2, 2025
@Aireed Aireed added the type:bug Something isn't working label Jan 2, 2025
Copy link
Copy Markdown
Contributor

@zhoujinsong zhoujinsong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Thanks for the contriburion!

@zhoujinsong zhoujinsong merged commit 8d5e5ef into apache:master Jan 10, 2025
zhoujinsong pushed a commit to zhoujinsong/amoro that referenced this pull request Jan 21, 2025
…e_changelog_view procedure (apache#3383)

* unified catalog support iceberg create_changelog_view procedure

* unified catalog support iceberg create_changelog_view procedure
@Aireed Aireed deleted the changelog_view branch May 15, 2025 07:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:mixed-spark Spark module for Mixed Format type:bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants