Skip to content

[AMORO-3385][bug]fix SparkUnifiedSessionCatalog should support load hive function for hive/mixed-hive table#3385

Merged
zhoujinsong merged 1 commit into
apache:masterfrom
Aireed:support_hive_function
Jan 20, 2025
Merged

[AMORO-3385][bug]fix SparkUnifiedSessionCatalog should support load hive function for hive/mixed-hive table#3385
zhoujinsong merged 1 commit into
apache:masterfrom
Aireed:support_hive_function

Conversation

@Aireed
Copy link
Copy Markdown
Contributor

@Aireed Aireed commented Jan 2, 2025

Why are the changes needed?

Environment:
format: mixed_hive/hive
Engine: spark 3.3
Catalog: SparkUnifiedSessionCatalog with mixed_hive format

bug:
Spark SQL cannot recognize a successfully registered Hive UDF

description

spark-sql> reload jar hdfs:///tmp/spark_udf-1.0-SNAPSHOT.jar;
spark-sql> create function hive_db.StringUdf as 'org.example.udf.StringUdf' using jar 'hdfs:///tmp/spark_udf-1.0-SNAPSHOT.jar';
Error in query: Function 'hive_db.StringUdf' already exists in database 'hive_db'
spark-sql> select id,hive_db.StringUdf(name) from amoro_like;
Error in query: Undefined function: hive_db.StringUdf; line 1 pos 10
spark-sql> select id,hive_db.StringUdf(name) from tbl_schema_chg;
Error in query: Undefined function: hive_db.StringUdf; line 1 pos 10
spark-sql>

Brief change log

  • fix load_function logic

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@Aireed Aireed added the type:bug Something isn't working label Jan 2, 2025
@github-actions github-actions Bot added the module:mixed-spark Spark module for Mixed Format label Jan 2, 2025
@Aireed Aireed changed the title [AMORO-xxx]: fix SparkUnifiedSessionCatalog should support load hive function for hive/mixed-hive table [AMORO-3385][bug]fix SparkUnifiedSessionCatalog should support load hive function for hive/mixed-hive table Jan 2, 2025
@Aireed Aireed force-pushed the support_hive_function branch from 9463801 to 4eb591c Compare January 2, 2025 07:49
Copy link
Copy Markdown
Contributor

@baiyangtx baiyangtx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhoujinsong zhoujinsong merged commit 44f41df into apache:master Jan 20, 2025
zhoujinsong pushed a commit to zhoujinsong/amoro that referenced this pull request Jan 21, 2025
…ive function for hive/mixed-hive table (apache#3385)

fix: unifiedCatalog should support load hive function
@Aireed Aireed deleted the support_hive_function branch May 15, 2025 07:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:mixed-spark Spark module for Mixed Format type:bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants