[ARORO-3289] Avoid calling getMixedTablePartitionSpecById in the scan loop#3290
Conversation
|
@zhoujinsong @majin1102 Do you have time to review it? Thanks |
|
Will the old code trigger a reload of TableMetadata? It seems that if the reloading of TableMetadata is not triggered, the performance of the new code is the same as that of the old code. Iceberg TableMetadata also returns Specs Map objects from memory
|
|
@baiyangtx |
trigger CI Add getSpec to TableFileScanHelper
czy006
left a comment
There was a problem hiding this comment.
If there are Fire flame Graph or some unit test cases, it would be best.
|
@czy006 Flame graph after optimization:
|
zhoujinsong
left a comment
There was a problem hiding this comment.
Based on the information provided by the flame graph, Table#specs does not reload table metadata. From its internal implementation, this process is controlled by the shouldRefresh variable. However, this call does trigger the Kerberos relogin check logic, which is unnecessary.
So, I am +1 with this improvement.
Why are the changes needed?
Close #3289 .
How was this patch tested?
In my environment, there is a table with many partitions waiting to be merged. Each plan is very time-consuming. After optimization, the time-consuming is significantly reduced