Skip to content

[opt](lance) reduce redundant work in LanceTableReader - #68335

Open
zhangstar333 wants to merge 2 commits into
apache:branch-4.1from
zhangstar333:lance-rowid
Open

zhangstar333 wants to merge 2 commits into
apache:branch-4.1from
zhangstar333:lance-rowid

Conversation

@zhangstar333

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:

optimize LanceTableReader by lazily initializing schemas and scanner metrics, reusing the record batch converter, caching Arrow normalization plans, and avoiding unnecessary full-column reads. Also improve code structure by extracting schema import and record batch conversion logic.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhangstar333

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR #68335 at head c64d2d7 (base 70bf96e).

Two actionable regressions are called out inline:

  • Nested registered-extension arrays: the new normalizer compacts a sliced parent before unwrapping extension children, so Arrow 24 cannot build list/struct parents containing ExtensionType children; the existing sliced nested JSON regression path fails.
  • Row-ID/take_rows lifecycle: scanner profile counters are now lazy, so scanner-less phase-two fetches skip Lance data-cache byte statistics that were previously created and collected.

Review checkpoints:

  • All ten authoritative changed paths were swept, including the converter extraction, nested normalization, scanner projection, generated search-result columns, global row IDs, runtime-filter/schema gating, close/reset paths, and test changes.
  • Projection, global-rowid, runtime-filter, normal scanner reuse, build integration, and other candidate concerns were independently checked; no additional actionable issue was substantiated.
  • Existing inline review comments were empty at the final duplicate-fence check, and the live base/head matched the authoritative bundle.
  • No user-provided focus points were supplied beyond the complete-review requirement.
  • This is a static review only: no builds or tests were run, so author/CI claims are not independent validation.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR #68335 at head c64d2d7 (base 70bf96e).

Two actionable regressions are called out inline:

  • Nested registered-extension arrays: the new normalizer compacts a sliced parent before unwrapping extension children, so Arrow 24 cannot build list/struct parents containing ExtensionType children; the existing sliced nested JSON regression path fails.
  • Row-ID/take_rows lifecycle: scanner profile counters are now lazy, so scanner-less phase-two fetches skip Lance data-cache byte statistics that were previously created and collected.

Review checkpoints:

  • All ten authoritative changed paths were swept, including the converter extraction, nested normalization, scanner projection, generated search-result columns, global row IDs, runtime-filter/schema gating, close/reset paths, and test changes.
  • Projection, global-rowid, runtime-filter, normal scanner reuse, build integration, and other candidate concerns were independently checked; no additional actionable issue was substantiated.
  • Existing inline review comments were empty at the final duplicate-fence check, and the live base/head matched the authoritative bundle.
  • No user-provided focus points were supplied beyond the complete-review requirement.
  • This is a static review only: no builds or tests were run, so author/CI claims are not independent validation.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR #68335 at head c64d2d7 (base 70bf96e).

Two actionable regressions are called out inline:

  • Nested registered-extension arrays: the new normalizer compacts a sliced parent before unwrapping extension children, so Arrow 24 cannot build list/struct parents containing ExtensionType children; the existing sliced nested JSON regression path fails.
  • Row-ID/take_rows lifecycle: scanner profile counters are now lazy, so scanner-less phase-two fetches skip Lance data-cache byte statistics that were previously created and collected.

Review checkpoints:

  • All ten authoritative changed paths were swept, including the converter extraction, nested normalization, scanner projection, generated search-result columns, global row IDs, runtime-filter/schema gating, close/reset paths, and test changes.
  • Projection, global-rowid, runtime-filter, normal scanner reuse, build integration, and other candidate concerns were independently checked; no additional actionable issue was substantiated.
  • Existing inline review comments were empty at the final duplicate-fence check, and the live base/head matched the authoritative bundle.
  • No user-provided focus points were supplied beyond the complete-review requirement.
  • This is a static review only: no builds or tests were run, so author/CI claims are not independent validation.

Comment thread be/src/format_v2/lance/lance_reader_helper.cpp Outdated
Comment thread be/src/format_v2/table/lance_reader.cpp
@zhangstar333

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 82.84% (309/373) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 74.58% (31510/42250)
Line Coverage 58.76% (350959/597290)
Region Coverage 55.45% (292723/527877)
Branch Coverage 56.32% (132097/234560)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants