Skip to content

[SYSTEMDS-3970] Obtain Nnz Information of Reads before Rewriting Main Program Block - #2606

Closed
ywcb00 wants to merge 6 commits into
apache:mainfrom
ywcb00:feat/recompile/sparsity/read
Closed

ywcb00 wants to merge 6 commits into
apache:mainfrom
ywcb00:feat/recompile/sparsity/read

Conversation

@ywcb00

@ywcb00 ywcb00 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Hi,
This PR adds the rewriting functionality to split a permanent read operation into a separate, preceding statement block if the meta data of the read object does not provide information about its number of non-zeros beforehand. By performing the read separately, the sparsity information can already be leveraged when recompiling the main program block.

The PR also adds a flag to disable and enable this new feature (disabled by default) as well as unit tests to cover the functionality.

All the best,
David

@codecov

codecov Bot commented Sep 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.89189% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.81%. Comparing base (b1e3b9e) to head (c802b58).

Files with missing lines Patch % Lines
...ds/hops/rewrite/RewriteSplitDagUnknownNnzRead.java 91.37% 1 Missing and 4 partials ⚠️
src/main/java/org/apache/sysds/hops/Hop.java 0.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2606      +/-   ##
============================================
+ Coverage     71.38%   71.81%   +0.42%     
- Complexity    50619    50655      +36     
============================================
  Files          1638     2075     +437     
  Lines        196872   222176   +25304     
  Branches      38253    38264      +11     
============================================
+ Hits         140534   159549   +19015     
- Misses        45283    51581    +6298     
+ Partials      11055    11046       -9     
Flag Coverage Δ
java 71.40% <91.89%> (+0.02%) ⬆️
python 74.95% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…write rule to split off permanent reads into a dedicated statement block such that we can use the sparsity information about this data in the main statment block

feat(main/hops/rewrite/ProgramRewriter.java): add the new rewrite rule to the set of applied rewriters
…add unit test for splitting the statement block due to unknown sparsity of permanent read

feat(test/scripts/functions/rewrite/RewriteSplitDagUnknownNnz.dml): add dml script for unit test
…f sparsity rewrites are enabled (pointless otherwise)
…): print the sparsity information together with the individual matrix dimensions when logging the parenthesization of a matrix multiplication chain
@ywcb00
ywcb00 force-pushed the feat/recompile/sparsity/read branch from 90e8e4c to c802b58 Compare September 10, 2026 07:08
@ywcb00 ywcb00 closed this in 8c7d556 Sep 10, 2026
@github-project-automation github-project-automation Bot moved this from In Progress to Done in SystemDS PR Queue Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant