Skip to content

Fragment large TsBlocks in MPP DataExchange responses - #18645

Open
Wei-hao-Li wants to merge 13 commits into
masterfrom
mppEx
Open

Wei-hao-Li wants to merge 13 commits into
masterfrom
mppEx

Conversation

@Wei-hao-Li

Copy link
Copy Markdown
Collaborator

No description provided.

Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
@codecov

codecov Bot commented Sep 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 78.57143% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 42.89%. Comparing base (25ab794) to head (c02cd11).
⚠️ Report is 36 commits behind head on master.

Files with missing lines Patch % Lines
...ine/execution/exchange/MPPDataExchangeManager.java 0.00% 29 Missing ⚠️
...engine/execution/exchange/source/SourceHandle.java 89.01% 10 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #18645      +/-   ##
============================================
+ Coverage     42.68%   42.89%   +0.20%     
  Complexity      442      442              
============================================
  Files          5441     5452      +11     
  Lines        393714   395864    +2150     
  Branches      51597    51874     +277     
============================================
+ Hits         168051   169794    +1743     
- Misses       225663   226070     +407     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Weihao Li <18110526956@163.com>

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the SourceHandle cancellation and allocation-failure handling described inline before merging, and correct the new configuration warning placeholders.

Validation: a clean reactor build with SourceHandleTest, SinkChannelTest, and PropertiesTest passed all 21 tests. Additional focused tests reproduced the three issues documented in the inline comments.

Comment on lines +788 to +792
while (!fetchProgress.isFinished()) {
TGetDataBlockRequest fragmentRequest = request.deepCopy();
fragmentRequest.setStartSequenceId(fetchProgress.nextSequenceId);
fragmentRequest.setOffset(fetchProgress.offset);
TGetDataBlockResponse response = client.getDataBlock(fragmentRequest);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Stop fetching fragments when the source is closed or aborted

If close() or abort() runs while a fragment RPC is in flight, this loop still consumes its response and downloads the remaining fragments because it only checks fetchProgress.isFinished(). The source has already returned its reservation to the query pool, so it can allocate the full PublicBAOS(totalLength) and retain accumulated data after cancellation without an accounted reservation. A focused test that closes or aborts during the first RPC of a three-fragment transfer still executes all three RPCs with getBufferRetainedSizeInBytes() == 0. For large blocks this also keeps the exchange worker and connection occupied until the entire range is fetched. Please check the terminal state with appropriate synchronization before each new RPC and again before consuming its response, and discard the pending fetch state when stopping.

}
fail(e);
return;
} catch (Exception e) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve failure cleanup for errors during fragment assembly

Narrowing the previous catch (Throwable) to catch (Exception) leaves an OutOfMemoryError from the new PublicBAOS(totalLength) allocation uncaught. This allocation happens after client.getDataBlock() returns, so the Thrift client wrapper cannot convert it into a TException. The submitted task then terminates without calling fail(): the executor only logs the error, leaving the source blocked and its reservation held until an outer timeout or cancellation. Using the JVM array-size limit to safely inject an allocation error reproduced zero failure callbacks, a blocked source, and a retained 1,024-byte reservation. Please also notify the fragment of failure and release the reservation on the Error path; failing immediately is sufficient and does not require retrying an OOM.

Comment on lines +3102 to +3106
LOGGER.warn(
DataNodeMiscMessages
.LOG_MPP_DATA_EXCHANGE_MAX_PAYLOAD_SIZE_ARG_IS_NOT_POSITIVE_USING_DEFAULT_VALUE_ARG_1AA821B2,
configuredSize,
DEFAULT_MPP_DATA_EXCHANGE_MAX_PAYLOAD_SIZE_IN_BYTES);

@JackieTien97 JackieTien97 Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Use the correct placeholders for the configuration warnings

The same placeholder mismatch occurs at all three warning calls in loadMppDataExchangeMaxPayloadSize():

  1. Non-positive configured value, lines 3102–3106.
  2. Below the minimum value, lines 3109–3114.
  3. Above the maximum value, lines 3119–3124.

All three new message constants contain %d, but LOGGER.warn(template, args...) performs SLF4J {} substitution. For example, the first warning is emitted as mpp_data_exchange_max_payload_size_in_bytes (%d) is not positive, using default value %d, without the configured or effective values. Both the English and Chinese templates are affected. Please use {} in the templates, or explicitly apply String.format() before logging, so operators can see which values were rejected and applied.

Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants