Fragment large TsBlocks in MPP DataExchange responses - #18645
Wei-hao-Li wants to merge 13 commits into
Conversation
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #18645 +/- ##
============================================
+ Coverage 42.68% 42.89% +0.20%
Complexity 442 442
============================================
Files 5441 5452 +11
Lines 393714 395864 +2150
Branches 51597 51874 +277
============================================
+ Hits 168051 169794 +1743
- Misses 225663 226070 +407 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Signed-off-by: Weihao Li <18110526956@163.com>
JackieTien97
left a comment
There was a problem hiding this comment.
Please address the SourceHandle cancellation and allocation-failure handling described inline before merging, and correct the new configuration warning placeholders.
Validation: a clean reactor build with SourceHandleTest, SinkChannelTest, and PropertiesTest passed all 21 tests. Additional focused tests reproduced the three issues documented in the inline comments.
| while (!fetchProgress.isFinished()) { | ||
| TGetDataBlockRequest fragmentRequest = request.deepCopy(); | ||
| fragmentRequest.setStartSequenceId(fetchProgress.nextSequenceId); | ||
| fragmentRequest.setOffset(fetchProgress.offset); | ||
| TGetDataBlockResponse response = client.getDataBlock(fragmentRequest); |
There was a problem hiding this comment.
[P2] Stop fetching fragments when the source is closed or aborted
If close() or abort() runs while a fragment RPC is in flight, this loop still consumes its response and downloads the remaining fragments because it only checks fetchProgress.isFinished(). The source has already returned its reservation to the query pool, so it can allocate the full PublicBAOS(totalLength) and retain accumulated data after cancellation without an accounted reservation. A focused test that closes or aborts during the first RPC of a three-fragment transfer still executes all three RPCs with getBufferRetainedSizeInBytes() == 0. For large blocks this also keeps the exchange worker and connection occupied until the entire range is fetched. Please check the terminal state with appropriate synchronization before each new RPC and again before consuming its response, and discard the pending fetch state when stopping.
| } | ||
| fail(e); | ||
| return; | ||
| } catch (Exception e) { |
There was a problem hiding this comment.
[P2] Preserve failure cleanup for errors during fragment assembly
Narrowing the previous catch (Throwable) to catch (Exception) leaves an OutOfMemoryError from the new PublicBAOS(totalLength) allocation uncaught. This allocation happens after client.getDataBlock() returns, so the Thrift client wrapper cannot convert it into a TException. The submitted task then terminates without calling fail(): the executor only logs the error, leaving the source blocked and its reservation held until an outer timeout or cancellation. Using the JVM array-size limit to safely inject an allocation error reproduced zero failure callbacks, a blocked source, and a retained 1,024-byte reservation. Please also notify the fragment of failure and release the reservation on the Error path; failing immediately is sufficient and does not require retrying an OOM.
| LOGGER.warn( | ||
| DataNodeMiscMessages | ||
| .LOG_MPP_DATA_EXCHANGE_MAX_PAYLOAD_SIZE_ARG_IS_NOT_POSITIVE_USING_DEFAULT_VALUE_ARG_1AA821B2, | ||
| configuredSize, | ||
| DEFAULT_MPP_DATA_EXCHANGE_MAX_PAYLOAD_SIZE_IN_BYTES); |
There was a problem hiding this comment.
[P3] Use the correct placeholders for the configuration warnings
The same placeholder mismatch occurs at all three warning calls in loadMppDataExchangeMaxPayloadSize():
- Non-positive configured value, lines 3102–3106.
- Below the minimum value, lines 3109–3114.
- Above the maximum value, lines 3119–3124.
All three new message constants contain %d, but LOGGER.warn(template, args...) performs SLF4J {} substitution. For example, the first warning is emitted as mpp_data_exchange_max_payload_size_in_bytes (%d) is not positive, using default value %d, without the configured or effective values. Both the English and Chinese templates are affected. Please use {} in the templates, or explicitly apply String.format() before logging, so operators can see which values were rejected and applied.
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
No description provided.