Prefer local DataNode when LOAD creates DataRegions - #18663
Open
luoluoyuyu wants to merge 3 commits into
Open
luoluoyuyu wants to merge 3 commits into
luoluoyuyu wants to merge 3 commits into
Conversation
Caideyipi
reviewed
Sep 17, 2026
Caideyipi
left a comment
Collaborator
There was a problem hiding this comment.
发现 3 个需要关注的问题,详见 inline comments。
Caideyipi
approved these changes
Sep 18, 2026
Collaborator
There was a problem hiding this comment.
复查当前提交 1b5eb12,未发现新的阻塞问题。
之前的 3 条问题均已修复:preferredDataNodeId 已仅保留在分区请求中,数据库 schema 不再持久化或经 Pipe 传播该字段;Tablet 转换的 tree/table 路径均能在同步 INSERT 分析期间传递 hint,并在 finally 中清理 ThreadLocal。已检查普通请求/关闭开关时的默认行为,以及 hint 不会将候选集合之外的 DataNode 强行加入副本集,移除中的节点仍被排除。
验证依据:代码调用链复查、diff 检查及当前提交的 CI。相关 LOAD 单测、Linux 单测、集群 IT 和中英文编译检查通过;Windows DataNode 单测唯一失败为 WALNodeWaitForRollFileTest.testWaitForNextReadyNotWokenByFlushWithoutRoll 的 10 秒等待超时,其调用链不涉及本次改动。本次未另行在本地运行测试。
非阻塞说明:PR 描述第 2/3 节仍写着数据库 schema 持久化和 fallback,建议同步为当前的请求级 hint 设计。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a hidden, hot-reloadable DataNode configuration
load_tsfile_prefer_local_node(defaulttrue). When enabled, LOAD passes the current DataNodeId to ConfigNode aspreferredDataNodeIdwhile creating a database or requesting get-or-create data partitions. ConfigNode then prefers placing newly created DataRegion replicas on the requesting node without modifying the existing region allocation algorithms themselves.Design / How It Works
1. DataNode configuration
IoTDBConfig.loadTsFilePreferLocalNodeis a volatile boolean, defaulttrue.IoTDBDescriptorloads it both at startup and during hot reload.iotdb-system.properties.template, so it stays hidden fromSHOW CONFIGURATION.2. Database creation path
When LOAD auto-creates a database:
TreeSchemaAutoCreatorAndVerifiersetspreferredDataNodeIdonDatabaseSchemaStatement.DatabaseSchemaTask.constructDatabaseSchemacopies it toTDatabaseSchema.preferredDataNodeId.RegionBalancercan later read it.3. Data-partition creation path
This covers the case where the database already exists and LOAD creates devices/tables or requests an unassigned series partition, which triggers DataRegion extension:
LoadTsFileScheduler.DataPartitionBatchFetcherreadsload_tsfile_prefer_local_node.IPartitionFetcher.getOrCreateDataPartition(..., preferredDataNodeId)overload.ClusterPartitionFetcherputs the value intoTDataPartitionReq.preferredDataNodeId.GetOrCreateDataPartitionPlan.PartitionManagerpasses the hint down to DataRegion extension.RegionBalancerprefers the request-level hint first, then falls back to the database-schema hint.4. RegionBalancer preference handling
The original allocator is still called first:
Only after that,
preferDataNodeIfPossibleadjusts the result:Replication factor, candidate validity, and region identity are preserved.
5. New time partitions
New time partitions do not create a new DataRegion. A new time slot is mapped into an existing DataRegion for the same series-partition slot, so this path needs no preference change.
Validation
Result:
BUILD SUCCESS.