[codex] Parallelize incremental parse step - #1052

@Extrodox is attempting to deploy a commit to the shashankss1205's projects Team on Vercel.

A member of the Team first needs to authorize it.

Overlaps merged #1071 watcher sync and conflicts with current watcher.py.

Rebased onto current main and pushed to your branch — it was 64 days stale and watcher.py has had 15 commits since you branched from v0.4.13. Suite is green (1111 passed, 7 skipped). Two things I had to resolve:

1. The conflict in _handle_modification. Kept your side — capturing the return of update_file_in_graph to reuse the parse. I checked that still works: graph_builder.py:1015/1019 returns file_data, so the reuse optimisation is valid against current main.
2. Your new test failed with AttributeError: 'RepositoryEventHandler' object has no attribute '_update_lock'. main added _update_lock/_timers_lock in fix(watcher): handle both endpoints of a rename; serialise concurrent updates #1424 after you branched, and tests building the handler via __new__ must supply them — the neighbouring tests in that file already do. Added the same two lines.

Why I have not merged it

The parallel parsing hits an open segfault bug — #1370.

That issue reports non-deterministic Fatal Python error: Segmentation fault when multiple worker threads call tree_sitter_manager.execute_query() concurrently on different languages. Its traceback shows three threads inside execute_query() at once (TypeScript, Python, HTML).

Your ThreadPoolExecutor runs graph_builder.parse_file across affected files, and in a mixed-language repo the affected set spans languages — the exact condition. And execute_query is still unsynchronised on main:

# utils/tree_sitter_manager.py:322
def execute_query(language, query_string, node):
    query = Query(language, query_string)      # per call, no lock
    cursor = QueryCursor(query)
    res = cursor.captures(node)

tree_sitter_manager locks its language cache (_cache_lock) but nothing guards query execution.

So merging this would move a known crash from cgc index into cgc watch, where it is worse: watch runs unattended for hours, and a segfault takes the whole process down mid-session.

What would unblock it

Either is fine, and either would be a genuinely valuable PR on its own:

• Fix Bug: Segfault during 'cgc index' — concurrent tree-sitter execute_query() calls across worker threads (race condition in native extension) #1370 first — a module-level lock around query execution in execute_query, or a per-language lock if the contention matters. Then this merges as-is.
• Or bound the parallelism to be safe — e.g. group paths_to_parse by suffix and parse each language group serially while running groups concurrently, so no two threads are ever in the same language's grammar. That sidesteps Bug: Segfault during 'cgc index' — concurrent tree-sitter execute_query() calls across worker threads (race condition in native extension) #1370 without waiting on it, though it is a guess at the race's shape.

The rest of the PR is good — reusing the already-parsed changed file is a clear win regardless, and if you wanted to split that out as its own PR it would merge immediately. Keeping this open as a draft.

#1370 is now fixed — #1593 merged, adding a process-wide lock around tree-sitter query construction and execution.

That was the blocker on this PR. Measured cost of the lock is ~1% (3.48s vs 3.45s median over 120 parses on 8 threads), because only the query step is serialised — parser.parse and all Python-level work stay parallel, so your ThreadPoolExecutor still buys what it was meant to buy.

One caveat I want to be straight about: I could not reproduce the segfault locally, so #1593 is a guard against the reported crash rather than one I demonstrated. It is not proof your parallelism is now safe — it removes the specific unsynchronised path the report pointed at.

Your branch is already rebased onto main and green. Since it is still marked draft, I have left it for you to mark ready. Worth re-running your own workload against it first if you have the mixed-language repo that motivated the change.

This is still marked as a draft, so I've left it alone — flagging that it's now conflict-free against main if you'd like to mark it ready for review. Parallelising the incremental parse step is worth having; happy to review once you take it out of draft.