fix(parsers): serialise tree-sitter query execution across threads - #1593

Fixes #1370. Full suite: 1157 passed, 7 skipped.

The bug

tree_sitter_manager locked its language cache (_cache_lock) but nothing guarded query execution:

def execute_query(language, query_string, node):
    query = Query(language, query_string)   # per call, no lock
    cursor = QueryCursor(query)
    res = cursor.captures(node)

cgc index runs parsers on a thread pool, so several threads reach this concurrently on different grammars. #1370's traceback shows exactly that — three threads inside execute_query at once, on TypeScript, Python and HTML.

It surfaces as Fatal Python error: Segmentation fault, not a Python exception, so nothing above it can catch or retry. The whole process dies mid-index.

The lock is process-wide rather than per-language on purpose: the reports involve different languages crashing together, so a per-language lock would not have prevented them.

Cost

Negligible, because only the query step is serialised — parser.parse (tree construction) and all Python-level work stay parallel:

120 parses, 8 threads, mixed Python/TypeScript
  with lock:    3.48s  median of 3
  without:      3.45s  median of 3

Within noise. Query execution is a small fraction of per-file work.

What I could not verify

I could not reproduce the segfault on this machine. 400 concurrent mixed-language parses across 16 threads survived five unlocked runs cleanly. The race is timing- and version-dependent — the reporter hit it on a mixed-language repo with a specific tree-sitter build.

So this is a guard against a reported native crash rather than one I reproduced, and the honest case for merging it is that the cost is ~1% and a segfault is unrecoverable and undebuggable for users. If you would rather wait for a local reproduction, that is a defensible call — but the trade looks strongly one-sided to me.

The tests therefore pin the guard, not the crash: a segfault cannot be asserted on from inside the process that takes it, and a passing concurrent run proves nothing about a nondeterministic race. test_queries_never_overlap_under_concurrency instruments the query body and asserts the peak concurrent count is exactly 1 across 64 calls on 8 threads. 3 of the 4 tests fail without the fix.

Knock-on

This unblocks #1052 (parallel incremental parse in the watcher), which I rebased but held precisely because it would have moved this crash from cgc index into cgc watch — where it is worse, since watch runs unattended for hours.

🤖 Generated with Claude Code