@Shashankss1205 I have a deterministic reproducer and what I believe is the root cause. Measured on cgc 0.5.7 — I diffed my installed core/watcher.py against v0.5.7 on GitHub and it is byte-identical, so none of this is local drift.

Root cause: one OS thread per filesystem event, held for the full debounce

RepositoryEventHandler._debounce (core/watcher.py:193) creates a threading.Timer — a real OS thread — for every filesystem event, and that thread stays alive for the whole debounce_interval (2.0 s) before it decides whether the file is even relevant.

The amplifier is on_deleted (core/watcher.py:357):

def on_deleted(self, event):
    if not event.is_directory:
        self._debounce(event.src_path, lambda: self._handle_modification(event.src_path))

on_created and on_modified both gate on _is_supported_code_file before _debounce, so an ignored path or an unparsed extension costs nothing. on_deleted has no gate at all — the .cgcignore check happens later, inside _handle_modification_locked (core/watcher.py:253), which is after the thread exists and after it has queued on _update_lock.

So the thread count follows the raw delete rate, independent of .cgcignore.

Reproducer

Against an already-indexed repo with cgc watch running, in a directory that is already ignored (target/), using an extension that is not parsed (.tmp) — i.e. doubly excluded:

mkdir -p target/probe
for i in $(seq 1 800); do : > target/probe/p_$i.tmp; done
sleep 1
rm -f target/probe/*.tmp &
for s in 1 2 3 4 5 6 8 10; do
  echo "threads=$(ps -M -p <WATCH_PID> | tail -n +2 | wc -l)"; sleep 0.35
done

Result:

baseline threads=4
t=2  threads=804      <- 800 delete events -> 804 live OS threads
t=6  threads=804
t=8  threads=660      <- draining as the 2 s timers fire
after: threads=4

All 804 threads did nothing: the path was ignored and the extension unparsed. They existed only to sleep 2 s.

Real-world triggers are ordinary: mvn clean, flutter clean, git checkout between branches, rm -rf node_modules. On a polyglot monorepo these delete tens of thousands of files in a burst.

Why the original report sees unbounded RAM and I do not

This is the part I think explains @Slamik's 128 GB.

On macOS there is a hard per-process thread ceiling:

$ sysctl kern.num_taskthreads
kern.num_taskthreads: 4096

So the watcher saturates at ~4096 threads and stops growing. Three separate watcher processes, same box, same repo:

pid	age	phys_footprint	peak	live threads	stack regions
A	2 h 57 m	290 MB	354 MB	3	4101
B	2 m	191 MB	238 MB	4	—
C	6 m	259 MB	311 MB	3	4097

A 6-minute-old process and a 3-hour-old process sit within 12% of each other — age is not the variable. ~4096 stacks x 32 KB is ~64 MB dirty plus ~33 MB of kernel page tables, on top of a ~150–190 MB interpreter/tree-sitter/driver baseline. That is the 250–350 MB plateau, and sampling every 2 min shows a sawtooth (stacks got reclaimed at one point, footprint fell 270 -> 192 MB, then climbed again) rather than a monotonic leak.

Instead of bloating past the ceiling, the process dies: a burst wider than 4096 concurrent timers makes pthread_create fail and CPython raises RuntimeError: can't start new thread. Two watcher processes died within minutes of my probes, and macOS wrote no crash report because it is a clean Python exception exit.

Hypothesis I could not test: Linux/WSL2 has no comparable low per-process thread cap (threads-max is typically in the tens of thousands and cgroup pids.max is often unlimited), and glibc's default thread stack is 8 MB. If the thread count there tracks the event rate with no 4096 ceiling to stop it, the same mechanism runs until RAM is gone rather than dying at a plateau — which matches "grows progressively until it exhausts all system memory (128 GB)" under Claude Code, an agent that generates continuous file churn. Someone on Linux could confirm by watching ls /proc/<pid>/task | wc -l during a git checkout of a large branch.

Relationship to #1275 and #716

#1275 fixed the correctness half of this — ignored paths no longer produce stale/duplicate graph nodes, because _handle_modification_locked checks _should_ignore. The cost half is still open: the thread is created, started, and scheduled before that check runs, so ignored paths are free in the graph but not free in memory.

Suggested fixes

1. Gate on_deleted like its siblings (small, removes the common trigger):

def on_deleted(self, event):
    if event.is_directory:
        return
    p = Path(event.src_path)
    if p.suffix not in self.graph_builder.parsers or self._should_ignore(p):
        return
    self._debounce(event.src_path, lambda: self._handle_modification(event.src_path))

Note it cannot reuse _is_supported_code_file verbatim: that calls path_obj.is_file(), which is always False for a path that was just deleted. I suspect that is why the filter was omitted.

2. Stop using one thread per event (the actual fix). A single long-lived worker thread draining a {path: deadline} map costs one thread total regardless of event volume, and gives correct coalescing for free. As it stands, an N-file burst inside the debounce window costs N threads even when they all collapse to one unit of work — and past 4096 on macOS it is fatal rather than merely wasteful.

Environment

codegraphcontext 0.5.7 (pipx), Python 3.14.6
macOS 15 (Darwin 25.5.0), Apple Silicon
Neo4j 2026.05.0, bolt://localhost:7687
Watched repo: polyglot Java / Dart / TypeScript monorepo, ~77k files, ~353k nodes indexed
cgc watch started on an already-indexed repo (perform_initial_scan=False, no --sync-on-start)

One measurement note for anyone reproducing on macOS: ps RSS is misleading here — it reported 29 MB while /usr/bin/footprint -p <pid> reported 290 MB for the same process, because the thread stacks are compressed. Use footprint, and count regions with vmmap <pid> | grep -c '^Stack '. Also pipe ps -M into wc -l rather than reading it directly, or terminal truncation will show you 3 threads when there are 804.

Uh oh!

CGC Watch continuously consumes memory until it eats all available RAM. #744

Description

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions