Skip to content

Fix: close SQLite connections from terminated threads (#2192) - #2193

Merged
nedbat merged 4 commits into
coveragepy:mainfrom
MattLloyd101:main
Jun 20, 2026
Merged

nedbat merged 4 commits into
coveragepy:mainfrom
MattLloyd101:main

Conversation

@MattLloyd101

@MattLloyd101 MattLloyd101 commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Summary

CoverageData keeps one SqliteDb per thread in self._dbs, keyed by threading.get_ident(). A connection is opened lazily in _open_db, but it was never closed when the thread that created it terminated. Entries were only reused if the same thread id recurred, and the whole dict was only closed at process end.

On workloads with many short-lived threads whose ids are not recycled, _dbs grows without bound and each dead thread leaks one open file descriptor to the data file, eventually hitting OSError: [Errno 24] Too many open files. This was observed on a multi-hour pytest run: ~6 live OS threads but 30k+ open fds to the .coverage file before failing.

Fixes #2192.

Changes

  • Add CoverageData._reap_dead_thread_dbs(), which closes and drops any SqliteDb whose owning thread is no longer in threading.enumerate(). It runs under the existing self._lock, and closing is safe from another thread because connections are created with check_same_thread=False. Closing is best-effort so a close failure can't break collection.
  • Call it from the cold path of _connect (only when the current thread has no connection yet), so the hot path on every add_lines/add_arcs is unchanged.
  • Add a regression test in tests/test_data.py.
  • Add a CHANGES.rst entry.

Test plan

  • New test test_dead_thread_dbs_are_reaped uses a threading.Barrier to keep ~50 worker threads concurrent (so they get distinct thread ids, since a naive sequential start/join can let the OS recycle one id and hide the leak), joins them, asserts all 50 connections are retained, then records coverage from the main thread (the cold path) and asserts _dbs falls back to the live-thread count of 1.
  • python -m tox -- tests/test_data.py passes (99 tests locally).

Notes

Thanks for maintaining coverage.py. Happy to adjust the approach — for example, reaping on a different trigger or under a separate config — if you'd prefer a different design.

@nedbat

nedbat commented Jun 16, 2026

Copy link
Copy Markdown
Member

Is there a way we could use weakrefs instead? To avoid having to explicitly enumerate the threads?

@MattLloyd101

Copy link
Copy Markdown
Contributor Author

We could use weakref.finalize on the current thread when we open the DB and call a tidy up function which closes the database handle. However that does mean the cleanup timing is then entirely driven by the GC. The main concern would be that if we spawn many threads in a small amount of time, before the GC triggers, you still could get file descriptor exhaustion. Though in practice I suspect that's highly unlikely, but in theory still possible.

I'm also unsure how I'd create a deterministic test for this, without explicitly forcing a GC as the timing is handled by the GC. Which is kind of the issue with that approach.

I'm happy to refactor down this path if you'd prefer it that way, I suspect it would still work for us. However I feel it trades reliability for elegance.

@nedbat

nedbat commented Jun 20, 2026

Copy link
Copy Markdown
Member

Thanks, we'll go with this.

@nedbat
nedbat merged commit f960696 into coveragepy:main Jun 20, 2026
44 checks passed
nedbat added a commit that referenced this pull request Jun 20, 2026
@nedbat

nedbat commented Jun 20, 2026

Copy link
Copy Markdown
Member

This is now released as part of coverage 7.14.2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CoverageData leaks one SQLite file descriptor per terminated thread

2 participants