Fix: close SQLite connections from terminated threads (#2192) - #2193
Conversation
|
Is there a way we could use weakrefs instead? To avoid having to explicitly enumerate the threads? |
|
We could use I'm also unsure how I'd create a deterministic test for this, without explicitly forcing a GC as the timing is handled by the GC. Which is kind of the issue with that approach. I'm happy to refactor down this path if you'd prefer it that way, I suspect it would still work for us. However I feel it trades reliability for elegance. |
|
Thanks, we'll go with this. |
|
This is now released as part of coverage 7.14.2. |
Summary
CoverageDatakeeps oneSqliteDbper thread inself._dbs, keyed bythreading.get_ident(). A connection is opened lazily in_open_db, but it was never closed when the thread that created it terminated. Entries were only reused if the same thread id recurred, and the whole dict was only closed at process end.On workloads with many short-lived threads whose ids are not recycled,
_dbsgrows without bound and each dead thread leaks one open file descriptor to the data file, eventually hittingOSError: [Errno 24] Too many open files. This was observed on a multi-hourpytestrun: ~6 live OS threads but 30k+ open fds to the.coveragefile before failing.Fixes #2192.
Changes
CoverageData._reap_dead_thread_dbs(), which closes and drops anySqliteDbwhose owning thread is no longer inthreading.enumerate(). It runs under the existingself._lock, and closing is safe from another thread because connections are created withcheck_same_thread=False. Closing is best-effort so a close failure can't break collection._connect(only when the current thread has no connection yet), so the hot path on everyadd_lines/add_arcsis unchanged.tests/test_data.py.CHANGES.rstentry.Test plan
test_dead_thread_dbs_are_reapeduses athreading.Barrierto keep ~50 worker threads concurrent (so they get distinct thread ids, since a naive sequential start/join can let the OS recycle one id and hide the leak), joins them, asserts all 50 connections are retained, then records coverage from the main thread (the cold path) and asserts_dbsfalls back to the live-thread count of 1.python -m tox -- tests/test_data.pypasses (99 tests locally).Notes
Thanks for maintaining coverage.py. Happy to adjust the approach — for example, reaping on a different trigger or under a separate config — if you'd prefer a different design.