Conversation
FsspecStore.from_url() left the filesystem it created in fsspec's global instance cache, so the instance -- and any cleanup finalizers the backend registered on it -- survived until interpreter shutdown. Backends whose finalizers call fsspec's sync machinery, e.g. adlfs binding weakref.finalize(self, sync, self.loop, ...), then raised 'RuntimeError: Loop is not running' from weakref._exitfunc as noise on every process exit (zarr-developersgh-4221). from_url() now creates the filesystem with skip_instance_cache=True, so its lifetime follows the store's and backend cleanup finalizers run while the process is still running, mirroring the cachable=False behavior of fsspec's own AsyncFileSystemWrapper. Callers can restore instance sharing with skip_instance_cache in storage_options. A subprocess regression test pins the reported symptom: a local async-capable backend with an adlfs-style finalizer no longer produces the RuntimeError at interpreter shutdown after the store is dropped. Fixes zarr-developers#4221
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4369 +/- ##
==========================================
- Coverage 94.22% 94.18% -0.04%
==========================================
Files 92 92
Lines 12942 12942
==========================================
- Hits 12195 12190 -5
- Misses 747 752 +5
🚀 New features to boost your workflow:
|
d-v-b
reviewed
Sep 17, 2026
| # with pending cleanup finalizers that touch dead event loops | ||
| # (https://github.com/zarr-developers/zarr-python/issues/4221). An | ||
| # explicit ``skip_instance_cache`` in ``storage_options`` wins. | ||
| opts = {"asynchronous": True, "skip_instance_cache": True, **opts} |
Contributor
There was a problem hiding this comment.
this amounts to changing the default skip_instance_cache behavior for url_to_fs. I think it's better to leave that default to fsspec. Folks who want to skip the instance cache can say so explicitly, using the kwarg, until fsspec changes its default.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FsspecStore.from_url()retained the filesystem it created in fsspec's global instance cache, so backends that register cleanup finalizers on their filesystems kept those instances alive until interpreter shutdown. At exit, finalizers that call fsspec'ssyncmachinery (e.g. adlfs) find the event loop gone and emitRuntimeError: Loop is not runningfromweakref._exitfuncon every process exit.from_url()now creates its filesystem withskip_instance_cache=True, mirroring thecachable=Falsebehavior of fsspec's ownAsyncFileSystemWrapper, so the filesystem's lifetime follows the store's and backend cleanup runs while the process is still executing. Callers who rely on cross-call instance sharing can passskip_instance_cacheinstorage_optionsto restore it. A subprocess regression test pins the reported symptom against a local async-capable backend, and a towncrier fragment documents the behavior change.Fixes #4221