Cache WordNet max depth lazily for lch_similarity() - #3592
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR optimizes WordNet’s Synset.lch_similarity() by lazily caching the maximum taxonomy depth per WordNetCorpusReader instance, keyed by (pos, need_root) (i.e., whether that POS requires a simulated root for the metric).
Changes:
- Cache
lch_similarity()’s max-depth lookup by(pos, need_root)instead of POS only. - Make
_compute_max_depth()return the computed depth and memoize results for subsequent calls. - Add/adjust internal
_max_depthcache initialization inWordNetCorpusReader.__init__.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2273
Summary
This PR speeds up
lch_similarity()by caching maximum taxonomy depth lazily perWordNetCorpusReaderinstance, keyed by part of speech and whether that taxonomy requires a simulated root.It addresses the same performance issue as #3584, but takes a different approach: instead of hardcoding precomputed depth constants for WordNet 3.0, it computes the values from the actually loaded corpus on first use and reuses them thereafter.
Motivation
lch_similarity()depends on the maximum depth of the taxonomy for the relevant part of speech. Computing that value by iterating over all synsets is expensive, but it is also stable for a given loadedWordNetCorpusReaderinstance.Because a reader instance is bound to a single underlying WordNet dataset, the maximum taxonomy depth does not need to be recomputed for repeated calls. Caching it lazily gives the same practical speedup as precomputing constants, while keeping the value derived from the real corpus data.
Design
This PR changes the max-depth cache from a flat POS-based lookup to an instance-local lazy cache keyed by:
posneed_rootThis is enough to uniquely determine the value within a single
WordNetCorpusReaderinstance.The cache does not need to be keyed by WordNet version, because version is already implicit in the reader instance. Different simultaneously loaded readers naturally maintain separate caches.
Why this approach
Compared to #3584, this design has a few advantages:
No hardcoded constants
19and verb=12.Derived from actual loaded data
Works for any WordNet version
Preserves existing semantics
lch_similarity().Preserves lazy behavior
Notes
This PR is intended as an alternative implementation of the optimization proposed in #3584. Both PRs address the same hotspot, but this version prefers lazy caching from source data over hardcoded per-version constants.
Testing
This change is intended to preserve the existing public behavior of
lch_similarity()while improving performance.Existing unit tests for
lch_similarity()continue to cover representative cases innltk/test/unit/test_wordnet.py, including:S("dog.n.01").lch_similarity(S("cat.n.01"))S("big.a.01").lch_similarity(S("long.a.01"))S("long.a.01").lch_similarity(S("big.a.01"))No additional tests of the internal max-depth cache are necessary, since the existing
lch_similarity()tests already provide sufficient proof that this change preserves behavior.