Skip to content

Cache WordNet max depth lazily for lch_similarity() - #3592

Merged
stevenbird merged 4 commits into
nltk:developfrom
ekaf:hotfix-2273
Jun 6, 2026
Merged

Cache WordNet max depth lazily for lch_similarity()#3592
stevenbird merged 4 commits into
nltk:developfrom
ekaf:hotfix-2273

Conversation

@ekaf

@ekaf ekaf commented Jun 4, 2026

Copy link
Copy Markdown
Member

Closes #2273

Summary

This PR speeds up lch_similarity() by caching maximum taxonomy depth lazily per WordNetCorpusReader instance, keyed by part of speech and whether that taxonomy requires a simulated root.

It addresses the same performance issue as #3584, but takes a different approach: instead of hardcoding precomputed depth constants for WordNet 3.0, it computes the values from the actually loaded corpus on first use and reuses them thereafter.

Motivation

lch_similarity() depends on the maximum depth of the taxonomy for the relevant part of speech. Computing that value by iterating over all synsets is expensive, but it is also stable for a given loaded WordNetCorpusReader instance.

Because a reader instance is bound to a single underlying WordNet dataset, the maximum taxonomy depth does not need to be recomputed for repeated calls. Caching it lazily gives the same practical speedup as precomputing constants, while keeping the value derived from the real corpus data.

Design

This PR changes the max-depth cache from a flat POS-based lookup to an instance-local lazy cache keyed by:

  • pos
  • need_root

This is enough to uniquely determine the value within a single WordNetCorpusReader instance.

The cache does not need to be keyed by WordNet version, because version is already implicit in the reader instance. Different simultaneously loaded readers naturally maintain separate caches.

Why this approach

Compared to #3584, this design has a few advantages:

  1. No hardcoded constants

    • Avoids embedding magic numbers such as noun=19 and verb=12.
    • Removes the maintenance burden of keeping those constants in sync with the corpus.
  2. Derived from actual loaded data

    • The cached values are computed from the real synsets exposed by the current reader.
    • This keeps the cache aligned with the dataset actually in use.
  3. Works for any WordNet version

    • No special-casing for WordNet 3.0.
    • Supports other WordNet versions automatically, including multiple versions loaded at the same time in separate reader instances.
  4. Preserves existing semantics

    • The cache is keyed by the same root-handling logic already used by lch_similarity().
    • This keeps the optimization behavior-preserving rather than introducing a semantic change.
  5. Preserves lazy behavior

    • The full scan still happens only when needed.
    • After the first computation, repeated similarity calls reuse the cached value.

Notes

This PR is intended as an alternative implementation of the optimization proposed in #3584. Both PRs address the same hotspot, but this version prefers lazy caching from source data over hardcoded per-version constants.

Testing

This change is intended to preserve the existing public behavior of lch_similarity() while improving performance.

Existing unit tests for lch_similarity() continue to cover representative cases in nltk/test/unit/test_wordnet.py, including:

  • S("dog.n.01").lch_similarity(S("cat.n.01"))
  • S("big.a.01").lch_similarity(S("long.a.01"))
  • S("long.a.01").lch_similarity(S("big.a.01"))

No additional tests of the internal max-depth cache are necessary, since the existing lch_similarity() tests already provide sufficient proof that this change preserves behavior.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes WordNet’s Synset.lch_similarity() by lazily caching the maximum taxonomy depth per WordNetCorpusReader instance, keyed by (pos, need_root) (i.e., whether that POS requires a simulated root for the metric).

Changes:

  • Cache lch_similarity()’s max-depth lookup by (pos, need_root) instead of POS only.
  • Make _compute_max_depth() return the computed depth and memoize results for subsequent calls.
  • Add/adjust internal _max_depth cache initialization in WordNetCorpusReader.__init__.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread nltk/corpus/reader/wordnet.py
Comment thread nltk/corpus/reader/wordnet.py Outdated
@stevenbird
stevenbird merged commit 2deaa35 into nltk:develop Jun 6, 2026
22 checks passed
@ekaf
ekaf deleted the hotfix-2273 branch June 9, 2026 06:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Max depth of the all wordnet POS should be returned and kept as static

3 participants