ENH: allow StringDType to use the sort paths in isin - #32693

PR summary

Fixes #32161. Supersedes #32217.

This follows up on recent improvements to StringDType (in particular #32563 and #32464) to enable substantial performance improvements for isin. Also adds more tests for isin in the StringDType tests.

Tests and benchmarks ran on an M1 MacBook Air using Python 3.14. The pandas.NA case uses pandas 3.0.5.

For pandas.NA, the patch also fixes an error and correctly reports missing entries as non-matches. All returned results were checked against the expected membership mask.

Each baseline was measured once; patched timings are the best of three runs. Input creation is excluded from the timings. Reproduce with repeats=1 on the baseline and repeats=3 on the patched build:

import timeit
import numpy as np
import pandas as pd

repeats = 3
sentinels = (None, np.nan, "NA", object(), pd.NA)
dtypes = [np.dtypes.StringDType()]
dtypes += [np.dtypes.StringDType(na_object=na) for na in sentinels]
for dtype in dtypes:
    values = ["abcd" * 5, "defg" * 5, "abhoj" * 6]
    if hasattr(dtype, "na_object"):
        values[-1] = dtype.na_object
    a = np.array(values * 10_000, dtype=dtype)
    print(dtype, min(timeit.repeat(lambda: np.isin(a, a),
                                  number=1, repeat=repeats)))

AI Disclosure

I used an AI to iterate on this.