Migrate MatchCategories to narwhals, add polars support - #1064

Summary

Migrates MatchCategories to narwhals, so it takes pandas and polars dataframes.

• pandas: the output is the same as before: category dtype with the categories learned in fit, unseen categories become NaN. It uses its own native path.
• polars: variables are cast to pl.Enum(categories) with the categories learned in fit. Unseen categories become null. Enum is the polars dtype that fixes the category set, like pandas category. pl.Categorical doesn't: it takes any string, so unseen categories would go through. Everything else works the same on both backends: the NaN checks, the warning or error when NaNs are introduced, return_empty, variables and ignore_format.
• fit and transform reuse the mixins in encoding/base_encoder.py (_check_or_select_variables, _check_na, _check_transform_input_and_state, _get_feature_names_in). transform still overrides the mixin's transform.
• The pandas transform is now ~2.5x faster. It builds the categorical from integer codes (Categorical.from_codes(levels.get_indexer(x))) instead of where(isin) followed by a new Categorical. It gets Categorical from nw.get_native_namespace, so there is no import pandas. It also avoids pandas' Pandas4Warning that the old code raised when the input was already categorical with different categories.
• The pandas input is copied before assignment, so the user's dataframe is not modified.
• Tests rewritten to the conventions (make_df, frame_to_dict, full-message match=, init-parameter section first). pandas-only tests cover the exact category dtype, integer column names and the index. One polars-only test covers the exact Enum schema. The data is used only in this file, so I added no conftest.py, which avoids clashing with the MatchVariables PR.
• Docs: docstring updated. The user guide gets a "With polars" section and its pandas outputs are refreshed to what the code returns now (pandas 3 reprs: Index(..., dtype='int64') instead of Int64Index, StringArray, dtype='str'). I also removed a sentence that pointed to a non-existent errors parameter and documented the warning/error that missing_values controls. Every example was run.

Benchmarks

Each candidate was timed as the whole body of fit / transform. Median of 9 runs, candidates alternated. 50 categories per variable, 1% unseen in the test set. Times in ms.

pandas transform (implemented: from_codes + get_indexer)

* narwhals' Enum gives an ordered pandas categorical with string categories, so it doesn't reproduce the current output anyway.

pandas fit (the categories of each variable): the old pd.Categorical(x).categories and x.astype("category").cat.categories are tied (e.g. 500k × 20: 258.6 vs 261.4 ms). They give identical results. I kept astype, which needs no pandas namespace.

Whole transformer, pandas, main vs this branch (median of 7, two rounds):

The whole fit is slower than on main, e.g. 500k × 20: ~1030 ms vs ~520–610 ms. The time goes to the shared find_categorical_variables → _is_categorical_and_is_not_datetime in variable_handling: 0.67 of 0.96 s in a profile. MatchCategories' own code isn't the cause, and this PR doesn't touch that helper. See "Pre-existing issues".

polars (implemented: narwhals, see "Needs decision")

Also measured and discarded for polars fit: narwhals unpivot().unique(), about 2x slower than per column. Sorting in Python after unique() tied with the narwhals per-column version. narwhals' lazy select(unique()) isn't supported. For polars transform, narwhals replace_strict(..., return_dtype=Enum) was about as fast as when/is_in, with noisy results.

Behaviour

• pandas: identical to main on 25 recorded cases. The comparison covers fit's category_dict_ (values, dtype, Index type), the output frame (assert_frame_equal), dtypes and categories, warnings, errors, and checks that the input is not modified. The cases were: unseen categories with ignore/raise, NaN in fit/transform with ignore/raise, ignore_format on float/int/bool/datetime, categorical input with unused categories in a non-sorted order, pandas string dtype, mixed-type object, integer column names, return_empty, a custom index, and the docstring example. Differences:
  • Fixed: two or more integer-named columns with introduced NaNs now warn/raise with "... feature(s) 0, 1.". Before, this raised TypeError: sequence item 0: expected str instance, int found.
  • Inherited from the base mixin already on narwhals-migration, not from this PR: transform no longer reorders the columns to the training order. An empty (0-row) test set gets sklearn's message "Found array with 0 feature(s)..." instead of "0 feature(s)...".
• polars vs pandas: same values, same categories, same warnings and errors for string variables, for categorical input (Enum keeps its dtype categories, including unused ones, like pandas) and for every missing-value case. Differences are listed under "Needs decision".

Tests

tests/test_preprocessing, one pytest call, base (origin/narwhals-migration) vs this branch:

• before: 27 failed, 8 passed
• after: 22 failed, 64 passed
• new failures: none. Fixed: the 3 old test_match_categories.py tests, test_check_estimator_from_feature_engine[MatchCategories] and test_transformers_in_pipeline_with_set_output_pandas[MatchCategories].
• Still failing for MatchCategories: test_check_estimator_from_sklearn[estimator0]. sklearn's check_estimator passes numpy arrays, which check_X rejects. Encoders fail the same way on the base branch. The other failures are MatchVariables, which is migrated in a separate PR.

flake8 feature_engine tests is clean. mypy feature_engine gives the same 2 errors as the base (datetime_subtraction.py, log.py).

Needs decision

1. Numerical (and bool/datetime) variables with ignore_format=True on polars. pl.Enum only accepts strings. Casting numbers straight to Enum treats them as physical indices, and polars deprecates it. I sort the categories in the original dtype, so they stay in numeric order, and then cast them to strings. So [3, 1, 10] gives Enum(["1", "3", "10"]), and the values in the output are strings. pandas keeps numeric categories. Alternatives:
   • (a) implemented: cast to string, then Enum.
   • (b) keep the numeric dtype and only null out unseen values. The values would match pandas, but the output is not categorical.
   • (c) raise an error for non-string variables on polars.
2. Type of category_dict_ values on polars. They are lists of strings (the Enum categories), while pandas keeps a pd.Index. A pl.Series would also work, but a list prints cleanly and is what Enum takes.
3. polars-native fast path. The polars-native versions are ~4–5x faster in fit and ~2x faster in transform at 500k+ rows with several variables (table above). I didn't add them: a polars-native else branch would break other narwhals backends. A third branch (pandas / polars / narwhals) would leave the narwhals fallback untested, because pyarrow isn't installed locally or in CI. Absolute times are small (500k × 20: fit 54 vs 11 ms, transform 26 vs 13 ms), and the whole fit is dominated by variable detection anyway. I can add the polars branch if you prefer speed here.
4. pl.Categorical input has no per-column category set, so its categories are the sorted observed values. pl.Enum input keeps its dtype categories, as pandas does for category.
5. NaN in polars float columns (with ignore_format=True) is treated as missing, like in pandas: it is not learned as a category, and it becomes null in the output.

Pre-existing issues, not fixed

• find_categorical_variables (via _is_categorical_and_is_not_datetime) dominates fit time on both backends. On 500k × 20 it takes ~0.67 s of a 0.96 s pandas fit and ~0.37 s of a 0.44 s polars fit. The cost comes from converting each column to a Python list to test for numbers and datetimes. This is shared by all encoders, so it's out of scope here.
• The sklearn check_estimator test fails because it passes numpy arrays (same for the encoders).