MAINT: Prepare patsy for pandas3 StringDtype

[bashtage]

Adds support for StringDtype
Fixes tests that are not valid with copy-on-write

[Copilot]

Pull Request Overview

This PR prepares the patsy library for pandas 3 compatibility by adding support for StringDtype and fixing tests that are incompatible with pandas 3's copy-on-write behavior. The changes ensure the library continues to work with newer pandas versions while maintaining backward compatibility.

• Adds detection logic for pandas 3 and StringDtype support
• Updates test assertions to account for copy-on-write behavior in pandas 3
• Extends dtype checking functions to handle StringDtype alongside existing categorical dtype support

[matthewwardrop]

Is this due to upstream indexing changes? Kind of annoying if ... no longer supports "zero" expansion.

[bashtage]

Yes, it appears that it is stricter and no longer supports expansion. I only found this by running against the statsmodels test suite.

[matthewwardrop]

Hmmm... this still works in my testing. Is there an upstream numpy change I should be worried about? (These are, or should be, numpy arrays still).

[matthewwardrop]

Hmmm... are there other places that string dtypes should be treated as categorical, no? I'll need to take a look, since I haven't looked at patsy code for a while.

[matthewwardrop]

Doesn't look like it, but I didn't do an exhaustive search.

[matthewwardrop]

Left a few questions/feedback. Once we've addressed those, let's get this in and released.

[matthewwardrop]

Hmmm... this still works in my testing. Is there an upstream numpy change I should be worried about? (These are, or should be, numpy arrays still).

[matthewwardrop]

Doesn't look like it, but I didn't do an exhaustive search.

[bashtage]

Hmmm... this still works in my testing. Is there an upstream numpy change I should be worried about? (These are, or should be, numpy arrays still).

This is probably a subtle bug in patsy. If you:

• call patsy.dmatrix using a DesignInfo
• with a missing value handler attached
• Using a DataFrame input

then patsy passes pandas series to the NA handler. pandas 3 changed the semantics about Series indexing, and so this is why this fix is included here.

This is the statsmodels call that hits this

                output = patsy.dmatrix(
                    formula, data, eval_env=eval_env, return_type=return_type, **kwargs
                )

Formula looks like

DesignInfo(['Intercept', 'var1', 'var2'],
           factor_infos={EvalFactor('var1'): FactorInfo(factor=EvalFactor('var1'),
                                    type='numerical',
                                    state=<factor state>,
                                    num_columns=1),
                         EvalFactor('var2'): FactorInfo(factor=EvalFactor('var2'),
                                    type='numerical',
                                    state=<factor state>,
                                    num_columns=1)},
           term_codings=OrderedDict([(Term([]),
                                      [SubtermInfo(factors=(),
                                                   contrast_matrices={},
                                                   num_columns=1)]),
                                     (Term([EvalFactor('var1')]),
                                      [SubtermInfo(factors=(EvalFactor('var1'),),
                                                   contrast_matrices={},
                                                   num_columns=1)]),
                                     (Term([EvalFactor('var2')]),
                                      [SubtermInfo(factors=(EvalFactor('var2'),),
                                                   contrast_matrices={},
                                                   num_columns=1)])]))

data is

            y    var1    var2    var3
obs01 -0.8197  0.1231  0.5486 -0.0282
obs03  2.6568  0.4688 -0.3997  1.9084
obs05 -1.4936  0.1264 -0.2380 -0.8709
obs07  0.3174  0.7396 -0.1641 -0.0267
obs09 -2.4240 -1.7268 -1.4262  0.2660

and return_type="dataframe".

[bashtage]

I wrote a test that will fail using existing pandas. It requries using a non-trivial index, e.g., strings.

[matthewwardrop]

LGTM. Thanks @bashtage !