Skip to content

Comments

[SPARK-55624][PS][TESTS] Ignore ArrowDtype in tests with pandas 3#54412

Open
ueshin wants to merge 1 commit intoapache:masterfrom
ueshin:issues/SPARK-55624/ignore_arrow_dtype
Open

[SPARK-55624][PS][TESTS] Ignore ArrowDtype in tests with pandas 3#54412
ueshin wants to merge 1 commit intoapache:masterfrom
ueshin:issues/SPARK-55624/ignore_arrow_dtype

Conversation

@ueshin
Copy link
Member

@ueshin ueshin commented Feb 21, 2026

What changes were proposed in this pull request?

Ignores ArrowDtype in tests with pandas 3.

Why are the changes needed?

The ArrowDtype is used for the result of the predicates operations on some dtypes in pandas 3 by default, but the values should be same as BooleanDtype.

It should be ignored in tests.

>>> pser = pd.Series(["x", "y", "z", None], dtype="string")
>>> other_pser = pd.Series([None, "z", "y", "x"], dtype="string")
>>>
>>> pser == other_pser
0     <NA>
1    False
2    False
3     <NA>
dtype: bool[pyarrow]
>>> (pser == other_pser).astype("boolean")
0     <NA>
1    False
2    False
3     <NA>
dtype: boolean

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Updated the test util.

Was this patch authored or co-authored using generative AI tooling?

No.

@ueshin
Copy link
Member Author

ueshin commented Feb 21, 2026

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant