Maybe buggy nullable integer floor division by 0

[TomAugspurger]

From @jschendel in #30183 (comment)

In [6]: a = pd.array([0, 1, -1, None], dtype="Int64")

In [7]: a // 0
Out[7]: 
<IntegerArray>
[0, 0, 0, NaN]
Length: 4, dtype: Int64

In [8]: a // -0
Out[8]: 
<IntegerArray>
[0, 0, 0, NaN]
Length: 4, dtype: Int64

In [9]: a // 0.0
Out[9]: array([nan, nan, nan, nan])

In [10]: a // -0.0
Out[10]: array([nan, nan, nan, nan])

Those should probably all be NA, to match the ndarray behavior.

[jbrockmendel]

to match the ndarray behavior

This is a case where the Series behavior is different from the ndarray behavior. Unless there's a compelling reason to match ndarray, I'd prefer to match Series (and DataFrame and Int64Index)

[AlexKirko]

@jbrockmendel Wouldn't matching it to Series be less intuitive for the users? They'd have a numpy array of integers and a pandas array of integers, and those would behave differently. In my opinion, that is less expected than IntegerArray and Series behaving differently (these are very different classes).
On the other hand, matching to Series promotes consistent processing of array-like objects in pandas, but I don't think that's as important as intuitive behavior.

[jorisvandenbossche]

Those should probably all be NA, to match the ndarray behavior.

The numpy behaviour is actually different for integers:

In [16]: np.array([0, 0, 1]) // 0  
/home/joris/miniconda3/envs/dev/bin/ipython:1: RuntimeWarning: divide by zero encountered in floor_divide
  #!/home/joris/miniconda3/envs/dev/bin/python
Out[16]: array([0, 0, 0])

In [17]: np.array([0., 0., 1.]) // 0  
/home/joris/miniconda3/envs/dev/bin/ipython:1: RuntimeWarning: invalid value encountered in floor_divide
  #!/home/joris/miniconda3/envs/dev/bin/python
Out[17]: array([nan, nan, nan])

Not fully sure what the rationale of that difference is, though

[AlexKirko]

@jorisvandenbossche, thanks. The behavior of IntegerArray turns out to be the same as np.array:

np.array([0,1,-1]) // 0
C:\Users\amata_n2pl6zt\.conda\envs\pandas-dev\lib\site-packages\ipykernel_launcher.py:1: RuntimeWarning: divide by zero encountered in floor_divide
array([0, 0, 0], dtype=int32)

pd.array([0,1,-1],dtype='Int32') // 0
<IntegerArray>
[0, 0, 0]
Length: 3, dtype: Int32

To me, this looks like a numpy bug.

[jorisvandenbossche]

The reasoning might be: floor division of an integer array returns a new integer array. So for that reason, the result cannot hold NaN, and 0 was taken instead.

[TomAugspurger]

Hmm so do we have a decision on the expected value? It's not clear to me what's best here.

[jbrockmendel]

The numpy behaviour is actually different for integers:

It's also different yet again if the zero you're dividing by is float instead of integer.

A lot of effort went in to making the Series implementation of truediv/floordiv by zero not have these surprises, and the tests (tests.arithmetic.test_numeric) are fairly thorough.

Could we do something like wrap the ops.array_ops function and then re-mask? Maybe even add IntegerArray to the parametrization in test_numeric

[jbrockmendel]

also seems like the options.treat_inf_as_na would be relevant

[jorisvandenbossche]

It's also different yet again if the zero you're dividing by is float instead of integer.

Yes, but then you are again doing a float floor division, not an integer one.
I suppose that numpy is going for type stability here, and decided on those return types:

int // int -> int
int // float -> float
float // float -> float

And if the result is int, then you are limited in what the value can be.

A lot of effort went in to making the Series implementation of truediv/floordiv by zero not have these surprises

What kind of surprises do you mean exactly? For true division I think Series and numpy array behave the same? (except for the warning that we hide)

also seems like the options.treat_inf_as_na would be relevant

I never use this, but I think this is about interpretation of inf as NA in functions that detect NA (isna, fillna, etc), not about creation of inf in operations (which is what we are discussing here I think)
Which surprises do you mean exactly? I don't think we treat true division differently as in numpy?

[jbrockmendel]

I never use this, but I think this is about interpretation of inf as NA in functions that detect NA (isna, fillna, etc), not about creation of inf in operations

I think you're right about how we use that option. What im thinking of is:

arr = pd.array([-1, 0, 1])
ser = pd.Series([-1, 0, 1])

>>> ser // 0
0   -inf
1    NaN
2    inf
dtype: float64

Suppose we agree that IntegerArray should behave like Series. Then the IntegerArray op gets back np.array([-inf, NaN, inf]) and has to decide how to case the result. If we decide (possibly via an option) to treat inf as NA, then we could return an all-NA IntegerArray result instead of a float array.

[jorisvandenbossche]

Suppose we agree that IntegerArray should behave like Series.

So what is the rationale of the current Series behaviour for floor division by 0? When returning a float result, why not follow numpy's behaviour for floats of returning all NaN ?

[jbrockmendel]

So what is the rationale of the current Series behaviour for floor division by 0?

My recollection of the reasoning is that the semantic meaning of division/floordiv isn't impacted by the dtypes of numerator or denominator, so the results shouldn't be either.