[Data] Fix reading from written parquet for numpy with NaNs

[owenowenisme]

Description

Currently when transforming pandas to arrow block, we will convert it into pa.null if all the value in TensorArray is NaN or empty string, however all NaN might actually represent something and could actually happen in transformation.

ray/python/ray/air/util/tensor_extensions/pandas.py

Lines 926 to 958 in 8b4a1ee

	def isna(self) -> "TensorArray":
	"""
	A 1-D array indicating if each value is missing.
	Returns
	-------
	na_values : Union[np.ndarray, ExtensionArray]
	In most cases, this should return a NumPy ndarray. For
	exceptional cases like ``SparseArray``, where returning
	an ndarray would be expensive, an ExtensionArray may be
	returned.
	Notes
	-----
	If returning an ExtensionArray, then
	* ``na_values._is_boolean`` should be True
	* `na_values` should implement :func:`ExtensionArray._reduce`
	* ``na_values.any`` and ``na_values.all`` should be implemented
	"""
	if self._tensor.dtype.type is np.object_:
	# Avoid comparing with __eq__ because the elements of the tensor
	# may do something funny with that operation.
	return np.array(
	[self._tensor[i] is None for i in range(len(self))], dtype=bool
	)
	elif self._tensor.dtype.type is np.str_:
	return np.all(self._tensor == "", axis=tuple(range(1, self._tensor.ndim)))
	else:
	return np.all(
	np.isnan(self._tensor), axis=tuple(range(1, self._tensor.ndim))
	)

And could triggered error when we tried to convert that arrow block back to pandas (try running the repro script from issue)

This PR skip coerced to pa.null if thecol.dtype is TensorDtype, which should tolerate the all NaN or all empty string table.

Related issues

Closes #59087

[gemini-code-assist]

Code Review

This pull request fixes an issue with reading Parquet files containing NumPy arrays with NaNs. The main changes are:

1. In TensorDtype.__from_arrow__, it now correctly handles Arrow arrays with null values by disabling zero-copy when converting to a NumPy array. This prevents failures when nulls are present.
2. In TensorArray.isna, the definition of a missing value is changed. Now, for non-object types, NaN is treated as a valid value, not a missing value. This aligns with the goal of correctly handling NaNs.

My review focuses on improving the implementation of these changes. I've suggested a simplification in __from_arrow__ and a bug fix in isna for object-dtype arrays that is exposed by the changes in this PR. Specifically, the isna method for object arrays should handle np.nan values, which can be introduced during the Arrow-to-NumPy conversion.

[owenowenisme]

/gemini review

[gemini-code-assist]

Code Review

This pull request aims to fix an issue with isna for numpy arrays with NaNs after a Parquet roundtrip. The change addresses a bug where np.isnan could be called on non-float dtypes, causing a TypeError. However, the current fix of returning np.zeros for all non-object dtypes is too broad and incorrectly disables NaN detection for float tensors. I've provided a critical review comment with a suggested implementation that correctly handles different dtypes (float, string, and others) to ensure isna behaves as expected without reintroducing the original bug.

[alexeykudinkin]

Why removing the comments? Let's keep them

[owenowenisme]

Added

[alexeykudinkin]

Let's assert that the type is tensor