Catalog.to_dask_dataframe

[hombit]

Feature request

Implement Catalog.to_dask_dataframe() which would return self._ddf.copy().

Before submitting
Please check the following:

• I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
• I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
• If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.

[delucchi-cmu]

Is this still relevant / the same with changes to nested-*?

[hombit]

The method name may differ, but we still need to provide a "safe" interface for casting catalogs to (nested) Dask dataframes. This would give users access to the (nested) dataframe functionality not wrapped by the Catalog class.

[dougbrn]

I see some issue with this due to nested-dask being deprecated. The code for nested-dask has been moved into LSDB, but only bits and pieces are used and a standalone lazy dataframe is not explicitly supported within our leaner code structure. If we want to export to dask, we may want to consider exporting nested structures to a more neutral data type.

[dougbrn]

I've moved forward with this, despite my above comment. While I think the limitations of working with nested columns is valid with dask dataframes, I think it's still a value add to convert to dask for non nested workflows, and even with nested columns the data types are still preserved and the underlying in-memory dataframe is still a nestedframe, potentially opening up some map_partitions workflows if the user needs to work on the dask level with nested-pandas functions.