Skip to content

bug: DuckDB backend returns pyarrow.RecordBatchReader instead of pyarrow.Table, breaking .execute() #11632

@ElenaKhaustova

Description

@ElenaKhaustova

What happened?

When using the DuckDB backend in Ibis with recent versions of DuckDB (≥1.4.0), calling .execute() on a table expression can fail because the backend returns a pyarrow.RecordBatchReader.

Ibis currently assumes the result is a pyarrow.Table and accesses attributes like .column_names, which RecordBatchReader does not have. This leads to:

AttributeError: 'pyarrow.lib.RecordBatchReader' object has no attribute 'column_names'

How to reproduce

  1. Install latest (will reproduce the issue with DuckDB ≥1.4.0)
pip install "ibis-framework[duckdb]" duckdb pyarrow
  1. Run the following example
import ibis

# Connect via Ibis DuckDB backend
con = ibis.duckdb.connect(database=":memory:")

# Create an in-memory table
data = ibis.memtable({"col1": [1, 2], "col2": [3, 4]})
con.create_table("t", data)

# Load the table back
t = con.table("t")

# Attempt to execute
result = t.execute()  # Fails with AttributeError when DuckDB >=1.4.0

What version of ibis are you using?

ibis-framework==10.8.0

What backend(s) are you using, if any?

duckdb==1.4.0 (regression not seen with <1.4.0)

Relevant log output

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/opt/miniconda3/envs/test-ibis-env/lib/python3.11/site-packages/ibis/expr/types/core.py", line 424, in execute
    return self._find_backend(use_default=True).execute(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/opt/miniconda3/envs/test-ibis-env/lib/python3.11/site-packages/ibis/backends/duckdb/__init__.py", line 1415, in execute
    for name, col in zip(table.column_names, table.columns)
                         ^^^^^^^^^^^^^^^^^^
AttributeError: 'pyarrow.lib.RecordBatchReader' object has no attribute 'column_names'

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugIncorrect behavior inside of ibis

    Type

    No type

    Projects

    Status

    done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions