[R] "Invalid metadata$r" warning

[r2evans]

Describe the bug, including details regarding any error messages, version, and platform.

I started adding this solely as a comment on #40423, but since the warning just started for me when I updated to arrow_22.0.0 (and R-4.5) and that other issue is from 2024, I thought it might be a different cause. It may also be related to #32729. A common theme is the presence of attributes in the columns (as shown in the second link).

I can trigger the issue with a named-vector as one of the columns:

mutate(mtcars, cyl = setNames(nm = cyl)) |>
  arrow_table() |>
  rename_with(.fn = toupper)
# Warning: Invalid metadata$r
# Warning: Invalid metadata$r
# Table (query)
# MPG: double
# CYL: double
# DISP: double
# HP: double
# DRAT: double
# WT: double
# QSEC: double
# VS: double
# AM: double
# GEAR: double
# CARB: double
# See $.data for the source Arrow object

This can be hacked in open_dataset() by removing the "names" component of the attributes, but this does not work with a table created with arrow_table().

This breaks at

arrow/r/R/metadata.R

Line 211 in 29586f4

attributes(x)[names(r_metadata$attributes)] <- r_metadata$attributes

attributes(x)[names(r_metadata$attributes)] <- r_metadata$attributes
# Error in attributes(x)[names(r_metadata$attributes)] <- r_metadata$attributes : 
#   'names' attribute [32] must be the same length as the vector [0]

### for context
x
# numeric(0)
attributes(x)
# NULL
r_metadata$attributes
# $names
#  [1] "6" "6" "4" "6" "8" "6" "8" "4" "4" "6" "6" "8" "8" "8" "8" "8" "8" "4" "4" "4" "4" "8" "8" "8" "8" "4" "4" "4" "8" "6" "8" "4"

The error is because base R itself requires the "names" attribute to be sized the same as the data, which at this point in the call x is length 0 (numeric(0)).

The underlying issue is that x here is still lazy with a placeholder numeric(0). Is it possible to change whether the data is realized already for this specific data path?

Component(s)

R

[thisisnic]

I asked Claude to investigate this. Here's what it found:

Root cause: When operations like rename_with need to determine the output schema, they create a zero-row placeholder table via Table__from_schema(). This placeholder table retains the original R metadata, including any names attribute. When converting to a data.frame, apply_arrow_r_metadata() at r/R/metadata.R:211 tries to apply the stored names attribute (32 elements) to a length-0 placeholder vector, which fails because R requires names to match vector length.

Call chain:

rename_with.ArrowTabular
  → column_select
    → as.data.frame(implicit_schema(.data))
      → Table__from_schema(x)  # creates 0-row table
        → apply_arrow_r_metadata()
          → attributes(x)[...] <- r_metadata  # fails at line 211

Why two warnings: rename_with internally calls both select and rename, each triggering the same path.

[thisisnic]

Claude again. The fix is straightforward - in r/R/schema.R:469-471:

# Current:
as.data.frame.Schema <- function(x, row.names = NULL, optional = FALSE, ...) {
  as.data.frame(Table__from_schema(x))
}

# Fixed:
as.data.frame.Schema <- function(x, row.names = NULL, optional = FALSE, ...) {
  Table__from_schema(x)$to_data_frame()
}

[thisisnic]

Issue resolved by pull request 49608
#49608