Add shredded Variant reader cases with variant logical type

[aihuaxu]

This is generated from apache/iceberg#13708 which includes the following changes on top of #90:

• Add variant logical type
• Update the decimal test data to align with the spec. E.g., the value with precision in [1,9] uses decimal, so 123456.7890 should uses decimal8 instead of decimal4. Update to use 12345.6789 for decimal4 test.
• Mark some tests as invalid since they don't align with the spec and the readers can choose different implementation.

I have tested from Parquet-Java against this data (https://github.com/apache/parquet-java/pull/3258/files).
Here shows an example of the schema with variant logical type annotated.

Schema:
message table {
  required int32 id = 1;
  optional group var (VARIANT(1)) = 2 {
    required binary metadata;
    optional binary value;
    optional group typed_value (LIST) {
      repeated group list {
        required group element {
          optional binary value;
          optional binary typed_value (STRING);
        }
      }
    }
  }
}

[aihuaxu]

cc @rdblue, @zeroshade. This is the same data set but I have included parquet logical type in the schema.

[xxubai]

Do we have any python script to generate shredding variants

[aihuaxu]

Do we have any python script to generate shredding variants

We don't have right now. We generated from Iceberg test cases.

[xxubai]

We don't have right now. We generated from Iceberg test cases.

OK. Anyway, this is very helpful for shredding variant testing in other projects.

[zeroshade]

@aihuaxu You updated the descriptions in shredded_variant/cases.json but the files that actually have the variant logical type in them are in data/shredded_variant/ can we consolidate to either one directory or the other?

[alamb]

My understanding is that this PR would replace #90 , is that correct?

[alamb]

Thank you @aihuaxu

I can't test with these values as we are still working on the shredding implementation in arrow-rs (see apache/arrow-rs#8150 and related tickets) but they look reasonable to me

I defer to @zeroshade to check with the Go implementation

cc @carpecodeum and @mprammer and @scovich

[alamb]

This error message looks clear to me

[aihuaxu]

My understanding is that this PR would replace #90 , is that correct?

That is correct. I build on top of #90 to add variant logical type and made small changes (update decimal test cases and change test case with INVALID).

[aihuaxu]

@zeroshade Can you check with GO implementation? I have tested with Java implementation and it works fine.

[zeroshade]

Hey @aihuaxu I've filed #94 with the files generated by Go, also fixing the issues you identified.

[aihuaxu]

Hey @aihuaxu I've filed #94 with the files generated by Go, also fixing the issues you identified.

@zeroshade I have verified from parquet-java and it works. Thanks.

[aihuaxu]

@zeroshade and @zeroshade can we merge this PR? Seems we have done enough cross-language check.

[carpecodeum]

arrow-rs is also going ahead with these files as the core test cases - apache/arrow-rs#8104

[alamb]

@zeroshade and @zeroshade can we merge this PR? Seems we have done enough cross-language check.

I think it is a good idea. However, I suspect that @zeroshade may not have the permissions to do so. I think it will require someone from this group (@emkornfield @julienledem @rdblue etc)

https://people.apache.org/phonebook.html?unix=parquet

[wgtmac]

Thanks all for validation! Let me merge this.

[alamb]

Thank you @wgtmac and @aihuaxu and @zeroshade