[Iceberg] Support Excluding Columns from Stats Collection on Data Write

[osscm]

Generating statistics for columns of type STRING, BINARY, or for columns not involved in joins can be time-consuming and lead to increased storage usage.

For large tables, it's important to have the ability to skip stats generation for certain columns to improve performance and reduce overhead.

While the ANALYZE command allows specifying which columns to collect statistics for, there is currently no way to control this during data write or update operations.

ref: #17057
ref: https://docs.delta.io/latest/optimizations-oss.html#data-skipping

[ebyhr]

@osscm Could you share how you are going to implement this feature? Intorucing new table properties or reusing Iceberg write.metadata.metrics.column. property?

[raunaqmorarka]

This has been considered before #9791
Copying my rationale for not pursuing this

We don't want the presence of column level metrics to be user configurable.
We want all metrics to be collected by default because Trino has been adding usage of column statistics in new optimizer rules besides join reordering, and it's not easy for users to discover which column metrics would be useful.
We already truncate large string statistics automatically in orc and parquet writers in Trino and the iceberg table metadata is just derived from these file format statistics. 
See `io.trino.parquet.writer.PrimitiveColumnWriter#MAX_STATISTICS_LENGTH_IN_BYTES` for example.
We are able to cache manifest file contents in memory https://github.com/trinodb/trino/pull/22739 or on local filesystem now https://github.com/trinodb/trino/pull/20803 to reduce latency.
We also request only projected column statistics from iceberg metadata after https://github.com/trinodb/trino/pull/22584 and https://github.com/trinodb/trino/pull/22555.
I don't see a reason to pursue this change for above reasons.

I would like to see numbers showing that this makes a significant difference in cpu/latency/storage.
It's also possible that there is a bottleneck which could be optimized without adding configurability.

[ebyhr]

FYI, apache/iceberg#12771 proposes write.parquet.stats-enabled.column. column property.

[raunaqmorarka]

FYI, apache/iceberg#12771 proposes write.parquet.stats-enabled.column. column property.

That seems like an even more misguided idea. There is no reason to skip writing column stats in parquet as long as you truncate long strings (which we already do).

[huaxiangsun]

Not sure about trinodb implementation. As mentioned in apache/parquet-java#3188, for parquet, it needs to hold whole min/max in memory before writing truncated string/bytes out. We found that this min/max stats for column chunks (for large size columns) causes OOMs.

[raunaqmorarka]

it needs to hold whole min/max in memory before writing truncated string/bytes out. We found that this min/max stats for column chunks (for large size columns) causes OOMs.

If you don't have sufficient memory to hold the min/max values in memory, then I'm not sure you have sufficient memory to complete the write even without the min/max.
Any writer implementation has to hold the entire row group in memory before flushing to disk. Even with compression this is 64-128MB of data. So the uncompressed min/max values would have to exceed this to be the driving factor for memory consumption.
You also have to buffer one uncompressed parquet page per-column in memory until the page is large enough to be flushed and compressed.
The Trino engine itself has to carry the uncompressed values in memory as Trino Page/Block through various operators. So we won't eliminate the requirement to hold large string values in memory even if we're not collecting min/max for parquet column statistics.