ALTER TABLE EXPORT to external table

[arthurpassos]

For MergeTree tables we need an efficient way to move parts to Parquet. One part should map to one Parquet file, and MergeTree blocks should map to row groups. Sorting should be preserved.

[arthurpassos]

copy will suffice for now

[arthurpassos]

remember about It doesn't filter out rows that are deleted with lightweight deletes.

[arthurpassos]

Shoudl the user be able to specify the output file name? What about format settings?

[arthurpassos]

I missed the "move to external table" part. Format and storage location should be inferred from table.

Something like:

alter table xyz export partition 'abc' to table

[arthurpassos]

Well, since the goal is to move to a different table, this is slightly more complex. Some constraints light schema and partition expression must match, similar to attach partition functionality.

[arthurpassos]

I ignored the constraints for now and I am able to move a clickhouse partition from a merge tree table to s3 table engine, but there is a caveat: the partition must contain only one part. The reason is: multiple parts require multiple files on s3, therefore the s3 engine should be created with something that uniquely identifies files, such as uuid or _part macros. If a s3 table engine is created with any of those, it goes into readonly mode.

[arthurpassos]

I need to pay attention to this: MergeTree blocks should map to row groups. The current implementation will just call StorageObjectStorage::write API for each data part, but that doesn't guarantee the mapping from blocks to row groups.

[arthurpassos]

Investigate why inserts are not supported and how can we fix it

[arthurpassos]

Paused for now because we first need to fix S3 writes.

[arthurpassos]

The reason is: multiple parts require multiple files on s3,
This is not true actually

A couple of design considerations:

1. Do we want a single file per partition or single file per part? If we are doing single file per part, do we want to be able to backtrace it later?
2. We could soft the constraints a bit when compared to the alter attach. Since the data is going to be completely rewritten, there is no need to assert partition expression is the same and etc. But of course repartitioning would take place.
3. is it a must to support exporting to table function in the first iteration of the feature?