simple export part #1009

arthurpassos · 2025-09-10T12:41:33Z

Changelog category (leave one):

Experimental Feature

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Very small subset of #939

Very simple idempotent retries (only if the part is the very same)
Part names are preserved and we rely on the fact that they remain stable in the remote storage
Should work for all *MergeTree implementations
No retries or failure recovery
No transactions

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

github-actions · 2025-09-10T12:44:46Z

Workflow [PR], commit [9140d43]

arthurpassos · 2025-09-10T12:54:50Z

src/Storages/MergeTree/MergeTreeData.cpp

            break;
+            case PartitionCommand::EXPORT_PART:
+            {
+                exportPartToTable(command, query_context);


I am now confused if I need to grab the DROP lock or not. MergeTreeData::lockForShare. I'll have a look soon

arthurpassos · 2025-09-10T12:56:32Z

src/Storages/MergeTree/MergeTreeData.cpp

+        return;
+    }
+
+    /// todo implement these settings


Need to decide what to do here

arthurpassos · 2025-09-10T13:37:03Z

src/Storages/ObjectStorage/ObjectStorageFilePathGenerator.h

@@ -0,0 +1,59 @@
+#pragma once


This is a hack to be able to override the filenames that would be generated from the storage engine itself.

This does not look good at all, but well, it is what it is

filimonov · 2025-09-11T13:00:49Z

src/Interpreters/PartLog.cpp

            "MutatePart — Mutating of a data part has finished, "
-            "MovePart — Moving the data part from the one disk to another one."},
+            "MovePart — Moving the data part from the one disk to another one."
+            "ExportPart — Exporting the data part from a merge tree table to one (e.g, object storage)."},


Suggested change

"ExportPart — Exporting the data part from a merge tree table to one (e.g, object storage)."},

"ExportPart — Exporting the data part from a MergeTree table into a target table that represents external storage (e.g., object storage or a data lake)."},

src/Storages/IPartitionStrategy.cpp

filimonov · 2025-09-11T13:05:41Z

src/Storages/IStorage.h

        ContextPtr /*context*/,
        bool /*async_insert*/);

+    virtual bool supportsImport() const


slightly confusing that to have export you need to do the import in the code :)

would be nice to havbe some comments in the header

I am still not sure what's the best name for this set of APIs. The current names (supportsImport and import) sound too generic, I could add mergeTree, but it's not like it can only be used for MergeTree parts.

For now, it is quite similar to the write api, it just won't do any partitioning and allows the filename to be overwritten.

In the future, we'll need to check for covering & intersecting parts. At that point, we'll have two options:

Polute StorageObjectStorage with MergeTree logic; Not ideal, but that's ok tbh

Create a third class that is friend with both *MergeTree and StorageObjectStorage that orchestrate the parts check + simplistic sink creation. By simplistic I mean no partitioning, data lake checks and etc

maybe smth like canAcceptExport / acceptExportedPart / receiveExport etc.? (but i'm not too convinced)

filimonov · 2025-09-11T13:14:35Z

src/Storages/MergeTree/MergeTreeData.cpp

+    }
+
+    if (!dest_storage->supportsImport())
+        throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Destination storage {} does not support importing merge tree partitions", dest_storage->getName());


Suggested change

throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Destination storage {} does not support importing merge tree partitions", dest_storage->getName());

throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Destination storage {} does not support importing MergeTree parts or uses unsupported partitioning", dest_storage->getName());

filimonov · 2025-09-11T13:18:53Z

src/Storages/MergeTree/MergeTreeData.cpp

+        throw Exception(ErrorCodes::INCOMPATIBLE_COLUMNS, "Tables have different structure");
+
+    if (query_to_string(src_snapshot->getPartitionKeyAST()) != query_to_string(destination_snapshot->getPartitionKeyAST()))
+        throw Exception(ErrorCodes::BAD_ARGUMENTS, "Tables have different partition key");


actually the scenario of moving from more granular partitioning to less granular should not brake any invariants (like moving part from PARTITION BY (date, customer) to PARTITION BY (date) )

But it's ok for now.

Yep, agreed

filimonov · 2025-09-11T13:43:46Z

src/Storages/MergeTree/MergeTreeData.cpp

+                }, moves_assignee_trigger, getStorageID()));
+            return true;
+        }
+    }


So active moves currently can prevent exports. But i have no strong opinion about what's best here.

I don't know either. As a safety measure, I gave preference to the existing functionality over this one. And was hoping to get feedback from users and you on this

I do need to fix this tho: https://github.com/Altinity/ClickHouse/pull/1009/files#diff-b7d33a7abda5f3365d5e4fb58ca98b756f6401372b2e2d5eb449a2bd3444fb5eL8554

filimonov · 2025-09-11T13:45:51Z

src/Storages/MergeTree/MergeTreeData.cpp

+        status.destination_database = manifest.destination_storage_id.database_name;
+        status.destination_table = manifest.destination_storage_id.table_name;
+        status.create_time = manifest.create_time;
+        status.part_name = manifest.data_part->name;


some form of progress would be nice to have here ? like bytes read / bytes total / progress like in system.merges

filimonov · 2025-09-11T13:54:19Z

src/Storages/System/StorageSystemExports.cpp

+        {"destination_table", std::make_shared<DataTypeString>(), "Name of the destination table."},
+        {"create_time", std::make_shared<DataTypeDateTime>(), "Date and time when the export command was submitted for execution."},
+        {"part_name", std::make_shared<DataTypeString>(), "Name of the part"}
+    };


some form of progress ?

filimonov · 2025-09-11T13:55:48Z

tests/queries/0_stateless/03572_export_merge_tree_part_to_object_storage_simple.sql

+
+DROP TABLE IF EXISTS 03572_mt_table, 03572_invalid_schema_table;
+
+CREATE TABLE 03572_mt_table (id UInt64, year UInt16) ENGINE = MergeTree() PARTITION BY year ORDER BY tuple();


Same test by for ReplicatedMergeTree ?

filimonov · 2025-09-11T14:08:34Z

src/Storages/ObjectStorage/StorageObjectStorage.cpp


+bool StorageObjectStorage::supportsImport() const
+{
+    return configuration->partition_strategy != nullptr && configuration->partition_strategy_type == PartitionStrategyFactory::StrategyType::HIVE;


need a better message for the user (see above)

arthurpassos · 2025-09-13T13:28:06Z

CICD looks ok. Only failure is test_database_hms and it does not seem to be related. As discussed on Friday, let's merge it as is and improve it in the next iterations.

Todo list from the PR comments:

Improve observability by adding progress
Relax partition and schema constraints
Add docs to internal APIs

simple export part

arthurpassos added 2 commits September 10, 2025 09:34

simple export part

f03b55c

Merge branch 'antalya-25.6.5' into simple_export_part

242bb50

arthurpassos commented Sep 10, 2025

View reviewed changes

arthurpassos added 2 commits September 10, 2025 10:17

small adjustments

2bb0018

fix build issues after merge

8756bb1

arthurpassos commented Sep 10, 2025

View reviewed changes

some modifications

6524a60

svb-alt added antalya antalya-25.6 antalya-25.6.5 labels Sep 10, 2025

arthurpassos added 2 commits September 10, 2025 14:22

some small fixes and shitty test

14f86c4

add some more tests

8f8caa3

filimonov reviewed Sep 11, 2025

View reviewed changes

arthurpassos added 12 commits September 11, 2025 13:42

fix settingchangehist

9706887

Merge branch 'antalya-25.6.5' into simple_export_part

e71b720

address minor comments

4479116

one more fix

b0c7e9e

add tests for replicated engine

ebdf147

add parallel formatting again, respect minbytestousedirectio

b4993cc

attempt to fix tests

993c115

one more change to tests

569df0b

fix flakiness

3aa23c2

Merge branch 'antalya-25.6.5' into simple_export_part

6447e54

filepath fix

c355ae6

aa

9140d43

arthurpassos merged commit 09ab657 into antalya-25.6.5 Sep 13, 2025
377 of 422 checks passed

svb-alt added the antalya-25.8 label Sep 29, 2025

arthurpassos added a commit that referenced this pull request Sep 29, 2025

Merge pull request #1009 from Altinity/simple_export_part

bc7f70e

simple export part

arthurpassos mentioned this pull request Sep 29, 2025

Antalya 25.8 - Forward port of # 1009 - Export merge tree part #1041

Merged

13 tasks

alex-zaitsev mentioned this pull request Oct 6, 2025

Project Antalya Roadmap 2025 - Real-Time Data Lakes #804

Open

37 tasks

	"ExportPart — Exporting the data part from a merge tree table to one (e.g, object storage)."},
	"ExportPart — Exporting the data part from a MergeTree table into a target table that represents external storage (e.g., object storage or a data lake)."},

	throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Destination storage {} does not support importing merge tree partitions", dest_storage->getName());
	throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Destination storage {} does not support importing MergeTree parts or uses unsupported partitioning", dest_storage->getName());


		DROP TABLE IF EXISTS 03572_mt_table, 03572_invalid_schema_table;

		CREATE TABLE 03572_mt_table (id UInt64, year UInt16) ENGINE = MergeTree() PARTITION BY year ORDER BY tuple();

simple export part #1009

simple export part #1009

Uh oh!

Conversation

arthurpassos commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Documentation entry for user-facing changes

Uh oh!

github-actions bot commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arthurpassos commented Sep 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

arthurpassos commented Sep 10, 2025 •

edited

Loading

github-actions bot commented Sep 10, 2025 •

edited

Loading