Support Zstd for archives

[michalmiddleton]

Is your feature request related to a problem? Please describe.
Not really a problem but I stop containers before backup and the time it takes to finish it is a few minutes.

Describe the solution you'd like
Support zstd as compression algorithm.

Describe alternatives you've considered
n/a

Additional context
Ok, here's the meat. I forked your repo and tested replacing gzipWriter with Zstd Encoder and it was a drop-in replacement. I hit only one small issue.

I know my stats are very rough but I hope the advantage is apparent:

Gzip:

time="2023-01-21T13:45:08-06:00" level=info msg="Stopping 3 container(s) labeled `docker-volume-backup.stop-during-backup=true` out of 10 running container(s)."
time="2023-01-21T13:48:13-06:00" level=info msg="Created backup of `/backup` at `/tmp/backup-2023-01-21T13-45-08.tar.gz`."
time="2023-01-21T13:48:15-06:00" level=info msg="Restarted 3 container(s) and the matching service(s)."
time="2023-01-21T13:48:18-06:00" level=info msg="Removed tar file `/tmp/backup-2023-01-21T13-45-08.tar.gz`."
time="2023-01-21T13:48:18-06:00" level=info msg="Finished running backup tasks."

Zstd:

time="2023-01-21T13:41:59-06:00" level=info msg="Stopping 3 container(s) labeled `docker-volume-backup.stop-during-backup=true` out of 10 running container(s)."
time="2023-01-21T13:42:43-06:00" level=info msg="Created backup of `/backup` at `/tmp/backup-2023-01-21T13-41-59.tar.zst`."
time="2023-01-21T13:42:45-06:00" level=info msg="Restarted 3 container(s) and the matching service(s)."
time="2023-01-21T13:42:47-06:00" level=info msg="Removed tar file `/tmp/backup-2023-01-21T13-41-59.tar.zst`."
time="2023-01-21T13:42:47-06:00" level=info msg="Finished running backup tasks."

These were two backups on the same system. Notice the gzip version took 3min 5sec (including container shut down) while zstd took only 42sec!

Uncompressed Tar size was about 6392644096, gzipped: 3990896812, zstd: 3931777630. However the time savings are the most important aspect to me.

[m90]

If you want to add this as an option (gzip should still be the default), I am happy to review PRs and merge them once ready.

[erwanlpfr]

If you want to add this as an option (gzip should still be the default), I am happy to review PRs and merge them once ready.

Hello,

I'm about to try to implement this.
I wish I have time to do it so this week.

[erwanlpfr]

Hello,

I'm here back to ask you some questions related to the structure of DVB.
I'm facing a problem when I upload a large file (+30GB and can go to 200GB)

archive: (*s3Storage).Copy: error uploading backup to remote storage: [Message]: 'Part number must be an integer between 1 and 1000, inclusive', [Code]: InvalidArgument, [StatusCode]: 400"

So, I'm going to change some values of the S3 client to make it possible.
Also, this is why I go there also to try to implement the Zstd support.

Do you think, is that state, the following can be implemented in one shot ?

• Increment values to allow large file on S3,
• Implement an interface that allows more than Zstd

Thanks

[m90]

@erwanlpfr I'd appreciate if these two issues are dealt with in separate PRs. They shouldn't be dependent on each other, so it also shouldn't create an my extra overhead to open two PRs. Thanks.

[ManuelBu96]

@michalmiddleton What is the current status of zstd support?

[michalmiddleton]

@ManuelBu96 I haven't had a chance to get back to this.I don't want to make any promises but I am still somewhat interested in adding zstd.

[ManuelBu96]

Okay, thats a pity. I would like to support, but unfortunately I am not familiar with the programming language "Go".

[m90]

@ManuelBu96 Not trying to sell anything here, but this repo has already seen major contributions by authors who never used Go before. If you wanna give it a try I'm happy to help out and review PRs and all that.

[michalmiddleton]

I opened PR#249 for zstd

[m90]

This is now released in v2.29.0. Thanks @michalmiddleton for your contribution 🎉