How To Get Your Backup to Half of Its Size - Introducing ZSTD Support in Percona...
source link: https://www.percona.com/blog/get-your-backup-to-half-of-its-size-introducing-zstd-support-in-percona-xtrabackup/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Introducing ZSTD Support in Percona XtraBackup
Having a backup of your database is like insurance, you have to pay a monthly price to ensure you have a service available when you need to. When talking about backups, the storage required to keep your backups is what comes into factor when talking about price, the bigger your backup, or the bigger the retention period, the more it will cost.
Compressing your backups is a common practice to reduce this cost. Currently, Percona XtraBackup (PXB) has support for two compression algorithms: quicklz (which is an abandoned project and will soon be deprecated in PXB) and LZ4.
Today we are glad to introduce support for a new compression algorithm in Percona XtraBackup 8.0.30 – Zstandard (ZSTD).
Zstandard is a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level and better compression ratios.
Usage
A new value has been added to the xtrabackup –compress option. Passing –compress=zstd to PXB will make it use the new Zstandard algorithm to run compression.
Compression can work in parallel by adjusting –parallel=X for parallel file copy and –compress-threads=x for parallel compression of those files.
For decompression either on PXB or xbstream the usage remains the same. Users only need to pass –decompress and the tool will use the same algorithm used for compression.
Please note that as in qpress and LZ4, you will require the ZSTD client to run the –decompress operation.
Compress
xtrabackup --backup --compress=zstd --parallel=16 --compress-threads=8 --target-dir=/backup |
Decompress
xtrabackup --decompress --parallel=16 --target-dir=/backup |
Testing
To test how ZSTD compares with LZ4 and uncompressed backups, we set up a test environment using an AWS EC2 instance c5.4xlarge (16 CPUs and 32G of RAM ) with data and backup going to the same disk partition, an EBS IO2 with 10K provisioned IOPS SSD.
For each round of tests, we created a set of 12 tables with 40M rows resulting in a 109G dataset.
We ran five rounds of tests creating a new database with the same amount of data on each round.
Percona XtraBackup was invoked using –parallel=16 –compress-threads=8 .
Size of resulting backup (full, LZ4, and ZSTD), time to run the backup (full, LZ4, and ZSTD), and time to decompress the resulting backup (LZ4 and ZSTD).
The second round of tests explored how the two algorithms and uncompressed backups perform when uploading the data to S3. For this round, we ran two tests.
The first one was to take the backup again and upload it to S3, time to complete was measured.
The second part of the test was to download the backup from S3 and in the case of uncompressed backups, just store it on disk, and for compressed backup, we were first decompressing it and then storing it on disk.
On all tests, ZSTD compression level of one was used.
Results
Summary
As we can see from the test results above, ZSTD not only overcame LZ4 results on all tests, but it also brought backup size to half of its original size.
When streaming is added to the mix is when we see the biggest difference between both algorithms, with ZSTD overcoming LZ4 with an even bigger margin.
This can bring users and organizations a huge amount of savings in backup storage, either on-premises or especially in the cloud – where we are charged for each GB of storage we use.
Recommend
-
27
v1 is the first GA version of our new MongoDB backup tool. This has been custom-built to assist those users who don’t want to pay for MongoDB E...
-
8
Percona XtraBackup: Introducing Support For Storage Class in xbcloudBack to the Homepage
-
10
-
15
This is a very s...
-
11
Percona Backup for MongoDB 1.6.0¶ Date:August 16, 2021 Installation:
-
11
Introducing xbcloud: Azure Blob Storage Support in Percona XtraBackup Back to the Blog delivers the xbcloud binary – an auxiliary tool to allow users to upload backups t...
-
9
Physical Backup Support in Percona Backup for MongoDB ...
-
4
Why is my iCloud backup size so large?
-
36
Microsoft Graph PowerShell v2 is now in public preview, half the size, and will speed up your automations
-
1
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK