Comparing S3 Streaming Tools with Percona XtraBackup

Making backups over the network can be done in two ways: either save on disk and transfer or just transfer without saving. Both ways have their strong and weak points. The second way, particularly, is highly dependent on the upload speed, which would either reduce or increase the backup time. Other factors that influence it are chunk size and the number of upload threads.

Percona XtraBackup 2.4.14 has gained S3 streaming, which is the capability to upload backups directly to s3-compatible storage without saving locally first. This feature was developed because we wanted to improve the upload speeds of backups in Percona Operator for XtraDB Cluster .

There are many implementations of S3 Compatible Storage: AWS S3 , Google Cloud Storage , Digital Ocean Spaces , Alibaba Cloud OSS , MinIO , and Wasabi .

We’ve measured the speed of AWS CLI , gsutil , MinIO client , rclone , gof3r and thexbcloud tool (part of Percona XtraBackup) on AWS (in single and multi-region setups) and on Google Cloud. XtraBackup was compared in two variants: a default configuration and one with tuned chunk size and amount of uploading threads.

Here are the results.

AWS (Same Region)

The backup data was streamed from the AWS EC2 instance to the AWS S3, both in the us-east-1 region.

tool settings CPU max mem speed speed comparison AWS CLI not changeable 66% 149Mb 130MiB/s baseline MinIO client not changeable 10% 679Mb 59MiB/s -55% rclone rcat not changeable 102% 7138Mb 139MiB/s +7% gof3r default settings 69% 252Mb 97MiB/s -25% gof3r 10Mb block, 16 threads 77% 520Mb 108MiB/s -17% xbcloud default settings 10% 96Mb 25MiB/s -81% xbcloud 10Mb block, 16 threads 60% 185Mb 134MiB/s +3%

Tip: If you run MySQL on an EC2 instance to make backups inside one region, do snapshots instead.

AWS (From US to EU)

The backup data was streamed from AWS EC2 in us-east-1 to AWS S3 in eu-central-1.

tool settings CPU max mem speed speed comparison AWS CLI not changeable 31% 149Mb 61MiB/s baseline MinIO client not changeable 3% 679Mb 20MiB/s -67% rclone rcat not changeable 55% 9307Mb 77MiB/s +26% gof3r default settings 69% 252Mb 97MiB/s +59% gof3r 10Mb block, 16 threads 77% 520Mb 108MiB/s +77% xbcloud default settings 4% 96Mb 10MiB/s -84% xbcloud 10Mb block, 16 threads 59% 417Mb 123MiB/s +101%

Tip: Think about disaster recovery, and what will you do when the whole region is not available. It makes no sense to back up to the same region; always transfer backups to another region.

Google Cloud (From US to EU)

The backup data were streamed from Compute Engine instance in us-east1 to Cloud Storage europe-west3. Interestingly, Google Cloud Storage supports both native protocol and S3(interoperability) API . So, Percona XtraBackup can transfer data to Google Cloud Storage directly via S3(interoperability) API.

tool settings CPU max mem speed speed comparison gsutil not changeable, native protocol 8% 246Mb 23MiB/s etalon rclone rcat not changeable, native protocol 6% 61Mb 16MiB/s -30% xbcloud default settings, s3 protocol 3% 97Mb 9MiB/s -61% xbcloud 10Mb block, 16 threads, s3 protocol 50% 417Mb 133MiB/s +478%

Tip: A cloud provider can block your account due to many reasons, such as human or robot mistakes, inappropriate content abuse after hacking, credit card expire, sanctions, etc. Think about disaster recovery and what will you do when a cloud provider blocks your account, it may make sense to back up to another cloud provider or on-premise.

Conclusion

xbcloud tool (part of Percona XtraBackup) is 2-5 times faster with tuned settings on long-distance with native cloud vendor tools, and 14% faster and requires 20% less memory than analogs with the same settings. Also, xbcloud is the most reliable tool for transferring backups to S3-compatible storage because of two reasons:

.md5/filename.md5

PS: Please find instructions on GitHub if you would like to reproduce this article’s results.