cloud computing. The data transfer and storage concept consists of a white polygonal interconnected structure within it. Dark blue background with small padlocks scattered on the background.

Image Credit: TU IS

Komprise, a data management tools provider, has rolled out Hypertransfer for Elastic Data Migration, a solution that can speed cloud data transfers by 25 times, the company claims. This engineering feat highlights the industry’s efforts to address challenges in repurposing legacy protocols for the cloud.  

One core issue is that the server message block (SMB) protocol, one of the most popular file-sharing protocols, has never been updated for the cloud. It was initially developed in 1985 to provide shared access to files and printers over a local area network (LAN). It worked well when files had to be shared across just a few hops. 

However, SMB is a chatty protocol, which adds a lot of overhead when communicating over TCP/IP and across multiple routers. A large data transfer needs to wait for acknowledgment for each file, so the lag time can add up when transferring many small files. This can be a big issue for use cases such as electronic design automation and multimedia workloads that often involve large numbers of small files. 

Komprise Hypertransfer helps cache the messages to minimize the number of roundtrips for wide area network (WAN) transfers. It also sends data across multiple parallel channels in parallel. The company reports transfers up to 25 times faster than with the Robocopy command popularized by Microsoft. The speed-up drops to about 20 times when transferring fewer, larger files, such as copying an Android phone directory. 

Other factors slowing data migration

Komprise CEO and co-founder Kumar Goswami said several other factors besides chatty protocols can slow data migrations. Unstructured data is one. The term refers to all types of files outside the well-formatted realm of databases. Most apps save raw data across multiple files for a given project, user or use case. As a result, more extensive unstructured data migrations can involve moving billions of files. 

Another issue is that these files may be strewn across multiple users, volumes or locations. Migrating this data is not as simple as starting at the first file and moving to the last in a single table or directory. The data migration tools need to know what data to migrate and in what order. This can add additional overhead. 

A third factor is that data migrations often involve moving files from data stored on one vendor’s equipment and software to another’s. It also sometimes requires translating across protocols and architectures, such as moving files stored on traditional PCs to objects stored in cloud services like S3. 

“Different systems do not have the same security, storage and metadata capabilities, so the data migration solution needs to be able to bridge across these,” Goswami explained.

A tough problem to solve

In addition to these technical problems, organizations must figure out how data migration fits into their overall cloud migration strategy. Teams need to consider the network, the security of the data in the cloud, the financial operations and ongoing costs of the data and where best to keep it. 

Larger enterprises have traditionally engaged professional services firms with cloud data migration experience to help address these challenges. Komprise has been working on integrating these capabilities into a standardized cloud service to help data migrations become part of a long-term data strategy rather than a one-off project.

“Data migrations are now a part of a broader data management strategy and are intricately linked,” Goswami explained.