Moving large data volumes to cloud services? Dedicated connection vs. upload utilities

Cloud services such as object storage are increasingly being adopted by businesses of all kinds, however standard mechanisms for cloud I/O (input/output) have significant limitations when moving large data volumes such as media files. 

Typically, data is moved to/from cloud services over the Internet through HTTP with TCP. While sufficient for most Internet traffic, the protocols struggle when sending large blocks of data to a destination, and slow down dramatically with high bandwidth and the latency and packet loss associated with large files and distance. 

Cloud services providers like as Amazon Web Services and Microsoft Azure offer several options to overcome this challenge. But if you are looking for an electronic, network-based approach, the options can be narrowed down to two basic categories that may or may not be ideal, depending on the situation.

Dedicated Connectivity

Dedicated network connections such as AWS Direct Connect and Azure ExpressRoute provide ways to connect directly into cloud infrastructure rather than going through the public Internet. This will minimize latency, but requires ordering a fixed circuit from your local Telco (if it’s available in your area) that runs from your location to the the cloud provider’s POP.

While a costly investment, it can be a good choice if you have a fixed amount of data being transported every day from the same location. However, it is a poor investment if you aren’t using it most of the time.

Upload Utilities

Cloud providers offer upload utilities that are cost-effective for moving low-volume data. However, they do not include acceleration technology, so can be slow and unreliable for large data volumes.  

Some upload utilities address latency by breaking objects into chunks and moving them via multiple parallel HTTP streams, which helps to some degree. However, these “multipart upload utilities” (such as Microsoft AzCopy) do not include acceleration, but they can be used with Signiant Flight to optimize each stream for the most efficient data movement possible.

For large data I/O to Amazon Web Services or Microsoft Azure, especially if time is a constraint, Flight is the fastest and most secure method available. Flight can be used with any client software (such as Signiant, AWS CLI or Microsoft AzCopy) to achieve unprecedented, multi-Gbps speeds.