Moving large data volumes to cloud services? Dedicated connection vs. upload utilities

A camera lens aperture with a rocky landscape in view.

Cloud services such as object storage are increasingly being adopted by businesses of all kinds, however standard mechanisms for cloud I/O (input/output) have significant limitations when moving large data volumes such as media files. 

Typically, data is moved to/from cloud services over the Internet through HTTP with TCP. While sufficient for most Internet traffic, the protocols struggle when sending large blocks of data to a destination, and slow down dramatically with high bandwidth and the latency and packet loss associated with large files and distance. 

Cloud services providers like as Amazon Web Services and Microsoft Azure offer several options to overcome this challenge. But if you are looking for an electronic, network-based approach, the options can be narrowed down to two basic categories that may or may not be ideal, depending on the situation.

Dedicated Connectivity

Dedicated network connections such as AWS Direct Connect and Azure ExpressRoute provide ways to connect directly into cloud infrastructure rather than going through the public Internet. This will minimize latency, but requires ordering a fixed circuit from your local Telco (if it’s available in your area) that runs from your location to the the cloud provider’s POP.

While a costly investment, it can be a good choice if you have a fixed amount of data being transported every day from the same location. However, it is a poor investment if you aren’t using it most of the time.

Upload Utilities

Cloud providers offer upload utilities that are cost-effective for moving low-volume data. However, they do not include acceleration technology, so can be slow and unreliable for large data volumes.  

Some upload utilities address latency by breaking objects into chunks and moving them via multiple parallel HTTP streams, which helps to some degree. However, these “multipart upload utilities” (such as Microsoft AzCopy) do not include acceleration, but they can be used with Signiant Flight to optimize each stream for the most efficient data movement possible.

For large data I/O to Amazon Web Services or Microsoft Azure, especially if time is a constraint, Flight is the fastest and most secure method available. Flight can be used with any client software (such as Signiant, AWS CLI or Microsoft AzCopy) to achieve unprecedented, multi-Gbps speeds.


Suggested Content

Signiant Post-NAB 2022 Wrap-Up Report!

What did Signiant present and learn at the 2022 NAB Show? Learn about new Signiant technologies and what trends...

Fit for Purpose

Without an appropriate media transfer solution, creative professionals will find workarounds that work best for them, not everyone. Read...

Metadata Everywhere: Metadata-driven Workflows

Metadata-driven workflows simplify complexity for our customers and can be extremely powerful. To understand their power, it helps to...