Fast Data Transfer In and Out of Amazon Web Services like S3
AWS is the leading cloud platform for a wide range of applications across many industries. As businesses adopt AWS or begin to use more advanced AWS based services, they face decisions around the optimal mechanism for transferring data from their on-premises locations to the cloud.
Data ingest presents a particularly significant challenge for data-intensive verticals such as Media & Entertainment, Life Sciences, Geospatial Imaging, and Gas & Oil, and for use cases that include ‘big data’ analysis or large-scale data migration. The sheer volume of data can be enormous, and it often arrives from multiple locations and is needed for time-critical operations.
For these situations, AWS has partnered with Signiant to provide fast, flexible and reliable solutions for accelerated data movement into and out of AWS.
Choosing the right approach to AWS data ingest
There are a number of options for moving data into AWS, including:
- Shipping physical storage devices (AWS Snowball product family),
- Private direct connectivity via AWS Direct Connect,
- AWS software utilities, including the AWS command line interface (CLI) or other applications built on AWS’s SDKs,
- Amazon S3 Transfer Acceleration which leverages Amazon’s CloudFront edge locations,
Signiant’s cloud-native Flight, Manager+Agents and Media Shuttle offerings.
Choosing the optimal solution for each use case depends on the amount of data, the frequency of data movement, the number of source locations, and specific attributes of those locations.
Here are some things to consider during the process of selecting the best tool for the job.
Physical Media: If you’re planning a one-time move of several Petabytes of data and don’t have a tight timeline, shipment of physical storage devices can be a good choice. Remember that you will need in-house resources at the data center to load the devices, and security concerns may prohibit physical shipments in some situations.
Private connectivity: If most of your data is in a single location and you’re moving it to a single AWS data center, buying a dedicated network pipe such as Direct Connect can provide an efficient means of data transfer. If your data is widely dispersed, however, it’s generally impractical to procure dedicated connectivity for all locations. Utilization is also an important consideration – depending on your data flow, you may end up paying for a connection that is often under-utilized.
Reverse CDN: If your ingest location or locations are near Amazon’s CloudFront edge locations, you may be able to take advantage of S3 Transfer Acceleration which can take advantage of optimized network paths from the CloudFront edge location to the AWS data center.
AWS Software Utilities: Many routine data transfer needs can be accommodated with applications like the AWS CLI. Because these applications rely on TCP, however, they are susceptible to degraded performance in the presence of network congestion and/or when transmitting over longer distances (high latency). It’s therefore important to consider specific attributes of the source locations – including network capacity, network conditions and distance from the destination AWS data center.
Signiant acceleration for fast AWS transfers
Signiant Flight with AWS CLI or SDK based applications: If you are experiencing unpredictable or slow performance with utilities like the AWS CLI, pairing the AWS client software with Signiant Flight can solve the problem. This is particularly true if you’re transferring from multiple locations around the world.
Flight can be easily deployed with any HTTP-based application and uses Signiant’s UDP acceleration technology to minimize the impact of latency and congestion – thereby enabling full utilization of the available bandwidth. Because Flight can be deployed anywhere and you will only be charged when you’re transferring data, it can be thought of as “Software-Defined Direct Connect”.
Signiant Flight with Signiant Manager+Agents: For advanced enterprise use cases, the combination of Manager+Agents and Flight provides a complete solution for managing and orchestrating the large-scale movement of data in and out of AWS.
Data can be transferred over standard Internet connections or private AWS Direct Connect connections, and you will have unified visibility and control of all transfers. This combination also provides a highly reliable mechanism as interrupted transfers are automatically restarted and the bandwidth used for the transfer can be controlled by the administrator.
Signiant Media Shuttle: Many cloud use cases involve data transfer by people, either as an ongoing part of the workflow or on an occasional, ad hoc basis. Signiant’s Media Shuttle SaaS offering meets this need via a simple Web interface that enables the transfer of large files to and from your AWS S3 buckets.
Benefits of Signiant for AWS data ingest
In summary, Signiant solutions provide unique value in situations where large volumes of data, long distances, multiple source locations, or flaky networks are involved. Unlike standard transfer mechanisms, our software enables cloud adoption by providing:
Speed: With Signiant’s patented transfer protocol and scale-out architecture, you can take advantage of multi-Gbps transfer rates to AWS – even in the presence of latency and congestion.
Cloud-Native Implementation: Both Flight and Media Shuttle are cloud-native solutions, so there’s no need to procure or manage any cloud infrastructure. We handle all of that, and our 24/7 DevOps team monitors the system for thousands of customers worldwide.
Security: All Signiant solutions are designed from the ground up to be highly secure. Our software is trusted by most of the world’s top media enterprises to move their valuable content around the world.
Learn more about Signiant Flight for fast data transfers to Amazon Web Services.