In this SlideShare, Matt Yanchyshyn, AWS Solutions Architect and Nelson Hsu, Signiant VP of Business Development & Alliances explain why companies are moving large data volumes to cloud storage, and how to use Signiant Flight to easily accelerate content into and out of Amazon S3.
Moving Big Data Into Cloud Object Storage | Signiant
Matt Yanchyshyn, AWS Solutions Architect
Nelson Hsu, Signiant VP Business Development & Alliances
How do we define Big Data?
Unconstrained growth: Big Data is moving fast
Why is Amazon S3 good for Big Data?
Moving Big Data to and from Amazon S3
Signiant launched a new product last year called Flight that provides an easy way for AWS customers to push large amounts of data into Amazon S3 (and easily pull it back out) without worrying about managing cloud infrastructure.
Signiant Flight is an easy way to move data to and from S3 at high speeds:
A few ways businesses are using Flight:
Flight is hybrid Software-as-a-Service (SaaS)
Flight is the only SaaS solution on the market for accelerated file transfers to and from cloud object storage. The SaaS component makes Flight unique for several reasons:
Not all SaaS is created equal
Subscription + Management v. BYOL
While BYOL (bring your own license) models certainly have their place, there is a significant difference between them and Subscription + Management payment models like Signiant’s.
Namely, with BYOL, you still have to manage, maintain and support your own servers.
With Signiant’s subscription service, they cover all of that for you, significantly reducing both Opex and Capex.
A fully managed service
Signiant Flight eliminates the overhead of managing compute resources in the cloud. Signiant manages the server-side component – the Amazon EC2 instances running Flight servers and the Amazon S3 transfer components – while end users run a lightweight, client-side agent.
All you have to do is:
Highly reliable without a complex setup
When you use Signiant Flight to send files to Amazon S3, its backend automatically scales during high-volume transfer cycles.
Flight’s backend is load-balanced across multiple Amazons EC2 instances spread across multiple AWS Availability Zones, so it is highly reliable without passing the complexity of configuration management on to you.
Encryption and Checkpoint Restart
Importantly, Signiant’s file transfer protocol also supports two features that are not supported in Tsunami UDP:
If a transfer is interrupted for any reason, the transfer is restarted (using numerous file retry algorithms) and continues transferring from the point of interruption. If a file already exists in Amazon S3 and hasn’t been changed, Flight won’t upload the file.
Why is Flight so fast?
Signiant’s patented accelerated file transfer protocol is often called UDP acceleration, but it actually implements both an advanced TCP on top of UDP and an advanced FTP. If that intrigues you, read more about it here.
Basically, this minimizes the impact of WAN latency on throughput which results in considerably faster transfers, especially for large files transferred over long distances. Once files arrive on Signiant Flight’s AWS-based backend, servers managed by Signiant write the data directly into Amazon S3 over HTTPS with the multipart upload API.
Over long distances, Signiant technology minimizes the impact of latency, while being able to capitalize on increases in bandwidth. TCP based protocols do not benefit from increases in bandwidth and are very in efficient over long distances due to latency.
Setting up Signiant Flight
Note: Flight comes with a command-line interface and other client options. To learn more, check out Flight’s client options.
Setting up the Command Line Interface (CLI)
A more complex file transfer may involve a large number of files listed, one file per line, in a manifest:
In this case, we use interactive move (-i) to see file transfer statistics in real-time and generate detailed transfer statistics (-z) at the end of the transfer.
Signiant’s Flight is an easy way to move big data into the cloud at high speed. Because it’s a SaaS solution, its highly available and high performance file transfer system is deployed and maintained for you.
Flight’s encryption in transit and intelligent file transfer guaranteed delivery means that you can send files securely and reliably. It’s easy to use and get started!