Many cloud storage applications involve the need to ingest data from multiple locations around the world. It’s common to start by outlining an architecture that involves sending all files to a central data center, and then uploading them to cloud storage. This is directly analogous to prior-generation processes of sending tapes or disks to a central location, so it’s familiar and makes sense at first glance.
In practice, however, there are a number of problems with this approach. Not only does it require unnecessarily sending the same data twice, it creates a data bottleneck in one location and doesn’t take full advantage of the distributed nature of the cloud.
This article will review these limitations and describe an architecture that is much more efficient in many situations: direct ingest to cloud storage from multiple locations. This approach is enabled by Signiant Flight, a SaaS solution for accelerated large file transfers to and from cloud storage. Flight is available for both Amazon S3 and Microsoft Azure blob storage, and is particularly impactful for remote locations without low-latency access to cloud data centers.
Every situation is different, but following factors will often make a direct-to-cloud architecture more efficient and cost-effective than a central repository model.
When the system is designed to require that all data flow first to a central data center, most content needs to be moved twice. Since every transfer involves time, cost, and potential failure, this is intrinsically inefficient. With a distributed architecture, each data set only needs to move once.
For the data center that’s receiving everything, inbound bandwidth constraints can slow transfers and create a single point of failure. A large, expensive storage cache is required to hold data before cloud upload, and a pipe large enough to handle the upload throughput is also going to be expensive, and — no matter how big the pipe — it will eventually fill up. Both of these issues can be eliminated if each data-generating location simply uploads directly to the cloud.
In the centralized scenario, the primary data center pipe would likely be a fixed circuit option from a cloud provider like Azure ExpressRoute or Amazon Direct Connect. These circuits are not only expensive, they also tie you in to a single geographic location and a single cloud platform provider.
If you choose a distributed ingest approach, each contributing location will need to be equipped with a means of initiating data transfers to the cloud. FTP is very clunky, and slow for remote locations. Multi-part upload utilities might provide some improvement, but will be unlikely to provide adequate speed and reliability across the entire ecosystem.
This is where Signiant Flight comes in. With a simple software client installed at each data source location, Flight’s advanced acceleration technology enables the transfer of any size file into either AWS or Azure – reliably, securely, and up to 200 times faster than FTP. Flight allows each business location access to the closest cloud vendor data center, and end-to-end encryption provides additional security for data in transit and at rest.
Some companies have a good reason for routing data through a central location before uploading to cloud storage. If a company simply must transfer data before upload, remember Signiant acceleration technology can improve file transfer speeds and is especially impactful for large files. Two of Signiant’s products are used for this kind of workflow.
There is one more scenario where Flight, in combination with cloud storage, is especially impactful. Many companies that need to distribute the same content to multiple locations around the world send that file out from one location, repeatedly. This architecture saturates the internal network with repeat load of the same file and doesn’t fully utilize the speed of the destination’s network. Because it is a single pipe from one source, it must be split several ways, slowing down the file transfer in direct proportion to the number of sends.
A far more efficient way is to send the file to cloud storage once, and provide each receiving location with a Flight client. When these locations require the file, they can use Flight to pull it in a secure, accelerate manner from a nearby cloud location. The removes the network bottleneck from the central data center, and enables each endpoint to fully utilize its local network bandwidth.
Flight provides many benefits that modern global businesses need to effectively utilize cloud storage, especially for content ingest and distribution. These are:
Speed: Up to 200 times faster than FTP/TCP, maintaining transfer speed regardless of latency and distance between end points.
Reliability: Auto recovery from interrupted file transfers allow restart without losing previously transferred data.
Security: AES256 Encryption in transit and encryption at rest. Obfuscation of cloud credentials and centralized access key management.
Elasticity and Flexibility: Scale up and down as data loads vary. Deploy to many for few endpoints. Transfer any size file and any amount of data. Maintain cloud independence, use AWS and/or Azure.
Ease of Use and Deployment: No need to procure and manage cloud servers and software for acceleration. Simple client download for on-premises endpoint.
Cost Effectiveness: Less expensive and more flexible than dedicated connectivity (AWS Direct Connect or Azure ExpressRoute). Only pay for what you use.