How Businesses Are Using Flight for Cloud-Based Initiatives
Automating the movement of big data to the cloud
As a utility for automated bulk file movement, Flight does not have end-user interfaces and is not intended as a tool that operational staff will directly interact with. But any organization that is transitioning key business processes to the cloud will derive significant operational benefits by adopting Flight.
It supports a wide array of use cases requiring the automated movement of high volumes of large unstructured data sets to and from cloud object storage, and these applications support key initiatives of operational managers responsible for the business-centric aspects of data operations.
Flight use cases that apply across many industries:
➜ Ingest From Multiple Sites:
For many cloud applications, the data is generated across multiple geographically distributed facilities. In the case of video content aggregation, the content might be delivered from hundreds of producers. Clinical trial data might arrive from thousands of medical facilities, and retail ‘big data’ input from thousands of stores.
It is possible to first send all data from the satellite sites to one central data center and then upload it all to the cloud from there, but this is a highly inefficient, time-consuming and resource-intensive approach. With this workflow, all data must be sent twice, inbound and outbound network bottlenecks at the data center are likely to be problematic, and a big, expensive on-premises storage cache will be required.
This model makes use of cloud storage, but it fails to exploit the inherently distributed nature of the cloud. Flight provides a less expensive, more robust alternative for this type of distributed ingest use case.
By altering the workflow so that each remote facility sends data directly to the cloud via Flight, the network load can be distributed — and bottlenecks, single points of failure, and the expensive storage cache can be eliminated. A single customer account can be used in as many endpoint locations as needed, and there is a secure means of providing individual access.
The payload amounts are simply aggregated across all sites and billed to the master Flight account. Alternatively, there may be a business need to have separate Flight customer accounts at each of the remote locations, each bearing their own Flight and connectivity costs but all writing/reading to/from the same cloud storage tenancy. This too is possible with Flight.
➜ Content Distribution:
While distributed ingest is a very common use case, the reverse situation is equally prevalent in today’s ultra-connected world, as organizations seek to provide content and data-centric services to widely distributed endpoints. With its global reach and massive scalability, the cloud has become the world’s preferred platform for content distribution — and Flight plays a role in many of these use cases.
For video delivery, Content Distribution Networks (CDNs) have long been used to deliver the same piece of content to thousands of consumers. The major public cloud platforms now offer their own integrated CDN services, such as Amazon’s AWS CloudFront CDN or Microsoft’s Azure CDN, which use cloud object storage as the CDN origin storage.
Content owners are increasingly turning to these cloud offerings for their CDN needs, finding the workflows very efficient if content has already been sent to the cloud platform for other purposes. Flight is a perfect fit for these use cases, serving as a fast, elastic on-ramp to cloud CDN origin storage in either Amazon or Microsoft Azure.
The Media & Entertainment industry is also increasingly making use of the cloud for various other content distribution tasks, including VOD distribution and sending digital cinema packages to theaters for simultaneous release.
Parallel use cases are emerging in other industries, such as enabling remote physicians to review patient MRI scans, distributing the latest satellite imagery and maps to global clients, or making new CAD designs immediately available to worldwide factories. Here too Flight offers the ideal solution, with its built-in support for accelerated, reliable and secure downloading from multiple locations.
➜ Time-critical Workflows:
In today’s highly competitive world, the need for speed is pervasive across many of the industries that are turning to the cloud for their IT infrastructure needs. In the media sector, content owners and aggregators are using media processing in the cloud to prepare content for linear television, in which case there is an airtime imperative.
They are also using the cloud to distribute VOD content, and can’t derive revenue until it’s available to all distribution partners. Big data analytics applications generally aim to provide results in near real time so that action can be taken, and the ability to quickly move data into cloud storage enables cloud-based backup and business continuity strategies.
Historically the slow speed of cloud storage access was a barrier to cloud adoption for all of these time-sensitive use cases involving very large data sets. Flight removes this barrier.
➜ Data Processing & Analytics:
As more tools such as Hadoop MapReduce are developed for performing compute-intensive processing work within cloud environments, organizations are looking to the cloud as the most efficient and cost-effective approach to data crunching. Applications include media processing, such as transcoding and rendering of large video files, and a completely different class of use cases involving big data analytics.
From genome sequencing and analysis of clinical trial data to seismic data analysis, countless business intelligence applications utilizing large unstructured data sets are being deployed in the cloud. The ability to cost-effectively, rapidly, reliably and securely move the data into the cloud is a pre-condition for all such use cases, and Flight fills this need.
➜ Archiving, Backup & Disaster Recovery:
A less exciting, but equally important, set of cloud use cases involves the use of cloud object storage for the secure archiving and/or backup of crucial organizational proprietary data — and Flight can be a key enabler. Consider for example a situation where a post production facility wants to back up Work-in-Progress video files every night.
Using standard Internet transfers, it wouldn’t be feasible to get all the data into the cloud during an 8-hour period. But by using Flight, each day’s content can be uploaded to the cloud during off-hours, keeping the network available for working traffic during the day and putting it to work on backups during the night.
A copy can be pushed into cold storage (such as Amazon’s AWS S3 Glacier object storage tier) perhaps once a week, and the interim copies can be discarded. The asymmetrical network pricing of major cloud platforms is favorable in this case — there are no network costs unless there is a need to restore data.
➜ Minimizing On-premises Storage:
Another set of broad- based use cases supported by Flight is purely economic. As the amount of data in hybrid on-premises/cloud workflows increases, cloud upload speed constraints can mean that data is trapped in an on-premises storage pool while it waits to get to the cloud.
Even if there is no imperative to have fast access to the data within the cloud, this on-prem ‘pooling’ is costly. While it’s counter-intuitive in a hybrid cloud deployment, many customers see growing requirements for on-prem storage as they are forced to increase the size of the storage cache to accommodate more and more data awaiting transfer to the cloud.
In these cases there is a strong ROI for Flight as a tool for quickly clearing the cache, thereby minimizing the need to invest in additional on-premises storage.
➜ Signiant Manager + Agents System Extension:
For customers who already have Signiant’s enterprise-class Manager + Agents product installed in their facilities, Flight makes it very easy to start using cloud object storage in new or existing workflows.
Using pre-built Flight Upload and Flight Download workflow components, and with confidence that transfers will be fast, reliable and secure, cloud object storage can be treated as just another storage type. A sample Signiant workflow is available for reference.
