Solutions for Big Data File Transfer

Moving Big Data between locations or into the cloud

Many organizations generate massive unstructured data sets that need to move across the world safely and efficiently as part of their global video production workflows or big data analytics with Hadoop. Unfortunately, many of the conventional transfer methods fail under the weight of these massive data volumes and distant networks.

Organizations that need to move large unstructured data sets across the world or into the cloud are limited by outdated technology like FTP.

Many businesses still use network protocols like FTP and HTTP to transfer their data, but they’re highly inefficient at moving massive files in high latency, high bandwidth networks. This is a pain that professionals in film and television have long known about, and where Signiant’s technology first gained footing transferring video files during production, post production and distribution.

Thanks to increasing innovations in cloud computing and the development of SaaS/PaaS (Software-as-a-Service/Platform-as-a-service), file acceleration technologies have become accessible to companies of every size. Signiant is the pioneer in developing true SaaS solutions for large video transfers and any other unstructured data type. Our person-to-person file acceleration solution, Media Shuttle is widely used across the media industry and increasingly in other industries that deal with video production, such as video marketing and advertising. And that same core technology so adept at quickly moving large files is also used within Flight, a solution that specializes in accelerating big data sets to and from cloud storage.

Public cloud computers give limitless, elastic supply of computing power, networking, and storage. For companies looking to adopt big data analytics, the cloud offers some distinct advantages. Combining the capabilities of cloud computing with SaaS/PaaS’s pay-per-use pricing model turns software into a utility service. This allows organizations to pay strictly for what they use without having to invest in infrastructure or maintenance.

Businesses that continue to expand their use of big data analytics are finding new and innovate ways of storing it. However, one major step of the process that stays neglected is the transfer of the data, which is where Flight comes in.

Companies with separate storing and analysis locations need a solution to quickly transfer the massive quantities of unstructured data to and from their shared storage locations like HDFS (Hadoop distributed file system) or Amazon S3, prior to analysis. Most unstructured data used for big data analytics aren’t created where they are analyzed. Instead, they are created at distributed locations that need to be transferred to a central analysis point.

For instance, effective image analysis of surveillance video through a Hadoop cluster requires the footage to be captured at a remote location. Then surveillance video must move from its remote camera location to a shared storage space that can be reached by the cluster. If each minute of HD video is recorded at 50 Mbps, almost half a gigabyte of data is produced. To move this much data quickly, as well as any other type of massive unstructured data, an organization will need a more advanced transfer technology than FTP or HTTP.

A scalable big-data file transfer solution through SaaS

Businesses need to transfer data out of storage and into analysis without the threat of meeting storage capacity. Accelerating the transfer process allows companies to better manage their storage space, cutting storage expenses and freeing management. Companies that need real-time analytics for their competitive advantage will see quicker turn-around times leading to greater return-on-investment.

With cloud computing taking foot as a business standard, moving data to the cloud for storage or analysis has become a part of every company’s workflow, particularly those that are becoming data driven. Understanding the many methods of data transfer — both old and new — are pivotal to business’ big data or cloud storage strategy.