The Magic Behind Media Shuttle: Acceleration, Reliability and Security
The extreme files of Media & Entertainment
Imaging technology is continuing to advance, bringing exponentially expanding file sizes along with it. At the same time, those “extreme” files (a.k.a. extra large) are being transferred more often during the production, post-production and distribution process throughout a global network of producers, studios and broadcaster.
With 4k only recently commonplace, 8k has already started being used, especially at major sports events across the world. Back at the 2015 Women’s World Cup, Japanese broadcaster NHK was already working with FIFA “to produce ten matches in ultra-high definition 8K visuals, including the Final Match on July 5th,” and is planning to have 8K in time for the 2020 Olympics in Tokyo.
The intensity of live professional sports is captured beautifully with 8K. “8K images have incredible clarity,” said FOX Sports’ Head of Business Operations, David Nathanson, “and for those watching these matches it will be the next best thing to being at the games in-person.”
But 8K isn’t just twice as good as 4K; image resolution actually squares incrementally going from 2K to 4K to 8K, with 4K = 8 megapixels and 8K = 32 megapixels, which also equates to huge leaps in file sizes. Raw footage files that were already big at over 6000 GB on average for 90 minutes in 4K are nearly three times bigger in 8K.
The technology developed to transfer huge files
However, handling very large files isn’t anything new to media, and many big companies have been employing Signiant’s advanced large file transfer software throughout their data centers for well over a decade. In fact, other industries like life sciences and manufacturing — think next-generation genome sequencing and CAD files — are starting to adopt software originally designed to move large media files. That’s because, compared to every other type of data that’s transferred over the Internet, media files have always been large. Even a 1GB SD file causes trouble for traditional protocols like FTP.
Until recently, most smaller media companies had to rely on a mix of FTP and courier services exclusively, while larger companies were confined to their on-premises infrastructures. Exchanging large files electronically with collaborators and clients around the world was difficult for everyone. However, with the launch of Media Shuttle in 2012, the best of that enterprise file transfer technology was incorporated into a simple SaaS solution that’s quickly being adopted by small and large companies alike.
Media Shuttle has the easy feel of other cloud file sharing solutions made for smaller data types, but is the only true SaaS (software-as-a-service) solution on the market that can quickly transfer any size file, anywhere. The “service” aspect of SaaS means it is a fully managed solution; all of the technical backend expertise needed for equivalent on-premises solutions like load balancing and server management are taken care of for you.
Along with the clean and graphically enticing user interface, the SaaS benefits give Media Shuttle an air of simplicity that doesn’t seem to match its ability to handle such large files so fast, and people often ask us to explain the “magic behind Media Shuttle.” Well, the advances in file transfer technology that Signiant has developed include methods to enhance both acceleration and reliability, but we also must include Signiant’s security here. In today’s world, multiple layers of security should protect every important asset exposed to the Internet.
Acceleration
UDP acceleration (user datagram protocol) is often the category our acceleration technology is categorized as, but “UDP acceleration” doesn’t really cover the depth of innovation involved. In order to explain how we developed our the core acceleration technology used in all of our products from Media Shuttle to Flight, we have to include a short lesson on the way the large majority of data moves over the Internet, and how we’ve improved on it.
The protocol that has long provided a reliable stream of data moving from one point to another is called TCP (transfer control protocol). TCP is the protocol that operates behind both HTTP and FTP, and for most every form of data that’s transferred over the Internet, it works fine. But, with large files (really anything over 500MB) TCP starts to break down, especially if there is substantial distance between endpoints. That’s because TCP uses a relatively unsophisticated “sliding window” mechanism, which only sends a fraction of a file’s data and waits for acknowledgement that it’s been received on the other end before sending more data. With larger files, this back and forth creates a lot of latency, room for data loss and failure.
All of the retries and acknowledgment necessary with TCP also limit its ability to utilize available bandwidth. Unfortunately, many companies end up purchasing more bandwidth to try and improve their FTP’s speed and reliability, only to realize they’ve wasted budget. Higher bandwidth doesn’t help TCP/FTP. Additionally, protocols like FTP do not have “checkpoint/restart” functionality, which picks up data transmission where it left off in the event of Internet or computer failure. Meaning the entire file has to be resent if transmission is interrupted for any reason.
Signiant’s acceleration protocol utilizes UDP acceleration but also replaces TCP with a more advanced transmission control protocol, as well as an advanced file transfer protocol in place of FTP.
Reliability
Signiant improved on UDP acceleration mostly because standard UDP doesn’t guarantee reliability. It allows a chunk of data to be sent from one place to another on a best effort basis only, so we’ve implemented several reliability mechanisms such as flow and congestion control to compensate for any data loss. Additionally, in the event of a file transfer interruption, Media Shuttle will automatically pick up transmission where it left off: a feature known as checkpoint/restart.
Media Shuttle’s acceleration technology and reliability work together create the magic. Compared to TCP/FTP, Signiant can moved large files up to 100 times faster, mostly due to Media Shuttle’s ability to eliminate latency and capitalize on available bandwidth. The graph below compares three transfers using Signiant and TCP. Note that there is no improvement in TCP transfer speed with higher bandwidths; a 100 Mbps pipe moves one hour of HD content at the same rate as a 1 Gbps pipe with TCP. And the longer the distance between endpoints the more TCP performance stalls due to latency, taking over 21 hours vs. only 32 minutes with Signiant on a transfer between L.A. and Singapore.
Security
Cyber criminals’ tactics are constantly adapting to security practices, and staying ahead of them is an ongoing endeavor for any vendor providing business-critical software to companies, especially if it handles high-value assets like Media Shuttle does. Additionally, today’s users demand easy to implement security functions and high usability. Media Shuttle’s ease of use is critical to its security, as well as its layered security design and extensive 3rd party review; we actively hire security experts to try and penetrate our network, and take their recommendations into consideration as we further develop Media Shuttle.
While security best practices aren’t as fun to discuss as speed, they certainly contribute to Media Shuttle’s reliability. Without them, no amount of “magic” would be possible.
