Seven Pitfalls of Relying on rsync to Automate Large File Transfers

The original rsync (remote sync) was launched in June of 1996, written by Andrew Tridgell and Paul Mackerras to offer synchronization and bulk file transfer between servers. The rsync algorithm quickly became widely deployed and is included today with most Linux distributions. As times change, the needs that Mackerras and Tridgell designed rsync to meet still exist, but — due to the increase in file sizes and the growing complexity of workflows — particularly in media and entertainment — the utility just isn’t well suited for the challenges of the modern media supply chain.

Why do Media & Entertainment Businesses Still use rsync? 

As is often the case with legacy technologies, many media and entertainment businesses continue to rely on rsync out of familiarity or the misguided perception that it’s free. While there’s technically no charge to use rsync, the “hidden costs”, as outlined below, can be enormous and unpredictable. Unfortunately, in failing to leverage more modern solutions, the media organizations that continue to depend upon a utility that’s rife with problems put themselves at a disadvantage. In exploring how to help businesses find the system-to-system transfer solution that they both need and deserve, Signiant has compiled some of the major pitfalls of continuing to rely on rsync and provides M&E leaders an introduction to the ideal alternative.

The Pitfalls of rsync

1. rsync wasn’t designed to move large files with speed 

Because rsync was designed back in the mid-1990s (more than 20 years ago!) without any file acceleration technology, moving large video files over long distances or congested networks takes far longer than necessary in the modern age. That deficiency alone can cost hours, or even days, depending on the file sizes and network conditions. Likewise, rsync can struggle when transferring folders that contain tens of thousands or even millions of files, which can be common in VFX and other media workflows. In an industry that moves fast and relies on massive files and global collaboration, rsync isn’t nearly fast enough.

2. rsync is a techy, command-line tool 

Anyone who has used rsync will likely admit that, while it may be familiar to them, for newcomers, it isn’t exactly a friendly tool. In order to make transfers, users must learn a swath of command-line arguments, which will read like a foreign language, and a few wrong characters can inadvertently wipe out important files and directories or cause other chaos. As with most techy tools, it also can be difficult to modify an rsync command that someone else put in place. So, when a change is needed or an error occurs, good luck if that original systems administrator isn’t around to help.

3. rsync lacks real-time visibility and reporting 

When moving large, high-value digital assets between locations, it is critical for media organizations to understand what’s going on with their content at all times. Media content is immensely valuable and deadlines are tight. Businesses need to know where their files are and if something does go wrong, they need to be able to quickly diagnose the problem and fix it. In this category, rsync offers little visibility and no visual reporting leaving IT teams to write scripts, integrate third-party tools and parse log files to understand what’s happening with any rsync jobs.

4. Without built-in alerts, rsync errors can be devastating 

Not only does rsync not provide visibility, but should a transfer fail, rsync offers no built-in alerting to make teams aware of such interruptions. Systems administrators are expected to build their own custom alerting scripts or worse, find out about a failed transfer long after it occurred. This is risky for such a deadline-driven industry.

5. What about intercompany transfers? 

When working regularly with partners, it’s common to set up automated transfers between companies. With rsync, that would require close coordination between tech resources on both ends to set up and maintain the jobs. It would also require a great deal of trust as a company would have to open up their network to an outside partner without straightforward control of what they can do or easy visibility into what the partner has done. Intercompany collaboration is a must in the media industry and rsync is not the right tool for that.

6. rsync security leaves much to be desired 

In an age of high-profile breaches at organizations as prominent as HBO and Disney, security is at the top of mind for most M&E companies, and — in that department — rsync doesn’t offer much in the way of help. While there are workarounds for the fact that rsync doesn’t natively encrypt data in flight, it’s just another concern for IT resources to include on their long rsync checklist.

7. The gap is only increasing 

As the size of files continues to grow, media channels diversify, businesses become more global, and the entire media supply chain becomes more interconnected, the gap between legacy tools such as rsync is only increasing.

It Doesn’t Have to be This Way

Signiant has long striven to provide the very best file transfer solutions to media and entertainment enterprises around the world and to help ensure that these enterprises don’t find themselves relying on subpar or outdated tools that might jeopardize the success of their businesses. Signiant Jet is a simple yet powerful SaaS solution for automated, lights-out, system-to-system file transfers. Jet allows M&E organizations to abandon rsync and all its pitfalls and embrace a better way to automate large file transfers all around the world.

Who is Jet For? 

For organizations with multiple locations or who regularly work with partners, Jet provides media businesses of all types with an easy, fast, reliable and secure option for system-to-system file transfers.

What Does Jet Offer?

• Lightning-fast acceleration
  Jet offers Signiant’s fastest transport yet. With multi-Gbps transfer speeds, Jet’s acceleration is immensely impressive, especially for organizations with recurring, time-critical transfers that involve large files moving over long distances. On top of all of this, Jet amplifies its efficiency and speed via an intelligent transport mechanism that uses machine learning to optimize performance.
• Reliability
  With robust visibility features, reporting capabilities, and error notifications, Jet allows businesses to stay on top of their transfers. With Checkpoint Restart, any transfer that encounters an interruption will restart from the exact point it left off, saving organizations time and anxiety. Its simple, web interface allows operations teams and administrators to see their content moving in real time, detect anomalies, and help diagnose and remedy problems.
• Enterprise-grade security
  Building on the core Transport Layer Security (TLS) in the Signiant transfer protocol, Jet adheres to the information assurance principle of defense-in-depth, incorporating multiple layers of security controls. Trusted across the Media & Entertainment industry and beyond, Signiant was awarded the DPP ‘Committed to Security’ mark for both production and broadcast.
• Easy, secure intercompany exchange
  Signiant has long served as the trusted intercompany broker for secure exchange of media assets throughout the global supply chain. Jet builds on this legacy by providing a simple next-generation SaaS solution for companies to exchange data with one another through a secure handshake mechanism, all managed from Jet’s web interface.
• Cloud-native SaaS
  Because Jet is a cloud-native SaaS solution, deployment and management are a snap. Jet is designed to be as intuitive as possible. It only takes a few steps to get started: establish endpoints, create routes, and set up transfer jobs. Furthermore, it’s easy to implement in organizations of all sizes, and — as enterprises and their content continue to grow — Jet will always help them achieve fast file transfer between as many locations and with as many partners as necessary.

• Storage independence
  In keeping with Signiant’s dedication to ensuring that organizations maintain full control over their own content, Jet offers the benefits of cloud-native SaaS while ensuring that companies are always in charge of their assets and always the owners of their storage. Like Signiant’s other solutions, Jet is compatible with on-premises storage from any vendor.

rsync is Holding Media Organizations Back

Media organizations that rely on legacy tools like rsync are missing out on the speed and agility provided by more modern solutions. With lack of visibility, its techy command-line interface, and the inability to efficiently handle the massive files that M&E businesses need such a solution for, rsync just isn’t the right tool for the modern media supply chain. 

For enterprises that are tired of settling for less and exhausted from hindered workflows, Jet offers a way forward for system-to-system file transfer. Designed specifically to make life easier, faster and more secure, Jet boasts a range of benefits and, most importantly, is supported by a company dedicated to addressing the specific challenges that media organizations too often face when saddled with the wrong tools. 

Businesses that ditch rsync won’t be disappointed. They can, however, look forward to an easier, faster, and more secure transfer experience, and to the thrill of watching their content truly soar. 

Contact us to learn more about replacing your outdated rsync scripts.