While data growth is happening in almost every industry today, no where is it being generated and put to beneficial use quite like the pursuit of scientific discovery. The cloud, with its virtually unlimited compute resources and global access, is an obvious partner to scientists. And the Microsoft Azure for Research project, launched in September 2013, was specifically developed to help the research community use the cloud to advance scientific discovery.
However, this is relatively uncharted territory, including both the movement of big data to the cloud and how to process it once it there. Fully utilizing the power and scalability of cloud computing in scientific research is still in the discovery phase itself. Even though the “fourth paradigm” of data-intensive science is well under way, cloud enabled potential for collaboration and greater access to compute power has yet to be fully realized.
Because the scale of data being generating is so staggering — from huge-scale projects like the Large Hadron Collider and Sloan Digital Sky Survey that regularly produce petabytes of data to a single next-generation genome sequencing run at nearly a terabase (about 1/3 of a terabyte) — moving all that data to the cloud has been a barrier to even beginning to utilize cloud servies. And some believe it’s actually impossible.
In a recent article on re/code called The Third Phase of Big Data, Talli Somekh, CEO of a data science platform company for computational biology, said, “If you speak to any company [in life sciences], it is a major issue. It’s a matter of physics … some of the data sets are so large you just can’t move them to the cloud.”
But Azure and Signiant are out to prove that wrong. Signiant has been developing advanced protocols for transferring very large data sets over IP networks for well over a decade and recently launched Flight for large file uploads (and downloads) to Azure blob storage.
Signiant Flight is one of the first SaaS solutions certified for Azure Marketplace Developer Services, and can move any size file or data set. Flight also brings additional permissioning and security not available with Azure alone.
Combined with the Azure for Research project, Flight can enable the seemingly impossible movement of very largest scientific data sets to the cloud. And, once all that data is in the cloud, the real discovery of cloud enabled research begins.
If you’d like to learn more about Azure for Research, the SlideShare below written by Dr. Kenji Takeda is excellent.