Best Practices in DevOps
DevOps brings together the two crucial elements of software engineering, development and operations. The DevOps methodology combines the entire software development, deployment and monitoring lifecycle and ensures 24/7 availability for customers. Ideally, DevOps is a cross-functional practice where developers are actively involved in releasing and monitoring the software they write. These days with cloud computing, there’s very little difference between the infrastructure and the code running on it.
“It used to be that developers would write code, testers would test the code, and operations would deploy and monitor,” says Dave North, Signiant’s Director of DevOps. “But we’ve slowly been moving towards a process where developers can release their own services and monitor them themselves. With the infrastructure and the code being so tightly coupled, developers understand their service best since they designed and wrote it, so it really improves efficiency and problem solving when there’s an issue.”
Automatic Product Updates and Continuous Monitoring
The benefits of DevOps services to customers can be simplified into two categories: product updates and monitoring. Signiant’s SaaS customers automatically receive product updates, with no downtime or maintenance windows needed. That customer benefit comes from a finely tuned system developed by our DevOps team.
“We have a fairly elaborate system that supports developer productivity and rapid release cycles. It also allows us to release new updates and roll them back quickly if there’s a problem,” says Dave.
DevOps takes responsibility for everything working as it should for customers at all times — which translates into watching over a large number of automated monitors. Signiant’s DevOps team has built up operational excellence over the course of several years as we transitioned to SaaS.
“When I first began working for Signiant over a decade ago, we were shipping CDs to customers and really only knew if something was broken when a customer called,” says Dave. “Today, we have hundreds of advanced checks running 24/7 that tell us all the different ways that things could be inoperative, whether its Amazon, Azure or our own stuff. If something goes wrong, we can usually fix it before it impacts customers. And a lot of it is automated.”
“If Amazon is having a problem in Virginia, for instance, our system will automatically failover to another region. We’ve invested heavily in a multi-region system, which is a lot of work but is incredibly important for our customers.”
While much of DevOps is automated, someone from Signiant’s DevOps team is on call at all hours, and they get alerted if there is a problem anywhere in the system, whether it’s in the middle of a meeting or REM sleep.
Security
Signiant is responsible for customer data and takes security very seriously. DevOps is a core aspect of Signiant’s defense-in-depth security practices and have software packages, audit tools and scanning tools to make sure we are as secure as we possibly can be.
“We run a commercial network security intrusion tool, and do on-going scans to make sure there aren’t mis-configurations or other known vulnerabilities we need to be aware of,” says Dave. “We also monitor for extraneous error conditions on our customer portals. For example, yesterday we received a large amount of suspicious errors from one of our customer’s Media Shuttle portals. After investigating the incident, we found that the customer themselves was doing a security scan, but we treat all suspicious activity the same just to make sure.”
