A little over a year ago, the DevOps team at Sonian was more system administration than development operations. While the team automated with Chef and had implemented basic monitoring, the typical day was spent responding to unplanned issues across our infrastructure. We were in the trough of disillusionment of DevOps and knew something needed to change. The catalyst for change came unexpectedly one night in the form of a production incident caused by unnecessary human error (that through good fortune had no direct customer impact).
What followed this incident was a commitment to take a different approach to DevOps. We brought in a new manager, added some talented engineers, provided the team needed breathing room, and invested in the critical path projects the engineers felt would pay the most dividends: proactive monitoring, cookbook standardization, and streamlined release management. What followed was a series of successive changes that resulted in a “tipping point” for our DevOps investment, driving increased reliability, quality, scalability and maintainability of our infrastructure. This investment has also resulted in a series of powerful innovations in asset, cost, security and system management, some of which we have found the time to provide back to the open source community (e.g. Sensu, SmartCloud implementation of Fog).
In reflecting back on the year, we iterated multiple times through what I now realize was a virtuous cycle. We initially invested in automation in order to increase the standardization and reliability of our cloud infrastructure. This investment resulted in better quality and maintainability of our infrastructure, which allowed us to be more agile in our release cycles. Faster releases allowed us to increase our pace of innovation, which provided better operational and system scalability. The increased scalability allowed us to get ahead of our growth, enabling investments in proactive automation and tools, which resulted in the virtuous cycle starting again.
While there is likely no single playbook to get a DevOps team to enter the virtuous cycle, ours started with a talented team, good leadership, commitment to the culture of DevOps, and a willingness to follow the instincts of the engineers. So as we reach the one year anniversary of the start of the virtuous cycle, I'd like to provide a thanks to the team that through talent, hard work and personal pride drove this transformation: Sonian DevOps.
Related Posts: Cheslock's Hierarchy of DevOps Needs, The Death of DevOps