DevOps has finally ushered in the era of greater collaboration between teams. Organizations today realize that they can no longer work in siloes. To achieve the required speed of delivery, all invested in the software delivery process, the developers, the operations, business teams, and the QA and testing teams have to function as one consolidated and harmonious unit. DevOps provides organizations this new IT model and enables teams to become cross-functional and innovation focused. The conviction that DevOps helps organizations respond and adapt to market changes faster, shrinks product delivery timelines, and helps to deliver high-quality software products is reflected in the DevOps adoption figures. According to the Puppet State of DevOps Report, in 2016, 76% of the survey respondents adopted DevOps practices, up from 66% in 2015.

One of the hallmarks of the DevOps methodology is an increased emphasis on testing. The approach has shifted from the traditional method of adding incremental tests for each functionality at the end of each development cycle. The accepted way now is to take a top-down approach to mitigate both functional and non-functional requirements. To achieve this DevOps demands a greater testing emphasis on test coverage and automation. Testing in DevOps also has to start early in the development process to enable the DevOps methodology of Continuous Integration and Continuous Delivery.

The Role of Testing in Continuous Delivery and Continuous Integration:

In order to deliver on the quality needs, DevOps demands that testing is integrated into the software development and delivery process and acts as a key driver of DevOps initiatives. Here, individual developers work to create code for features or for performance improvements and then have to integrate it with the unchanged team code. A unit test has to follow this exercise to ensure that the team code is functioning as desired. Once this process is complete, this consolidated code is delivered to the common integration area where all the working code components are assembled for Continuous Integration. Continuous Integration ensures that the code in production is well integrated at all levels and is functioning without error and is delivering on the desired functionalities.

Once this stage is complete, the code is delivered to the QA team along with the complete test data to start the Continuous Delivery stage. Here the QA runs its own suites of performance and functional tests on the complete application in its own production-like environment. DevOps demands that Continuous Integration should lead to Continuous Delivery in a steady and seamless manner so that the final code is always ready for testing. The need is to ensure that the application reaches the right environment continuously and can be tested continuously.

Using the staging environment, the Operations teams too have to run their own series of tests such as system stability tests, acceptance tests, and smoke tests, before the application is delivered to the production environment. All test data and scripts for previously conducted application and performance tests have to be provided to the operations teams so that ops can run its own tests comprehensively and conveniently. Only when this process is complete, the application is delivered to production. In Production, the operations team has to monitor that the application performance is optimal and the environment is stable by employing tools that enable end-to-end Continuous Monitoring.

If we look at the DevOps process closely we can see that while the aim is faster code delivery, the focus is even more on developing error free, ready for integration and delivery code by ensuring that the code is presented in the right state and to the right environment every time. DevOps identifies that the only way to achieve this is by having a laser sharp focus on testing along with making it an integrated part of the development methodology. In a DevOps environment, testing early, fast and often becomes the enabler of fast releases. This means that any failure in the development process is identified immediately and prompt corrective action can be taken by the invested stakeholders. Teams can fail fast and also recover quickly – and that is how to ensure Quality in DevOps.

“The most powerful tool we have as developers is automation.” – Scott Hanselman

In a previous blog about some stats that told the developing software testing story, we had identified the impact on testing strategies due to the growing adoption of DevOps as a key trend. Our CEO has also written in greater detail on how testing is changing due to DevOps. Looking back though it’s clear this story is still developing and that something more remains to be said. In other words, it deserves a sequel and the central role in this continuing story is reserved for Test Automation.

You may ask, why does Test Automation deserve this star billing? Well, consider the DevOps way for a bit. This is a world with several, almost continuous iterative releases, each within days, even minutes, of each other, all being pushed out to the final production environment into the demanding hands of paying customers. So many releases, so little testing time and so much pressure to deliver quality – has there ever been a more, theoretically, perfect case for automated testing? Let’s hope that puts the “Why” question to bed – now let’s move on to the “How” and “What”.

First, a look at the “How”. As was already apparent in Agile, with so many iterative releases following so close on the heels of each other, it is absolutely impossible to build your automation in parallel with the product-under-test. Thus, with DevOps, it becomes critically important to involve the test automation team at an early enough stage of the product planning to be able to anticipate, as much as possible, the direction the product is likely to take and automate early. This is also the time to plan for the automation carefully. Factors to consider include what conditions are most likely to remain reasonably unchanged and which are likely to undergo frequent changes? How reusable can you make the components of the automation framework? This is also a good time to define the objectives you are looking to achieve with the automation – Faster deployment? Better code quality? Greater confidence based on better regression tests? Essentially, start with the end in mind and measure as you go along to know if you are on the right track.

1. So, on to the “What”. There is both the opportunity for creating a comprehensive test automation framework and the threat that some of it could be rendered irrelevant by the pace of change in the product.
2. That said, though, there is value to considering automating the Unit Tests as there is a reasonable chance that several specific components will remain relatively stable over the course of the many iterations.
3. The greatest value could well be in automating the regression testing – the maximum bang for the buck considering the sheer number of releases. Many DevOps models allow code to be delivered late in the cycle and fixes can be applied right until the time that specific release goes live. Automating the regressions will allow you to test the entire code after each such addition and this makes it far more likely that a high-quality, bug-free product goes out to the end customer.
4. Among the central value propositions of the DevOps, way is continuous deployment on the operational infrastructure. Continuous deployment means continuous integration of code and of the code into the operational infrastructure. This is where automation can play a key role. An interesting approach being followed by many is to run integration testing separately and parallel from unit testing and sometimes even before unit testing. This approach believes that the integration testing is not impacted by the business logic and is only concerned with whether the product works on the deployment infrastructure. Automation helps test this quickly and relatively comprehensively.
5. There is also great value on automating the testing for the build – deployment stage starting at the test environment itself. The objective is to run the tests in the test or development environment and ensure smooth deployment in the production environment.

A quote we like about DevOps goes, “DevOps is not a goal, but a never-ending process of continual improvement”. While agreeing with Jez Humble, who said this, perhaps we would like to add that this continual improvement is driven by continual testing, which is turn is based on a solid test automation platform. What do you think?

By Rajiv Jain (CEO, ThinkSys)

Mark Berry said, “Programmers don’t burn out on hard work, they burn out on change-with-the-wind directives and not ‘shipping’”. I don’t know about the first but the desire to “ship” seems to be a powerful motivation for the push towards the adoption of DevOps practices among software development teams and companies everywhere. That and the business benefits obviously. I remember a Puppet Labs survey from a couple of years ago that showed that of the organizations surveyed, those that adopted DevOps shipped code up to 30 times more frequently with lead times of as little as a few minutes. They also experienced only half as many failures and had the ability to recover from those failures up to 12 times faster. These are not just numbers, I recall reading about one of the early DevOps adopters, Etsy. Apparently they update their site every 20 minutes without any service disruption to over 20 million users – that’s a phenomenal competitive edge to possess. Small wonder then that the DevOps tide is rising.

Across most SaaS-based product development efforts, the approach seems to be to bring Development and Operations together in, a largely, automation-driven effort to keep pushing releases out, more or less continuously. Given the sheer number, it’s perhaps not even fair to term these as releases any longer! Coming from a company that has a significant Software Testing practice though an interesting facet of this partnership between Development and Operations occasionally occurs to me. It seems to me that a third, equally interested party, Testing and QA, may not be getting the attention it deserves in this discussion. In fact, the case has sometimes been made that the greater degree of automation and the souped-up release cycles imply a reduced role for software testing. I couldn’t disagree more.

Let’s go back to the Puppet Labs survey I quoted earlier – notice that one of the key benefits stated was the reduced rate of failures and the ability to recover from those failures quickly. Clearly the aim is not to ship code, it’s to ship market-ready product on a “Continuous” basis. To me it seems that would be impossible to achieve without an even greater emphasis on testing. Testing, while Agile had brought out the need to involve testing strategy at a very early stage of the product planning. How else to keep up with the short release cycles? DevOps with its Continuous Integration and Continuous Delivery is, in this context, like Agile on steroids and the need, thus, is even greater to get testing involved early in the product definition and planning stage. In many ways, the approach now has to be to plan to prevent defects rather than detect them.

So, in DevOps testing comes in early, plans for the way the product is going to pan out and prepares itself accordingly so that as code starts rolling out it gets tested. This is a significant change, at least of mindset. Earlier the development used to get done and the testing would start – today these seem to have to go, more or less, in parallel. Many have called this Continuous Testing – a perfectly valid term.
Another significant change that this “Continuous” model is engineering is the integration of functions. DevOps teams now tend to contain people from Development, Operations and, increasingly, Testing. Without that level of integration, getting quality code out in such short time frames would be impossible. This integration is also throwing up new roles for everyone, including testing. Carl Schmidt, CTO of Unbounce explains it well, “I’m of the mindset that any change at all (software or systems configuration) can flow through one unified pipeline that ends with QA verification. In a more traditional organization, QA is often seen as being gatekeepers between environments. However, in a DevOps-infused culture, QA can now verify the environments themselves, because now infrastructure is code.”

That statement points to a third significant change for testing in the DevOps way – what to test? Now that the Operations, i.e the nuts and bolts that get the SaaS product out to the millions of subscribers, is a part of the effort of getting the product out is it not necessary to test that too? There are also some subtle changes of emphasis that emerge here – functional testing is always important but now there is a premium placed on testing for load conditions and for performance. The environment is also, as much a part of the SaaS product now as the code. There is a role here that testing can excel in – clearly they know more than most about the difficulties of taking the application code, deploying it in test environments, testing it and then moving it out to production. The process may have become shorter and sharper but the skills are the same.

I like a quote, sadly unattributed, that I read somewhere, it goes, “Software testers do not make software, they only make them better.” In the context of DevOps it would seem that finally they are getting an opportunity to not only make software but also to make it better!