Continuous Testing from DevOps Eye

DevOps is one of the best culture being adapted by many organisations these days to effectively deliver their product to the production in short span of time via delivery pipelines.

In case if you are new to DevOps and want to get started, click here.

A team can effectively achieve DevOps Pipeline culture by religiously adopting several practices like Continuous Integration, Continuous Delivery and Continuous Deployment.

Hang on! Then what is Continuous Testing and when does it comes into picture?

Continuous Testing (CT) is a practice which focuses on the automated testing life-cycle within the CI/CD pipelines. And this process will be carried out in every stage of the CI/CD process.

Is CT different?

In a typical Waterfall model, the testing will happen as a “Stage”. Which means, the testers will be given an environment after the deployment and they have to execute their automated tests in a specific time. This tedious process of time dedication will eventually waste efforts and delay the delivery of the project. Unfortunately, many organisations still practices this culture.

The evolution of testing strategies has come up with methods that can mitigate this issue.

Testing is NO more an isolated process!

CT will be involved in the pipeline where we arrange a set of testing activities to ensure whether the build has no or less defects.

Let’s now see how we can achieve it. Like I said earlier, its a practice in which we align our tests and involves stages.

Stage 1: Once the code is committed, the build tools like Jenkins will check for any build errors.

Stage 2: When no errors found, then runs the Unit Tests.

Stage 3: Run the SAST tests to check the vulnerabilities in the code

Stage 4: Run a Smoke test pack covering some basic functionalities of the existing features

Stage 5: Run a Sanity test pack to check whether there are any breakages in the build.

Note: You can see this article for differences between Smoke and Sanity Testing

Stage 6: Have you Regression Pack run to make sure all the existing functionalities work without any issues. Adopt the Parallelism to run your tests to save more time. Also, at this stage, try to use perform Accessibility testing, Visual Testing etc., to gain the confidence that there are no issues related to the look and feel

Stage 7: Once the Testing is done, use the Continuous Deployment process to deploy the code automatically to the next environment

So, the entire process of executing the Unit tests, Security analysis, Smoke tests, Sanity tests, and Regression tests in an automated fashion during the build process is Continuous Testing.

The core understanding is to reduce the manual effort required to approve any build for next environment deployments.

Tips:

1. Set a threshold limit of number of tests cases that should pass. Only then the next stage of CT will be executed. For example, set a threshold limit of 95% of Sanity should pass to execute the Regression tests.
2. There is a high possibility of flakiness due to unreachable browsers or application not responding. Take early measures to mitigate them.
3. You can integrate the Jenkins jobs to Test management tools like JIRA to create a ticket about the testing status automatically for futuristic purpose

What’s in for me?

So, lets see some key benefits of CT –

1. No manual intervention is required during the build pipeline
2. Release cycle can be reduced from weeks to hours
3. Identifies the issues much earlier
4. Agility and Reliability
5. Improved code quality

The Catch?

All tools and methodologies has its own cons. CT too.

1. Skills for DevOps has to be enhanced by the resources
2. E2E journeys covering many integrated system may cause delays and issues
3. Flakiness, if wrong infrastructure is adopted
4. Possibility of test failures when the environment is not available

So, do you think you can adopt CT in your organisation or project? Get the maximum benefits of reduction of release cycle and time saving by adopting this successful path.

Feature Image Credits –

http://www.tricentis.com/products/what-is-continuous-testing/