Microsoft Azure DevOps Test Management Review – Test Management

Introduction

The Microsoft Azure DevOps Test Management component offers the possibility of a fully integrated development and test capability for QA and Development teams. It gives agile teams the capability of collaborating in one tool for all the different disciplines.

Again, in this review of the test management component, we’ll be focusing on 7 of the most prominent features and assessing Azure DevOps against them. These features are:

1. Assignment
2. Version Control
3. Parameterisation
4. Libraries
5. Results Aggregation
6. Retesting
7. Configurations and Releases

Assignment

When implementing a process or selecting a test management tool, you’ll always need to assign a test to a tester at some point. Question is do you need to assign at the step, case or cycle level. Do you need to allocate different tests within a cycle to different QA engineers (or not) also needs to be considered. And then there’s how you track the difference between assignment for writing and executing the test cases, which complicates a supposedly simple concept. Different tools support this capability in different ways.

Assigning tests to testers works on 3 levels which are:

1. At the test case level
2. The plan /suites
3. Executions and runs

Assignment of test cases at various levels

Version Control

Usually, you would want to be able to identify different versions of test cases that you write and modify, and which versions were used against which versions/build of the application under test. ADO tracks the history of changes. Each time you save changes, the history is updated.

There are two scenarios to consider here:

1. If you have a test case included in a plan which you’re waiting to run
2. If the test case is already in a plan/suite and you’ve already run it.

    

In both cases ADO will manage and track the right version that you’ve executed.

Displaying the correct version details of the test cases that have been executed

Parameters

ADO gives all that is needed when it comes to parameterising tests. When defining your test cases use the @ nomenclature. With this nomenclature parameters are automatically added to the test case.

The parameter can be converted to a shared parameter set within the project easily. The shared param set is viewed in the param section of the test plans, where values or additional parameters can be added. Also, while you’re here you can see a list of test cases that reference this parameter set and edit the work item properties.

ADO takes parameter sets and generates multiple iterations of the test case in the run, each of which has its own test result value, and an overall ‘test result’ and a result for each iteration is presented. When the ‘test result / run’ is viewed, you’ll see this overall result and the individual iteration results. Regardless of the number of iterations you have, it’s always reported at ‘one’ test result.

Iterations and individual test results

Libraries

The basic concept of managing your test cases in a test library under Azure DevOps is structured around ‘work items’. Your tests live in the same repository as all other work items like stories, defects, etc. 

With this approach, you’re focused on building test cases that link to stories so that you have test cases that provide coverage of stories. Another aspect that’s often important to managing a library of tests is the ability to bulk update tests. It’s a bit of a mindset shift for testers coming from traditional tools like Microfocus ALM and Quality Center because the approach is somewhat different. It’s all possible though. 

Bulk updating multiple test cases

Result Aggregation

The ability to aggregate test results across different builds and versions, even across sprints or even test plans is critical for status reporting. You definitely need to be able to aggregate test results. With ADO, you link your Test Plan to a build and all tests executed within their suites are tracked against that build. With this approach you can report across and combine different result sets.

Aggregate test result

Retesting

When we identify failed tests, we pull them together to retest in subsequent builds. ADO makes this easy with a well thought out approach.

Retesting a failed test

Configurations and Releases

With ADO, we track our results against configurations or environments that we’re testing against. With this we know which build or version of the application was tested against which configuration.

Configuration variables allow us to set aspects of our configurations that will have various values such as the types of browsers and operating systems.

Adding configuration variables

Summary

The only area that is slightly lacking in Azure Dev Ops test management is version control of test cases. Assignment works on many levels and parameterisation of test cases is supported. The concept of managing your tests in a library and creating a test plan is a slightly different paradigm to other tools. You could think of it being more Agile.

Test result aggregation and retesting is easily managed with ADO. It takes time to get used to using ADO but it clever approach. This coupled with the integration to other CI and CD parts of the process makes this a very effective solution.