Software Quality: the different types of software testing • Tuleap

A piece of software is more than hundreds of lines of code. It is rather a complex system, combining dozens of separate functional components and third-party integrations. That’s why, a reliable quality assurance policy should go far beyond just finding bugs in the source code. Usually, the testing process covers several levels of software. Some can be automated, others are more efficient done manually. Let’s dig into the differences to better understand.

Software testing levels

Software testing levels describe the stages of software development when testing is conducted. Generally, there are four progressive testing levels based on the area they focus on the software development process: unit testing, integration testing, system testing, and user acceptance testing (UAT).

Levels of software testing

Unit Testing

What is it

The smallest testable part of the software system is often referred to as a unit: a new code, a refactoring of legacy code. Unit testing is commonly performed early in the development process by the engineers themselves, not the testing team.

Why to run it

To ensure code is developer correctly. Therefore, this testing level is aimed at examining every single unit of a software system in order to make sure that it meets the original requirements and functions as expected.

Performed by

Developers or Technical architects

How to do

As soon a new code is written, automated tools with continuous integration as a part of the build, are very efficient: JUnit, Test NG, PHP Unit.

Integration Testing

What is it

The objective of the next testing level is to verify whether the combined units work well together as a group.

Why to run it

Integration testing is aimed at detecting the flaws in the interactions between the units within a module. The aim is to make sure the interconnections between the system components run as required.

Performed by

Developers or Technical architects

How to do

As soon a new component (new web services, controllers…) is added, continuous integration and automated tools help: REST client, SOAP UI.

System Testing

What is it

At this level, a complete software system is tested as a whole. System testing includes user flows, performance and security.

Why to run it

This stage serves to verify the product’s compliance with the functional and technical requirements and overall quality standards.

Performed by

System testing should be performed by a highly professional testing team (Business analyst, Product owner, Manual QA) in an environment as close to the real business use scenario as possible.

How to do

Exploratory testing and automated tools.

User Acceptance Testing (UAT) or End-to-End tests

What is it

This is where the product is validated against the end user requirements. It is the last stage of the testing process that ensure customers’ expectations are met, verifying acceptance tests on the stories and the features.

Why to run it

This final step helps the team decide if the product is ready to be shipped or not. While small issues should be detected and resolved earlier in the process, this testing level focuses on overall system quality, from content and UI to performance issues.

Performed by

The acceptance stage might be followed by an alpha and beta testing, allowing a small number of actual users (product end end-users, Product owner, developers) to try out the software before it is officially delivered.

How to do

Most of the time with automated tools as Cucumber or Selenium but also with manual tests, to be centralized with integrated environnement as Tuleap tool.

Manual vs Automated Testing

At a high level, we need to make the distinction between manual and automated tests.
Manual testing is done in human, by clicking through the software or interacting with the application and APIs. This requires one people to set up an environment and execute the tests themselves. This can be very expensive : human error can occur as the tester might make typos or forget one phase during the testing cycle.
On the opposite, automated tests are performed by a machine that executes a scrip written in advance. These tests can vary a lot in complexity, from checking a single method in a class to making sure that performing a sequence of complex actions in the UI leads to the same results. It’s much more robust and reliable than manual tests but automation can’t replace manual testing, for example to check visually interface or understand an end-user journey. In addition, keep in mind that the quality of automated testing depends on the quality of automation, in other words it depends on how well your test scripts have been written.

In DevOps approach and tools, automated testing is a key component with continuous integration and continuous development. This’s a great way to accelerate your quality assurance process as you add new features, so new checks to your application. But there’s still value in doing some manual testing with what is called exploratory testing and end-user acceptance validation.

Manual testing pros & cons

Pros of manual testing

• Get fast and accurate visual feedback,
• Help to identify anomalies related to the look and feel of the application,
• Better to catch usability issues in the application,
• While testing a small change, an automation test would require coding which could be time-consuming. While you could test manually on the fly.

Cons of manual testing

• Some tests are difficult to run manually -as performance tests-, others are almost not testable at all,
• Less reliable testing method because it is conducted by humans, so mistakes are always possible,
• The manual testing process can not more difficult to record and track,
• Not possible to reuse manual tests.

Automated testing pros & cons

Pros of automated testing

• Minimize the human effort required to efficiently run tests. Automation helps you to find more bugs compare to a human tester,
• Reduce cost of bugs because tests are performed up to 10 times faster compared to manual testing,
• Identify any new bug introduced as a result of the changes, by running a set of regression checks when the application is upgraded, Ideally the development team should fix any incidents as soon as they are appear,
• The more you automate tests, the faster is your testing process,
• Automation process can be saved and tracked for software traceability and quality assurance audits,
• Automated tests can be reused and executed several times for the same kind of operations,
• Increase teams’ productivity as it provides fast testing results.

Cons of automated testing

• Without human feedback, it’s harder to get insight into visual aspects of your user, interface : button location, form sizes, colors, font, sizes, contrast,
• To get started, initial costs can be very high. Writing reliable tests can be a hard task: too detailed scripts can become unmaintainable and fuzzy scripts can not serve as expected,
• False sense of quality : an automated check only checks what is been programmed to check.
• Basic programming knowledge is required so automated tests should not be adapted for user acceptance tests
• Test maintenance can be costly. This is usually underestimated,
• Every automation tool has their limitations which reduces the scope of automation.

Why do you need both automated and manual testing?

Automated testing does not have to down play the role of manual testing. They enhance one another and create more complete quality control for delivering a better product.

Automated testing increase most of the time testing speed and consistency. But, as mentioned before, it is only as good as the scripts you wrote. Manual testing is complementary to the automatic process as it raise mistakes from the user’s perspective or identify unexpected incidents from unscripted scenarios. Manual testing still has a critical role in the QA process. There are benefits to manual testing and automated testing. It’s important to consider both testing approaches as you design your quality assurance strategy. 

Testing in Agile Development

In Agile software development, testing typically represents an iterative process. It runs continuously alongside the development progress. It is a collaborative effort between testers, developers, product owners and even customers.

While testing levels seen above, generally refer to the whole software product, they can also be applied to a new piece of feature. In this case, every small unit of the new functionality is being verified. The QA team checks the interconnections between these units, the way the feature integrates with the rest of the system and if the new update is ready to be delivered.

That’s why, implementing a continuous evolving quality assurance testing policy is a key of success when launching a new product.

Challenges when managing manual and automatic testing

The tools to do the job : having a centralized test management environment, integrated with one or more automated tools is the best way to keep software quality in check. Most of the time, QA teams use different testing tools, disconnected to each other. The key here is verify they have strong integration for a complete testing process overview and to reach an A to Z traceability.

A centralized test management software as the dedicated module in Tuleap, can unify automated and manual testing to drive reuse and efficiency and gain visibility into application readiness. QA teams can manage both automated testing with tools as Jenkins, and manual testing together to gain consistency, coverage, and a seamless transition between them. 

Keep in mind, that automation is an efficient tool if it supports your manual testing efforts by freeing time, so that complex, exploratory and heuristic manual testing can be focused and valuable, contributing in whole to a successful QA process.

To combine the best of the two worlds, it is needed to run in parallel automated and manual test campaigns that reduces overall execution time and drives efficiency. Taking a look at Tuleap, you will see how real-time reporting and analytics provides full traceability to issues raised, associated results, and releases progress.

webinar

Compliance: how to set up a Software Quality Assurance plan for traceability from A to Z

Access the webinar replay