What is System Testing – A Ultimate Beginner’s Guide

What is System Testing in Software Testing?

System Testing means testing the system as a whole. All the modules/components are integrated in order to verify if the system works as expected or not.

System Testing is done after Integration Testing. This plays an important role in delivering a high-quality product.

List of Tutorials:

The process of testing an integrated hardware and software system to verify that the system meets its specified requirements.

Verification: Confirmation by examination and provisions of objective evidence that specified requirements have been fulfilled.

If an application has three modules A, B, and C, then testing done by combining the modules A & B or module B & C or module A& C is known as Integration testing. Integrating all the three modules and testing it as a complete system is termed as System testing.

My Experience

So…do you really think it will take that huge amount of time to test, what you call System Testing, even after spending a lot of effort on Integration Testing?

The client we recently approached for the project was not convinced about the estimation we provided for each testing effort.

I had to chime in with an example:

Mike, I would like to elaborate on our efforts and the importance of system testing with an example.

Shoot, he replied.

System Testing Example

A car manufacturer does not produce the car as a whole car. Each component of the car is manufactured separately, like seats, steering, mirror, break, cable, engine, car frame, wheels etc. 

After manufacturing each item, it is tested independently whether it is working the way it is supposed to work and that is called Unit testing.

Now, when each part is assembled with another part, that assembled combination is checked if assembling has not produced any side effect to the functionality of each component and whether both components are working together as expected and that is called integration testing.

Once all the parts are assembled and the car is ready, it is not ready actually.

The whole car needs to be checked for different aspects as per the requirements defined like if car can be driven smoothly, breaks, gears, and other functionality working properly, car does not show any sign of tiredness after being driven for 2500 miles continuously, color of car is generally accepted and liked, car can be driven on any kind of roads like smooth and rough, sloppy and straight, etc and this whole effort of testing is called System Testing and it has nothing to do with integration testing.

The example worked the way it was expected and the client was convinced about the efforts required for the system test.

I narrated the example here to encourage the importance of this testing.

Approach

It is performed when Integration Testing is completed.

It is mainly a Black-box type testing. This testing evaluates the working of the system from a user point of view, with the help of a specification document. It does not require any internal knowledge of systems like the design or structure of the code.

It contains functional and non-functional areas of application/product.

Focus criteria:

It mainly focuses on the following:

1. External interfaces
2. Multiprogram and complex functionalities
3. Security
4. Recovery
5. Performance
6. Operator and user’s smooth interaction with the system
7. Installability
8. Documentation
9. Usability
10. Load/Stress

Why System Testing?

#1) It is very important to complete a full test cycle and ST is the stage where it is done.

#2) ST is performed in an environment that is similar to the production environment and hence stakeholders can get a good idea of the user’s reaction.

#3) It helps to minimize after-deployment troubleshooting and support calls.

#4) In this STLC stage Application Architecture and Business requirements, both are tested.

This testing is very important and it plays a significant role in delivering a quality product to the customer.

Let’s see the importance of this testing through the below Examples which include our day to day tasks:

• What if an online transaction fails after confirmation?
• What if an item placed in a cart of an online site does not allow to place an order?
• What if in a Gmail account creating a new label gives an error on clicking the create tab?
• What if the system crashes when a load is increased on the system?
• What if the system crashes and is not able to recover the data as desired?
• What if installing software on the system takes much more time than expected and at the end gives an error?
• What if a website response time increases much more than expected after enhancement?
• What if a website becomes too slow that the user is unable to book his/her travel ticket?

Above are just a few examples to show how System Testing would affect if not done in a proper manner.

All the above examples are just the result of either system testing not performed or not done properly. All the integrated modules should be tested in order to ensure that the product works as per the requirements.

Is This A White-box Or Black-box Testing?

System testing can be considered as a black-box test technique.

Black box Testing technique does not require internal knowledge of the code whereas the white box technique requires internal knowledge of the code.

While performing System testing functional & non-functional, security, Performance and many other testing types are covered and they are tested using a black-box technique wherein the input is provided to the system and the output is verified. System internal knowledge is not required.

Black Box Technique:

How To Perform System Test?

It is basically a part of software testing and the Test Plan should always contain specific space for this testing.

To test the system as a whole, requirements and expectations should be clear and the tester needs to understand the real-time usage of the application too.

Also, most used third-party tools, versions of OSes, flavors and architecture of OSes can affect the system’s functionality, performance, security, recoverability or installability.

Therefore, while testing the system a clear picture of how the application is going to be used and what kind of issues it can face in real-time can be helpful. In addition to that, a requirements document is as important as understanding the application.

Clear and updated requirements document can save tester from a number of misunderstandings, assumptions, and questions.

In short, a pointed and crisp requirement document with the latest updates along with an understanding of real-time application usage can make ST more fruitful.

This testing is done in a planned and systematic manner.

Given below are the various steps involved while performing this testing:

• The very first step is to create a Test Plan.
• Create System Test Cases and test scripts.
• Prepare the test data required for this testing.
• Execute the system test cases and script.
• Report the bugs. Re-testing the bugs once fixed.
• Regression testing to verify the impact of the change in the code.
• Repetition of the testing cycle until the system is ready to be deployed.
• Sign off from the testing team.

What To Test?

The points stated below are covered in this testing:

• End to End testing which includes verifying the interaction between all the components and along with the external peripherals to ensure if the system works fine in any of the scenarios is covered in this testing.
• It verifies that the input provided to the system provides the expected result.
• It verifies if all the functional & non–functional requirements are tested and if they work as expected or not.
• Ad-hoc and exploratory testing can be performed in this testing after scripted testing has been completed. Exploratory testing and ad-hoc testing helps to unfold the bugs which cannot be found in scripted testing as it gives freedom to the testers to test as their desire is based on their experience and intuition.

Advantages

There are several advantages:

• This testing includes end to end scenarios to test the system.
• This testing is done in the same environment as of the Production environment which helps to understand the user perspective and prevents the issues which can occur when the system goes live.
• If this testing is done in a systematic and proper manner, then it would help in mitigating the post-production issues.
• This testing tests both the application architecture and business requirements.

Entry/Exit Criteria

Let’s take a detailed look at the Entry/Exit criteria for System Test.

Entry Criteria:

• The system should have passed the exit criteria of Integration testing i.e. all the test cases should have been executed and there should be no critical or Priority P1, a P2 bug in an open state.
• Test Plan for this testing should be approved & signed off.
• Test cases/scenarios should be ready to be executed.
• Test scripts should be ready to be executed.
• All the non–functional requirements should be available and test cases for the same should have been created.
• The testing environment should be ready.

Exit Criteria:

• All the test cases should be executed.
• No critical or Priority or security-related bugs should be in an open state.
• If any medium or low priority bugs are in an open state, then it should be implemented with the acceptance of the customer.
• Exit Report should be submitted.

System Test Plan

Test Plan is a document that is used to describe the purpose, objective, and scope of a product to be developed. What has to be tested and what should not be tested, testing strategies, tools to be used, environment required and every other detail is documented to proceed further with the testing.

The Test Plan helps to proceed with testing in a very systematic and strategic manner and that helps to avoid any risks or issues while testing is done.

System Test Plan covers the following points:

• Purpose & Objective is defined for this test.
• Scope (Features to be tested, Features not to be tested are listed).
• Test Acceptance Criteria (Criteria on which the system will be accepted i.e. mentioned points in acceptance criteria should be in the pass state).
• Entry/Exit criteria (Defines the criteria when system testing should start and when it should be considered as complete).
• Test Schedule (Estimation of testing to be completed at a specific time).
• Test Strategy (Includes testing techniques).
• Resources (Number of resources required for testing, their roles, resource availability, etc).
• Test Environment (Operating System, Browser, Platform).
• Test Cases (List of test cases to be executed).
• Assumptions (If any assumptions, they should be included in the Test Plan).

Procedure To Write System Test Cases

System test cases cover all the scenarios & use cases and also it covers functional, non- functional, user interface, security-related test cases. The test cases are written in the same way as they are written for functional testing.

System test cases include the below fields in the template:

• Test Case ID
• Test Suite name
• Description – Describes the test case to be executed.
• Steps – Step by step procedure to describe how to perform testing.
• Test Data – Dummy data is prepared to test the application.
• Expected Result – Expected result as per the requirement document is provided in this column.
• Actual Result – Result after the execution of the test case is provided in this column.
• Pass/Fail – Comparison in actual & expected result defines the Pass/fail criteria.
• Remarks

System Test Cases

Here are some sample test scenarios for an eCommerce Site:

1. If the site launches properly with all the relevant pages, features, and logo
2. If the user can register/login to the site
3. If the user can see products available, he can add products to his cart can do payment and can get the confirmation via e-mail or SMS or call.
4. If the major functionality like searching, filtering, sorting, adding, changing, wishlist, etc work as expected
5. If the number of users (defined as in requirement document) can access the site simultaneously
6. If the site launches properly in all major browsers and their latest versions
7. If the transactions are being done on the site via a specific user are secure enough
8. If the site launches properly on all the supported platforms like Windows, Linux, Mobile, etc.
9. If the user manual/guide return policy, privacy policy and terms of using the site are available as a separate document and useful to any newbie or first time user.
10. If the content of pages is properly aligned, well managed and without spelling mistakes.
11. If session timeout is implemented and working as expected
12. If a user is satisfied after using the site or in other words user does not find it difficult to use the site.

Types Of System Testing

ST is called a superset of all types of testing as all the major types of testing are covered in it. Although a focus on types of testing may vary on the basis of product, organization processes, timeline, and requirements.

The overall it can be defined as below: 

Functionality Testing: To make sure that functionality of the product is working as per the requirements defined, within the capabilities of the system.

Recoverability Testing: To make sure how well the system recovers from various input errors and other failure situations.

Interoperability Testing: To make sure whether the system can operate well with third-party products or not.

Performance Testing: To make sure the system’s performance under the various condition, in terms of performance characteristics.

Scalability Testing: To make sure the system’s scaling abilities in various terms like user scaling, geographic scaling, and resource scaling.

Reliability Testing: To make sure the system can be operated for a longer duration without developing failures.

Regression Testing: To make sure the system’s stability as it passes through an integration of different subsystems and maintenance tasks.

Documentation Testing: To make sure that the system’s user guide and other help topics documents are correct and usable.

Security Testing: To make sure that the system does not allow unauthorized access to data and resources.

Usability Testing: To make sure that the system is easy to use, learn and operate.

More System Testing Types

#1) Graphical User Interface Testing (GUI):

GUI testing is done to verify if the GUI of a system works as expected or not. GUI is basically what is visible to a user while he uses the application. GUI testing involves testing buttons, icons, checkboxes, List box, Textbox, menus, toolbars, dialog boxes, etc.

#2) Compatibility Testing:

Compatibility testing is done to ensure that the developed product is compatible with different browsers, Hardware Platforms, Operating System and databases as per the requirement document.

#3) Exception Handling:

Exception Handling Testing is performed to verify that even if an unexpected error occurs in the product, it should show the correct error message and does not let the application stop. It handles the exception in a way that the error is shown meanwhile the product recovers and allows the system to process the incorrect transaction.

#4) Volume Testing:

Volume Testing is a type of non-functional testing wherein testing is done using a huge amount of data. For Example, the Volume of data is increased in the database to verify the system performance.

#5) Stress Testing:

Stress Testing is done by increasing the number of users (at the same time) on an application to an extent that the application breaks down. This is done to verify the point at which the application will break down.

#6) Sanity Testing:

Sanity Testing is performed when the build is released with a change in the code or functionality or if any bug has been fixed. It verifies that the changes done have not affected the code and no other issue has occurred because of that and the system works as previously.

If in case any issue occurs, then the build is not accepted for further testing.

Basically, thorough testing is not done for the build in order to save time & cost as it rejects the build for an issue found. Sanity testing is done for the change done or for the fixed issue and not for the complete system.

#7) Smoke Testing:

Smoke Testing is a testing that is performed on the build to verify if the build is further testable or not. It verifies that the build is stable to test and all the critical functionalities are working fine. Smoke testing is done for the complete system i.e. end to end testing is done.

#8) Exploratory Testing:

Exploratory Testing as the name itself suggests it is all about exploring the application. No scripted testing is performed in exploratory testing. Test cases are written along with the testing. It focuses more on execution than planning.

Tester has the freedom to test on his own using his intuition, experience, and intellect. A tester can choose any feature to test first i.e. randomly he can pick the feature to test, unlike the other techniques where the structural way is used to perform testing.

#9) Adhoc Testing:

Adhoc Testing is informal testing where no documentation or planning is done to test the application. Tester tests the application without any test cases. The aim of a tester is to break the application. The tester uses his experience, guess and intuition to find the critical issues in the application.

#10) Installation Testing:

Installation Testing is to verify if the software gets installed without any issues.

This is the most important part of testing as the installation of the software is the very first interaction between the user and the product. The type of installation testing depends on various factors like operating system, Platform, distribution of software, etc.

Test cases which can be included if an installation is done via internet:

• Bad network speed and broken connection.
• Firewall and security-related.
• Size and approximate time are taken.
• Concurrent installation/downloads.
• Insufficient memory
• Insufficient Space
• Aborted installation

#11) Maintenance Testing:

Once the product goes live, the issue can occur in a live environment or some enhancement might be required in the product.

The product needs maintenance once it goes live and that is taken care by the maintenance team. The testing done for any issues or enhancement or migration to the hardware falls under maintenance testing.

What Is System Integration Testing?

It is a type of testing in which the system’s ability to maintain data integrity and operation in coordination with other systems in the same environment, is being checked.

Example of System Integration Testing:

Let’s take the example of a well-known online ticket booking site – http://irctc.co.in.

This is a ticket booking facility; an online shopping facility interacts with PayPal. Overall you can consider it as A*B*C=R.

Now on the system level, online ticket booking facility, online shopping facility, and online payment option facility can be system tested independently, followed by check perform Integration tests for each of them. And then the whole system needs to be systematically tested.

So where does System Integration testing come into the picture?

The web portal http://Irctc.co.in is a combination of systems. You may perform tests at the same level (single system, the system of systems), but at each level, you may want to focus on different risks (integration problems, independent functionality).

• While testing the Online Ticket booking facility, you may verify if you are able to book tickets online. You may also consider integration problems

  For Example,

  Ticket booking facility integrates back-end with front-end (UI).

  For Example,

   how front-end behaves when the database server is slow to respond?

• Testing of Online ticket booking facility with online shopping facility. You may verify that the online shopping facility is available for the users logged into the system to book tickets online. You may also consider verification of integration in the online shopping facility.

  For Example,

   if the user is able to select and buy a product without a hassle.

• Testing of online ticket booking facility’s integration with PayPal. You may verify whether, after booking tickets, money was transferred from your PayPal account to the Online Ticket Booking account. You may also consider the verification of integration in PayPal.

  For Example,

  what if the system puts two entries in a database after debiting money for once only?

Difference between System Testing and System Integration Testing:

The main difference is:

• System Testing looks after a single system’s integrity with relevant environment
• System Integration Testing looks after multiple systems’ integrity with each other, being in the same environment.

Thus, the system test is the beginning of real testing where you test a product as a whole and not a module/feature.

Difference Between System And Acceptance Testing

Given below are the major differences:

System TestingAcceptance Testing

1 System testing is the testing of a system as a whole. End to end testing is performed to verify that all the scenarios are working as expected.Acceptance testing is done to verify if the product meets customer requirement.

2System testing includes functional & non- functional testing and is performed by the testers.Acceptance testing is functional testing and is performed by testers as well as a customer.

3Testing is performed using test data created by the testers.Real/Production data is used while performing acceptance testing.

4A system as a whole is tested to check the functionality & Performance of the product.Acceptance testing is done to verify that business requirement i.e. it solves the purpose what customer is looking for.

5Defects found in the testing can be fixed.Any defects found while acceptance testing is considered as a failure of the Product.

6System and system integration testing are types for System testing.Alpha and Beta testing come under acceptance testing.

Tips To Perform The System Test

1. Replicate real-time scenarios rather than doing ideal testing as the system is going to be used by an end-user and not by the trained tester.
2. Verify the system’s response in various terms as the human does not like to wait or to see the wrong data.
3. Install and configure the system as per the documentation because that is what the end-user is going to do.
4. Involving people from different areas like business analysts, developers, testers, customers can send in a better system.
5. Regular testing is the only way to make sure that the littlest change in the code to fix the bug has not inserted another critical bug into the system.

Conclusion

System testing is very important and if not done properly critical issues can be faced in the live environment.

A system as a whole has different characteristics to be verified. A simple example would be any website. If it’s not tested as a whole then the user might find that site to be very slow or the site might get crashed once a large number of users log in at the same time.

And these characteristics cannot be tested until the website is tested as a whole.

Hope this tutorial was very useful for understanding the concept of System Testing.