11. Testing Levels : System Testing
Objectives of System Testing Now that we know that audience are working together, the next step is to consider the behavior and capabilities of the whole system from an end to end perspective. This activity is called system testing. The objectives of system testing include reducing risk verifying whether the functional and nonfunctional behaviors of the system are as designed and specified validating that the system is complete and will work as expected building confidence in the quality of the system as a whole, finding defects, preventing defects from escaping to higher test levels of production. For certain systems, verifying data quality may be an objective in system testing. The test environment, which is the hardware and software used as an environment for testing, should correspond to the final target or production environment as much as possible in order to minimize the risk of environment specific failures not being found in testing.
System testing is concerned with testing all the scenarios that the system might go through. Thinking of all the possible scenarios is tricky and needs a good understanding of the system domain and the potential users. Most often, it’s carried out by specialist testers that form a dedicated and sometimes independent test team within development reporting to the development manager or project manager. As with component testing and integration testing, in some cases automated system migration tests provide confidence that changes have not broken existing features or end to end capabilities. System testing often produces information that is used by stakeholders to make release decisions. System testing may also satisfy legal or regularity requirements or standards. Examples of vocabulary that can be used as a test basis for system testing include system and software requirement specification functional
and nonfunctional risk analysis reports use cases EBIT and user stories models of system behaviors, state diagrams, system and user manuals. EBICS are a bigger user story a group of user stories. Test Objects typical test objects for system testing include applications, hardware, software, systems, operating systems, system under, test sut system configuration and configuration data. Typical Defects and Failures examples of typical defects and failures for system testing include incorrect calculations. Incorrect or unexpected system functional or nonfunctional behavior incorrect control and or data flows within the system failure to properly and completely carry out end to end functional tasks failure of the system to work properly in the production environments failure of the system to work as described in System and User Manuals.
Finally, specific approaches and responsibilities related to system testing. System testing should focus on the overall end to end behavior of the system as a whole. Both functional and non functional system testing should use the most appropriate techniques for the aspects of the system to be tested. We will learn about various techniques in the Test Design Techniques section. Independent testers typically carry out system testing defects in specifications, for example, missing user stories, incorrectly stated business requirements, and so on can lead to a lack of misunderstanding or disagreement about expected system behavior such situations can cause false positives and false negatives which waste time and reduce defect detection effectiveness respectively. Early involvement of testers in user story refinement or a static testing activities such as reviews helps to reduce the incidents of such situations.
12. Testing Levels : Acceptance Testing
Acceptance testing is simply a form of yes or no testing. Should we accept the software? Yes or no so acceptance testing should answer the question can the system be released and deployed or not? Objectives of Acceptance Testing Acceptance testing, like system testing, typically focuses on the behavior and capabilities of of a whole system or border. Objectives of acceptance testing include establishing confidence in the quality of the system as a whole validating that the system is complete and will work as expected verifying that functional and nonfunctional behaviors of the system are as specified and designed. So remember that acceptance testing main objective is acceptance.
Yes or no defects may be found during acceptance testing, but finding defects is often not an objective for acceptance testing, whereas the component integration and system testing main objective is to find defects. Therefore, we say that test levels have different objectives. Acceptance testing may produce information to assess the system’s readiness for deployment and use by the customer or end user. Finding a significant number of defects during acceptance testing may in some cases be considered a major bullshit risk. Acceptance testing may also satisfy legal or regularity requirements or standards. Examples of worker products that can be used as a test basis for any form of acceptance testing include business processes, user or business requirements regulations, legal contracts and standards. Use cases, system requirements, system or user documentation installation procedures risk analysis reports.
Typical test objects for any form of acceptance testing include system under, test system configuration and configuration data, business processes for a fully integrated system, recovery systems and hot sites for business continuity and disaster recovery testing. Operational and maintenance processes, forms reboots existing and converted production data. Typical defects and failures found in any form of acceptance testing include system workflows. Do not meet business or user requirements. Business rules are not implemented correctly. System doesn’t satisfy contractual or regularity requirements, non functional failures such as security, inadequate performance efficiency under high loads or improper operation on a supported platform.
Specific approaches and Responsibilities acceptance testing is often the responsibility of the customers. Business users, product owners or operators of a system and other stakeholders may be involved as well. Acceptance testing is often thought of as the last test level in a sequential development life cycle. But remember that acceptance testing is a yes or no kind of testing. So anywhere where we test something to get a yes or no answer, then it’s a form of acceptance testing which may also occur at other times. For example, acceptance testing of COTS or commercial of the shelf software product may occur when it’s installed or integrated. Acceptance testing of a new functional enhancement may occur before system testing. Common forms of acceptance testing include the following user acceptance testing operational acceptance testing contractual and regularity acceptance testing alpha and beta testing let’s explain each one of them in detail first.
User acceptance testing or UAT the acceptance testing of the system by users is typically focused on validating the fitness for use of the system by intended users in a real or simulated operational environment. The main objective is building confidence that the users can use the system to meet their needs, fulfill requirements, and perform business processes with minimum difficulty, cost and risk. Users can perform any test they wish, usually based on their business processes. It can happen at any time during the project. So yes, you can demo the software to the customer in the middle of the project and actually they can decide not to continue if they want.
And of course, UAT usually happens before the final user sign off. Operational Acceptance Testing or Oat Operational acceptance testing tests the system for acceptance from the system administrator’s point of view. It’s usually done by operations or system administration staff and usually performed in a simulated production environment. The tests focus on operational aspects and may include testing of backup and restore, installing, uninstalling and upgrading disaster recovery, user management, maintenance tasks, data load and migration tasks. It checks for security vulnerabilities I hate this word.
I cannot pronounce it, so move along. Performance Testing the main objective of operational acceptance testing is building confidence that the operators or system administrators can keep the system working properly for the users in the operational environment, even under exceptional or difficult conditions. Besides the test basis we have mentioned before testing for driving test cases for operational acceptance testing, one or more of the following worker products can be used backup and restored procedures, disaster recovery procedures, non functional requirements, operations, documentation, deployment and installation instructions, performance targets, database packages, and security standards or regulations. The third type of acceptance testing is contract and regulation acceptance testing. First, let’s talk about contract acceptance testing. Sometimes the criteria for accepting a system are decremented in a contract.
If a customer needs to buy a C OTS software but will add a minor requirement, say to add his company name and logo to the first display screen, then I guess you will agree with me that we don’t need to change the requirements document for this and go through the software development lifecycle again. Such a minor change could be documented in the contract and that’s it. Testing then is conducted to check that these criteria have been met before the system is accepted. Contractual acceptance testing is often performed by users or by independent testers. Regulation acceptance Testing in some industries, systems must meet governmental, legal or safety standards. Examples of these are the defense, banking and pharmaceutical industries.
For example, if a software company wants to add the ISO logo to its software, then it should follow ISO guidelines or regulations in developing the software. Testing then is conducted to test whether we should pass the ISO guidelines or not. Yes or no? Regularity acceptance testing is often performed by users or by independent testers, sometimes with the result being witnessed or audited by regularity agencies. The main objective of contractual and regularity acceptance testing is building confidence that contractual or regularity compliance has been achieved. Finally, alpha and beta Testing some systems are developed for the mass market. For example, commercial off the shelf software or COTS, where there is no actual customer who provided you with his requirements. But instead the marketing team of your company suggested some requirements for you and you built the system based on those suggestions.
Still, you want to get feedback of how the user who will buy the system feels when using the system before the software product is put on the market. Very often, this type of system undergoes two stages of acceptance testing alpha testing, which takes place at the developers site. We invite some potential users to our site using our machines and let them play with the software for a while and get feedback from them. The second type is beta testing, which takes place at the customers site. We send the software to some potential users and ask them to play with the software using their machines and ask them to send us feedback when they are done. Beta testing may come after alpha testing or may occur without any preceding alpha testing.
So remember, alpha is done as a development site and beta is done at the customer site. By the way, when you download any beta version of a software from the net, that means you are currently a beta tester. One objective of alpha and beta testing is building confidence among potential or existing customers and or operators that they can use the system under normal everyday conditions and in the operational environments to achieve their objectives with minimum difficulty, cost and risk.
Another objective may be the detection of defects related to the conditions and environments in which the system will be used, especially those conditions and environments that are difficult to replicate by the development team. So I want you to give it a thought which is better from your point of view alpha or beta testing? Actually, the answer is both. Each one of them has its Cones and Bose, so both are good. You cannot say one is better than the other.
In iterative development, project teams can employ various forms of acceptance testing during and at the end of each iteration, such as those focused on verifying a new feature against its acceptance criteria, and those focused on validating that a new feature satisfies the user needs. In addition, alpha tests and beta tests may occur either at the end of each iteration, after the completion of each iteration, or after a series of iterations. Also, user acceptance tests, operational acceptance tests, regularity acceptance tests and contractual acceptance test also may occur either at the close of each iteration, after the completion of each iteration, or after a series of.
