In most software-development organizations, the testing program functions as the final "quality gate" for an application, allowing or preventing the move from the comfort of the software-engineering environment into the real world. With this role comes a large responsibility: The success of an application, and possibly of the organization, can rest on the quality of the software product. It is tempting for testing teams to focus purely on the mechanics of testing a software application and pay little attention to the surrounding tasks required of a testing program.

Effective Software Testing, by CSC author Elfriede Dustin, provides concepts that can be used by an organization to implement a successful and efficient testing program. It contains practices drawn from Dustin’s work as assistant director of integration, test and deployment at CSC. One of the most crucial phases of testing comes before programming begins, when requirements are being set. The following excerpt details how testing teams should become involved at this critical stage. The principles benefit both testers and project managers in business units.

Chapter 1: Requirements Phase

The most effective testing programs start at the beginning of a project, long before any program code has been written. The requirements documentation is verified first; then, in the later stages of the project, testing can concentrate on ensuring the quality of the application code. Expensive reworking is minimized by eliminating requirements-related defects early in the project’s life, prior to detailed design or coding work.

The requirements specifications for a software application or system must ultimately describe its functionality in great detail. One of the most challenging aspects of requirements development is communicating with the people who are supplying the requirements. Each requirement should be stated precisely and clearly, so it can be understood in the same way by everyone who reads it.

If there is a consistent way of documenting requirements, it is possible for the stakeholders responsible for requirements gathering to effectively participate in the requirements process. As soon as a requirement is made visible, it can be tested and clarified by asking the stakeholders detailed questions. A variety of requirement tests can be applied to ensure that each requirement is relevant, and that everyone has the same understanding of its meaning.

Item 1: Involve Testers from the Beginning

Testers need to be involved from the beginning of a project’s life cycle so they can understand exactly what they are testing and can work with other stakeholders to create testable requirements.

Defect prevention is the use of techniques and processes that can help detect and avoid errors before they propagate to later development phases. Defect prevention is most effective during the requirements phase, when the impact of a change required to fix a defect is low: The only modifications will be to requirements documentation and possibly to the testing plan, also being developed during this phase. If testers (along with other stakeholders) are involved from the beginning of the development life cycle, they can help recognize omissions, discrepancies, ambiguities, and other problems that may affect the project requirements’ testability, correctness, and other qualities.

A requirement can be considered testable if it is possible to design a procedure in which the functionality being tested can be executed, the expected output is known, and the output can be programmatically or visually verified.

Testers need a solid understanding of the product so they can devise better and more complete test plans, designs, procedures, and cases. Early test-team involvement can eliminate confusion about functional behavior later in the project life cycle. In addition, early involvement allows the test team to learn over time, which aspects of the application are the most critical to the end user and which are the highest-risk elements. This knowledge enables testers to focus on the most important parts of the application first, avoiding over-testing rarely used areas and under-testing the more important ones.

Some organizations regard testers strictly as consumers of the requirements and other software development work products, requiring them to learn the application and domain as software builds are delivered to the testers, instead of involving them during the earlier phases. This may be acceptable in smaller projects, but in complex environments it is not realistic to expect testers to find all significant defects if their first exposure to the application is after it has already been through requirements, analysis, design, and some software implementation. More than just understanding the "inputs and outputs" of the software, testers need deeper knowledge that can come only from understanding the thought process used during the specification of product functionality. Such understanding not only increases the quality and depth of the test procedures developed, but also allows testers to provide feedback regarding the requirements.

The earlier in the life cycle a defect is discovered, the cheaper it will be to fix it. The table above outlines the relative cost to correct a defect depending on the life-cycle stage in which it is discovered.

Item 2: Verify the Requirements

In his work on specifying the requirements for buildings, Christopher Alexander describes setting up a quality measure for each requirement: "The idea is for each requirement to have a quality measure that makes it possible to divide all solutions to the requirement into two classes: those for which we agree that they fit the requirement and those for which we agree that they do not fit the requirement." In other words, if a quality measure is specified for a requirement, any solution that meets this measure will be acceptable, and any solution that does not meet the measure will not be acceptable. Quality measures are used to test the new system against the requirements.

Attempting to define the quality measure for a requirement helps to rationalize fuzzy requirements. For example, everyone would agree with a statement like "the system must provide good value," but each person may have a different interpretation of "good value." In devising the scale that must be used to measure "good value," it will become necessary to identify what that term means. Sometimes requiring the stakeholders to think about a requirement in this way will lead to defining an agreed-upon quality measure. In other cases, there may be no agreement on a quality measure. One solution would be to replace one vague requirement with several unambiguous requirements, each with its own quality measure.

It is important that guidelines for requirement development and documentation be defined at the outset of the project. In all but the smallest programs, careful analysis is required to ensure that the system is developed properly. Use cases are one way to document functional requirements, and can lead to more thorough system designs and test procedures. (In most of this book, the broad term requirement will be used to denote any type of specification, whether a use case or another type of description of functional aspects of the system.)

In addition to functional requirements, it is also important to consider nonfunctional requirements, such as performance and security, early in the process: They can determine the technology choices and areas of risk. Nonfunctional requirements do not endow the system with any specific functions, but rather constrain or further define how the system will perform any given function. Functional requirements should be specified along with their associated nonfunctional requirements. (Chapter 9 discusses nonfunctional requirements.)

Following is a checklist that can be used by testers during the requirements phase to verify the quality of the requirements. Using this checklist is a first step toward trapping requirements-related defects as early as possible, so they don’t propagate to subsequent phases, where they would be more difficult and expensive to find and correct. All stakeholders responsible for requirements should verify that requirements possess the following attributes.

Testers should insist that associated nonfunctional requirements, such as performance, security, usability, compatibility, and accessibility, are described along with each functional requirement. Nonfunctional requirements are usually documented in two steps:

• A system-wide specification is created that defines the nonfunctional requirements that apply to the system. For example, "The user interface of the Web system must be compatible with Netscape Navigator 4.x or higher and Microsoft Internet Explorer 4.x or higher."

• Each requirement description should contain a section titled "Nonfunctional Requirements" documenting any specific nonfunctional needs of that particular requirement that deviate from the system-wide nonfunctional specification.

• Consistency verifies that there are no internal or external contradictions among the elements within the work products, or between work products. By asking the question, "Does the specification define every essential subject-matter term used within the specification?" we can determine whether the elements used in the requirement are clear and precise. For example, a requirements specification that uses the term "viewer" in many places, with different meanings depending on context, will cause problems during design or implementation. Without clear and consistent definitions, determining whether a requirement is correct becomes a matter of opinion.

• Testability (or verifiability) of the requirement confirms that it is possible to create a test for the requirement, and that an expected result is known and can be programmatically or visually verified. If a requirement cannot be tested or otherwise verified, this fact and its associated risks must be stated, and the requirement must be adjusted if possible so that it can be tested.

• Feasibility of a requirement ensures that it can be implemented given the budget, schedules, technology, and other resources available.

• Necessity verifies that every requirement in the specification is relevant to the system. To test for relevance or necessity, the tester checks the requirement against the stated goals for the system. Does this requirement contribute to those goals? Would excluding this requirement prevent the system from meeting those goals? Are any other requirements dependent on this requirement? Some irrelevant requirements are not really requirements, but proposed solutions.

• Prioritization allows everyone to understand the relative value to stakeholders of the requirement. Pardee suggests that a scale from 1 to 5 be used to specify the level of reward for good performance and penalty for bad performance on a requirement. If a requirement is absolutely vital to the success of the system, then it has a penalty of 5 and a reward of 5. A requirement that would be nice to have but is not really vital might have a penalty of 1 and a reward of 3. The overall value or importance stakeholders place on a requirement is the sum of its penalties and rewards-in the first case, 10, and in the second, 4. This knowledge can be used to make prioritization and trade-off decisions when the time comes to design the system. This approach needs to balance the perspective of the user (one kind of stakeholder) against the cost and technical risk associated with a proposed requirement (the perspective of the developer, another kind of stakeholder).

• Unambiguousness ensures that requirements are stated in a precise and measurable way. The following is an example of an ambiguous requirement: "The system must respond quickly to customer inquiries." "Quickly" is innately ambiguous and subjective, and therefore renders the requirement untestable. A customer might think "quickly" means within 5 seconds, while a developer may think it means within 3 minutes. Conversely, a developer might think it means within 2 seconds and over-engineer a system to meet unnecessary performance goals.

• Traceability ensures that each requirement is identified in such a way that it can be associated with all parts of the system where it is used. For any change to requirements, is it possible to identify all parts of the system where this change has an effect?

To this point, each requirement has been considered as a separately identifiable, measurable entity. It is also necessary to consider the connections among requirements-to understand the effect of one requirement on others. There must be a way of dealing with a large number of requirements and the complex connections among them. Suzanne Robertson suggests that rather than trying to tackle everything simultaneously, it is better to divide requirements into manageable groups. This could be a matter of allocating requirements to subsystems, or to sequential releases based on priority. Once that is done, the connections can be considered in two phases: first the internal connections among the requirements in each group, then the connections among the groups. If the requirements are grouped in a way that minimizes the connections between groups, the complexity of tracing connections among requirements will be minimized.

Traceability also allows collection of information about individual requirements and other parts of the system that could be affected if a requirement changes, such as designs, code, tests, help screens, and so on. When informed of requirement changes, testers can make sure that all affected areas are adjusted accordingly.

As soon as a single requirement is available for review, it is possible to start testing that requirement for the aforementioned characteristics. Trapping requirements-related defects as early as they can be identified will prevent incorrect requirements from being incorporated in the design and implementation, where they will be more difficult and expensive to find and correct.

After following these steps, the feature set of the application under development is now outlined and quantified, which allows for better organization, planning, tracking, and testing of each feature.

Related Information:

Contact Us and Let Our Experience Help You Produce Results.

Learn about CSC’s application outsourcing business.

Learn about CSC’s systems integration capabilities.