Software and System Test Documentation

The IEEE 829-2008 standard for “Software and System Test Documentation” has been revised. The prior version described the format and content of numerous items of test documentation. The updated standard removes some items of test documentation and modifies the format and content of the remaining items.

Test processes determine whether the development products of a given activity conform to the requirements of that activity, and whether the system and/or software satisfy the intended use and user needs.

Testing process tasks are specified in 829-2008 for different “integrity levels”. These process tasks determine the appropriate breadth and depth of test documentation. The documentation elements for each type of test documentation can then be selected.

The scope of testing addressed by the standard includes software-based systems, computer software, hardware, and their interfaces. This standard applies to software-based systems being developed, maintained, or reused (legacy, commercial off-the-shelf, and non-developmental items).

The term “software” also includes firmware, microcode, and documentation. Test processes can include inspection, analysis, demonstration, verification, and validation of software and software-based system products.

The key changes in the 829-2008 standard are described below:

New Direction
The revised standard introduces the concept that the test effort has tasks to accomplish during the entire development life cycle, not merely during the test activity. It moves from a document focus to a process focus.

New Test-Related Documentation
The 829 standard adds a Master Test Plan for the management of a large and/or complex test effort. It adds a Master Test Report to summarize the results of tasks identified in the Master Test Plan. The Master Test Report may also be used to consolidate the results for multiple Level Test Reports. It adds a Level Interim Test Status Report for use during the test execution activity.

The standard recognizes that some projects may want some stand-alone and some combined documents, and allows for any combination of plan, design, test cases, and test procedures within test levels. It adds a process for choosing the appropriate documentation and contents.

The standard moves away from requiring identical documentation, providing instead, documentation based on the integrity level of the project. It identifies the minimum recommended tasks for the identified integrity level.

New Processes
The standard introduces the concept of integrity levels. It provides a mechanism by which projects can identify their integrity level. The higher the integrity level, the more test tasks that are recommended. 829-2008 also introduces the concept of test management. It describes tasks that are exclusive to those who manage a test effort.

The following key concepts are emphasized in this new 829 Standard:

Integrity Levels
The standard defines four integrity levels to describe the importance of the software or system aspects to the user. The process of identifying the integrity level is the criticality analysis. Each project or organization identifies the aspect of the system or software that is most important.

Minimum Testing
The standard defines the recommended minimum testing tasks required for each of the four integrity levels. It includes a table of optional testing tasks for tailoring the test effort to meet project needs and application specific characteristics. A low integrity level project such as an internal bug-tracking program requires fewer test tasks than would a high integrity level project like developing software/firmware for medical devices.

Testing Intensity
The standard introduces the notion that the integrity and rigor applied to testing tasks vary according to the integrity level. Higher integrity levels require the application of greater intensity and rigor. A high integrity level project such as developing medical devices may execute a myriad of tests for each unit, as well as, for integration and system/acceptance tests. These tests will likely go to the depth of each test level looking for every conceivable deficiency. A low integrity level project may only do acceptance testing against the primary functionalities rather than system testing against the requirements.

Testing Criteria
The standard defines specific criteria for each testing task including minimum recommended criteria for correctness, consistency, completeness, accuracy, readability, and testability.

Systems Viewpoint 
The standard includes recommended minimum testing tasks to respond to system needs. It recognizes that software does not exist in isolation, and that much of current software development may actually be for software intensive systems or for embedded firmware. Therefore, the entire system needs to be taken into account when identifying the system integrity level and the resultant test tasks.

Selection of Test Documentation 
The types of test documentation, and the content topics within each documentation type, need to be selected based on the testing tasks associated with the identified integrity level.

The prior standard required every project to use the same test documents and to include the same information. The new standard provides for tailoring based on the integrity level. Therefore, a high integrity level project (e.g., medical devices) will require the full range of test documentation and contents as described in the 829-2008 standard. Conversely, a low integrity level project may require only a minimum quantity of test plan information and a full range of test case and test procedure information.

The information for this article was based on an article written by Eva Freund and Claire Lohr in the ASQ 2Q2008 Software Division Newsletter. You can order the 829-2008 standard at the IEEE Standards web site.