Reporting test results

Reporting test results

The test plan and its attachments are test-related documents that are prepared prior to test execution. There are additional documents related to testing that are prepared during and after execution of the tests. The IEEE Standard for Software Test Documentation describes the following documents .

Test Log

The test log should be prepared by the person executing the tests. It is a diary of the events that take place during the test. It supports the concept of a test as a repeatable experiment [14]. In the experimental world of engineers and scientists detailed logs are kept when carrying out experimental work. Software engineers and testing specialists must follow this example to allow others to duplicate their work.

The test log is invaluable for use in defect repair. It gives the developer a snapshot of the events associated with a failure. The test log, in combination with the test incident report which should be generated in case of anomalous behavior, gives valuable clues to the developer whose task it is to locate the source of the problem. The combination of documents helps to prevent incorrect decisions based on incomplete or erroneous test results that often lead to repeated, but ineffective, test-patch-test cycles.

Retest that follows defect repair is also supported by the test log. In addition, the test log is valuable for (i) regression testing that takes place in the development of future releases of a software product, and (ii) circumstances where code from a reuse library is to be reused. In all these cases it is important that the exact conditions of a test run are clearly documented so that it can be repeated with accuracy.

Test Log Identifier

Each test log should have a unique identifier.

Description

In the description section the tester should identify the items being tested, their version/revision number, and their associated Test Item/Transmittal Report. The environment in which the test is conducted should be described including hardware and operating system details.

Activity and Event Entries

The tester should provide dates and names of test log authors for each event and activity. This section should also contain:

1.  Execution description: Provide a test procedure identifier and also the names and functions of personnel involved in the test.

2.  Procedure results: For each execution, record the results and the location of the output. Also report pass/fail status.

3.Environmental information: Provide any environmental conditions specific to this test.

4. Anomalous events: Any events occurring before/after an unexpected event should be recorded. If a tester is unable to start or compete a test procedure, details relating to these happenings should be recorded (e.g., a power failure or operating system crash).

5.  Incident report identifiers: Record the identifiers of incident reports generated while the test is being executed.

Test Incident Report

The tester should record in a test incident report (sometimes called a problem report) any event that occurs during the execution of the tests that is unexpected, unexplainable, and that requires a follow-up investigation. The IEEE Standard for Software Test Documentation recommends the following sections in the report:

1. Test Incident Report identifier: to uniquely identify this report.

2.  Summary: to identify the test items involved, the test procedures, test cases, and test log associated with this report.

3.                 Incident description: this should describe time and date, testers, observers, environment, inputs, expected outputs, actual outputs, anomalies, procedure step, environment, and attempts to repeat. Any other information useful for the developers who will repair the code should be included.

4.                 Impact: what impact will this incident have on the testing effort, the test plans, the test procedures, and the test cases? A severity rating should be inserted here.

Test Summary Report

This report is prepared when testing is complete. It is a summary of the results of the testing efforts. It also becomes a part of the project‘s historical database and provides a basis for lessons learned as applied to future projects. When a project postmortem is conducted, the Test Summary Report can help managers, testers, developers, and SQA staff to evaluate the effectiveness of the testing efforts. The IEEE test documentation standard describes the following sections for the Test Summary Report :

1. Test Summary Report identifier: to uniquely identify this report.

2.  Variances: these are descriptions of any variances of the test items from their original design. Deviations and reasons for the deviation from the test plan, test procedures, and test designs are

discussed.

3. Comprehensiveness assessment: the document author discusses the comprehensiveness of the test effort as compared to test objectives and test completeness criteria as described in the test plan. Any features or combination of features that were not as fully tested as was planned should be identified.

4. Summary of results: the document author summarizes the testing results. All resolved incidents and their solutions should be described. Unresolved incidents should be recorded.

5. Evaluation: in this section the author evaluates each test item based on test results. Did it pass/fail the tests? If it failed, what was the level of severity of the failure?

6. Summary of activities: all testing activities and events are summarized.

Resource consumption, actual task durations, and hardware and software tool usage should be recorded.

7. Approvals: the names of all persons who are needed to approve this document are listed with space for signatures and dates.

Figure 7.4 shows the relationships between all the test-related documents