Reporting test results
plan and its attachments are test-related documents that are prepared prior to test execution. There are
additional documents related to testing that are prepared during and after
execution of the tests. The IEEE Standard
for Software Test Documentation describes the following documents .
log should be prepared by the person executing the tests. It is a diary of the
events that take place during the test. It supports the concept of a test as a
repeatable experiment . In the experimental world of engineers and
scientists detailed logs are kept when carrying out experimental work. Software
engineers and testing specialists must follow this example to allow others to
duplicate their work.
log is invaluable for use in defect repair. It gives the developer a snapshot
of the events associated with a failure. The test log, in combination with the
test incident report which should be generated in case of anomalous behavior,
gives valuable clues to the developer whose task it is to locate the source of
the problem. The combination of documents helps to prevent incorrect decisions
based on incomplete or erroneous test results that often lead to repeated, but
ineffective, test-patch-test cycles.
that follows defect repair is also supported by the test log. In addition, the
test log is valuable for (i) regression testing that takes place in the
development of future releases of a software product, and (ii) circumstances
where code from a reuse library is to be reused. In all these cases it is
important that the exact conditions of a test run are clearly documented so
that it can be repeated with accuracy.
Test Log Identifier
log should have a unique identifier.
description section the tester should identify the items being tested, their
version/revision number, and their associated Test Item/Transmittal Report. The
environment in which the test is conducted should be described including
hardware and operating system details.
Activity and Event Entries
tester should provide dates and names of test log authors for each event and
activity. This section should also contain:
1. Execution description: Provide a
test procedure identifier and also the names and functions of personnel involved in the test.
2. Procedure results: For each
execution, record the results and the location of the output. Also report pass/fail status.
information: Provide any environmental conditions specific to
4. Anomalous events: Any
events occurring before/after an unexpected event should be recorded. If a tester is unable to start or compete
a test procedure, details relating to these happenings should be recorded
(e.g., a power failure or operating system crash).
5. Incident report identifiers: Record
the identifiers of incident reports generated while the test is being executed.
Test Incident Report
tester should record in a test incident report (sometimes called a problem
report) any event that occurs during the execution of the tests that is
unexpected, unexplainable, and that requires a follow-up investigation. The IEEE Standard for Software Test
Documentation recommends the following sections in the report:
1. Test Incident Report identifier: to
uniquely identify this report.
2. Summary: to identify the test items
involved, the test procedures, test cases, and test log associated with this report.
description: this should describe time and date, testers,
observers, environment, inputs,
expected outputs, actual outputs, anomalies, procedure step, environment, and
attempts to repeat. Any other information useful for the developers who will
repair the code should be included.
impact will this incident have on the testing effort, the test plans, the test procedures, and the test cases? A
severity rating should be inserted here.
Test Summary Report
report is prepared when testing is complete. It is a summary of the results of
the testing efforts. It also becomes a part of the project‘s historical
database and provides a basis for lessons learned as applied to future
projects. When a project postmortem is conducted, the Test Summary Report can
help managers, testers, developers, and SQA staff to evaluate the effectiveness
of the testing efforts. The IEEE test documentation standard describes the
following sections for the Test Summary Report :
1. Test Summary Report identifier: to
uniquely identify this report.
2. Variances: these are descriptions of any
variances of the test items from their original design. Deviations and reasons for the deviation from the test plan, test
procedures, and test designs are
document author discusses the comprehensiveness of the test effort as compared to test objectives and test completeness
criteria as described in the test plan. Any features or combination of features
that were not as fully tested as was planned should be identified.
of results: the
document author summarizes the testing results. All resolved incidents and their solutions should be described.
Unresolved incidents should be recorded.
5. Evaluation: in this section the author
evaluates each test item based on test results. Did it pass/fail the tests? If it failed, what was the level of
severity of the failure?
of activities: all
testing activities and events are summarized.
consumption, actual task durations, and hardware and software tool usage should
7. Approvals: the names of all persons who are
needed to approve this document are listed with space for signatures and dates.
7.4 shows the relationships between all the test-related documents