Software testing principles
Principles
play an important role in all engineering disciplines and are usually
introduced as part of an educational background in each branch of engineering.
Figure 1.1 shows the role of basic principles in various engineering
disciplines. Testing principles are important to test specialists/ engineers
because they provide the foundation for developing testing knowledge and
acquiring testing skills. They also provide guidance for defining testing
activities as performed in the practice of a test specialist.A principle can be
defined as:
•
a general or fundamental, law, doctrine, or
assumption;
•
a rule or code of conduct;
•
the laws or facts of nature underlying the working
of an artificial device.
Extending
these three definitions to the software engineering domain we can say that
software engineering principles refer to laws, rules, or doctrines that relate
to software systems, how to build them, and how they behave. In the software
domain, principles may also refer to rules or codes of conduct relating to
professionals who design, develop, test, and maintain software systems. Testing
as a component of the software engineering discipline also has a specific set
of principles that serve as guidelines for the tester. They guide testers in
defining how to test software systems, and provide rules of conduct for testers
as professionals. Glenford Myers has outlined such a set of execution-based testing principles in
his pioneering book, The Art of Software
Testing [9]. Some of these principles are described below. Principles 1-8,
and 11 are derived directly from Myers‘ original set. The author has reworded
these principles, and also has made modifications to the original set to
reflect the evolution of testing from an art, to a quality-related process
within the context of an engineering discipline. Note that the principles as
stated below only relate to execution-based testing. Principles relating to
reviews, proof of correctness, and certification as testing activities are not
covered.
Principle 1. Testing is the process of exercising a
software component using a selected set of test cases, with the intent of (i) revealing
defects, and (ii) evaluating quality.
Software
engineers have made great progress in developing methods to prevent and
eliminate defects. However, defects do occur, and they have a negative impact
on software quality. Testers need to detect these defects before the software
becomes operational. This principle supports testing as an execution-based
activity to detect defects. It also supports the separation of testing from
debugging since the intent of the latter is to locate defects and repair the
software. The term software component is used in this context to represent any
unit of software ranging in size and complexity from an individual procedure or
method, to an entire software system. The term defects as used in this and in
subsequent principles represents any deviations in the software that have a
negative impact on its functionality, performance, reliability, security,
and/or any other of its specified quality attributes.
Principle 2. When the test objective is to detect
defects, then a good test case is one that has a high probability of revealing
a yet undetected defect(s).
Principle
2 supports careful test design and provides a criterion with which to evaluate
test case design and the effectiveness of the testing effort when the objective
is to detect defects. It requires the tester to consider the goal for each test
case, that is, which specific type of defect is to be detected by the test
case. In this way the tester approaches testing in the same way a scientist
approaches an experiment. In the case of the scientist there is a hypothesis
involved that he/she wants to prove or disprove by means of the experiment. In
the case of the tester, the hypothesis is related to the suspected occurrence
of specific types of defects. The goal for the recording and analyzing results.
A tester can justify the expenditure of the resources by careful test design so
that principle 2 is supported
Principle 3. Test results should be inspected
meticulously.
Testers
need to carefully inspect and interpret test results. Several erroneous and
costly scenarios may occur if care is not taken. For example: A failure may be
overlooked, and the test may be granted a ―pass‖ status
when in reality the software has failed the test. Testing may continue based on
erroneous test results. The defect may be revealed at some later stage of
testing, but in that case it may be more costly and difficult to locate and
repair.
A failure
may be suspected when in reality none exists. In this case the test may be
granted a fail status. Much time and effort may be spent on trying to find the
defect that does not exist. A careful reexamination of the test results could
finally indicate that no failure has occurred.
The
outcome of a quality test may be misunderstood, resulting in unnecessary
rework,or oversight of a critical problem.
Principle 4. A test case must contain the expected
output or result.
It is
often obvious to the novice tester that test inputs must be part of a test
case. However, the test case is of no value unless there is an explicit
statement of the expected outputs or results, for example, a specific variable
value must be observed or a certain panel button that must light up. Expected
outputs allow the tester to determine (i) whether a defect has been revealed,
and (ii) pass/fail status for the test. It is very important to have a correct
statement of the output so that needless time is not spent due to
misconceptions about the outcome of a test. The specification of test inputs
and outputs should be part of test design activities. In the case of testing
for quality evaluation, it is useful for quality goals to be expressed in
quantitative terms in the requirements document if possible, so that testers
are able to compare actual software attributes as determined by the tests with
what was specified.
Principle 5. Test cases should be developed for
both valid and invalid input conditions.
A tester
must not assume that the software under test will always be provided with valid
inputs. Inputs may be incorrect for several reasons. For example, software
users may have misunderstandings, or lack information about the nature of the
inputs. They often make typographical errors even when complete/correct
information is available. Devices may also provide invalid inputs due to erroneous
conditions and malfunctions. Use of test cases that are based on invalid inputs
is very useful for revealing defects since they may exercise the code in
unexpected ways and identify unexpected software behavior. Invalid inputs also
help developers and testers evaluate the robustness of the software, that is,
its ability to recover when unexpected events occur (in this case an erroneous
input). Principle 5 supports the need for the independent test group called for
in Principle 7 for the following reason. The developer of a software component
may be biased in the selection of test inputs for the component and specify
only valid inputs in the test cases to demonstrate that the software works
correctly. An independent tester is more apt to select invalid inputs as well.
Principle 6. The probability of the existence of
additional defects in a software component is proportional to the number of
defects already detected in that component.
What this
principle says is that the higher the number of defects already detected in a
component, the more likely it is to have additional defects when it undergoes
further testing. For example, if there are two components A and B, and testers
have found 20 defects in A and 3 defects in B, then the probability of the existence
of additional defects in A is higher than B . This empirical observation may be
due to several causes. Defects often occur in clusters and often in code that
has a high degree of complexity and is poorly designed. In the case of such
components developers and testers need to decide whether to disregard the
current version of the component and work on a redesign, or plan to expend
additional testing resources on this component to insure it meets its
requirements. This issue is especially important for components that implement
mission or safety critical functions.
Principle 7. Testing should be carried out by a
group that is independent of the development group.
This
principle holds true for psychological as well as practical reasons. It is
difficult for a developer to admit or conceive that software he/she has created
and developed can be faulty. Testers must realize that (i) developers have a
great deal of pride in their work, and (ii) on a practical level it may be
difficult for them to conceptualize where defects could be found. Even when
tests fail, developers often have difficulty in locating the defects since
their mental model of the code may overshadow their view of code as it exists
in actuality. They may also have misconceptions or misunderstandings concerning
the requirements and specifications relating to the software. The requirement
for an independent testing group can be interpreted by an organization in
several ways. The testing group could be implemented as a completely separate
functional entity in the organization. Alternatively, testers could be members
of a Software Quality Assurance Group, or even be a specialized part of the
development group, but in the latter case especially, they need the capability
to be objective. Reporting management that is separate from development can
support their objectivity and independence. As a member of any of these groups,
the principal duties and training of the testers should lie in testing rather
than in development. Finally, independence of the testing group does not call
for an adversarial relationship between developers and testers. The testers
should not play ―gotcha‖ games
with developers. The groups need to cooperate so that software of the highest
quality is released to the customer.
Principle 8. Tests must be repeatable and reusable.
Principle
2 calls for a tester to view his/her work as similar to that of an experimental
scientist. Principle 8 calls for experiments in the testing domain to require
recording of the exact conditions of the test, any special events that
occurred, equipment used, and a careful accounting of the results. This
information is invaluable to the developers when the code is returned for
debugging so that they can duplicate test conditions. It is also useful for
tests that need to be repeated after defect repair. The repetition and reuse of
tests is also necessary during regression test (the retesting of software that
has been modified) in the case of a new release of the software. Scientists
expect experiments to be repeatable by others, and testers should expect the
same!
Principle 9. Testing should be planned.
Test
plans should be developed for each level of testing, and objectives for each
level should be described in the associated plan. The objectives should be
stated as quantitatively as possible. Plans, with their precisely specified
objectives, are necessary to ensure that adequate time and resources are
allocated for testing tasks, and that testing can be monitored and managed.
Test planning activities should be carried out throughout the software life cycle
(Principle 10). Test planning must be coordinated with project planning. The
test manager and project manager must work together to coordinate activities.
Testers cannot plan to test a component on a given date unless the developers
have it available on that date. Test risks must be evaluated. For example, how
probable are delays in delivery of software components, which components are
likely to be complex and difficult to test, do the testers need extra training
with new tools? A test plan template must be available to the test manager to
guide development of the plan according to organizational policies and
standards. Careful test planning avoids wasteful ―throwaway‖ tests and unproductive and
unplanned ―test- patch-retest‖ cycles
that often lead to poor-quality software and the inability to deliver software
on time and within budget.
Principle 10. Testing activities should be
integrated into the software life cycle.
It is no
longer feasible to postpone testing activities until after the code has been written.
Test planning activities as supported by Principle 10, should be integrated
into the software life cycle starting as early as in the requirements analysis
phase, and continue on throughout the software life cycle in parallel with
development activities. In addition to test planning, some other types of
testing activities such as usability testing can also be carried out early in
the life cycle by using prototypes. These activities can continue on until the
software is delivered to the users. Organizations can use process models like
the V-model or any others that support the integration of test activities into
the software life cycle [11].
Principle 11. Testing is a creative and challenging
task [12].
Difficulties
and challenges for the tester include the following:
• A tester
needs to have comprehensive knowledge of the software engineering discipline.
•
A tester needs to have knowledge from both
experience and education as to how software is specified, designed, and
developed.
• A tester
needs to be able to manage many details.
• A tester
needs to have knowledge of fault types and where faults of a certain type might
occur in code constructs.
•
A tester needs to reason like a scientist and
propose hypotheses that relate to presence of specific types of defects.
• A tester
needs to have a good grasp of the problem domain of the software that he/she is
testing. Familiarly with a domain may come from educational, training, and
work-related experiences.
• A tester
needs to create and document test cases. To design the test cases the tester
must select inputs often from a very wide domain.
Related Topics
Privacy Policy, Terms and Conditions, DMCA Policy and Compliant
Copyright © 2018-2024 BrainKart.com; All Rights Reserved. Developed by Therithal info, Chennai.