Chapter 1: The fundamental of Testing
Reasons of system failure:
- Pressure of deadline
- Complexity of system
- Changing technology
- Error in specifications
- Error in code
Other conditions of failure are as:
- Environmental condition
- Presence of radiation
- Magnetism
- Electronic field or pollution
Can affect the operation of the hardware and firmware and lead to system failure.
Faulty Component:
If specification document is not correct as a result lead a faulty component and faulty components will cause system failure.
An error (or mistake) leads to a defect, which can cause an observed failure.
Error Defect Failure
Incorrect software can harm:
- People (i.e. aircraft crash, life support system failure)
- Companies (i.e. incorrect billing, loss of money)
- Environment (i.e. releasing radiation and chemical into the atmosphere)
Software failure can lead:
- Loss of money
- Loss of time
- Loss of business reputation
- Injury
- Death
Keeping software under control:
More and better software testing seems a reasonable aim but that aim is not quite as simple to achieve as we might expect.
- Exhaustive testing of complex systems is not possible
- Testing and risk (greater risk implies more and better testing)
- Testing and quality (functional and non functional weakness i.e. online tax one user could view other details and a web site to handle peak load )
The vertices of the triangle are:
So key role of testing is to ensure that key functional and non functional requirements are examined prior system goes live. So detects defects and reported to development team for ratification.
Deciding when “enough is enough”
The most important aspect of achieving an acceptable result from a finite and limited amount of testing is prioritization. So priorities and completion criteria provide a basis for planning.
What is testing and debugging?
Debugging is the process that developers go through to identify the cause of bugs or defects in code and undertake corrections. Whereas testing does not include correction of defects however ensure that changes and corrections are checked for their effects on other parts of the component or system.
Static testing:
It is a term where code is not executed. It involves techniques of reviews, preventing defects, removing ambiguities and errors in specification documents.
Dynamic testing:
It tests the program under some test data, mean of test execution.
Testing process:
It is the process of not missing critical steps and that we do things in the right order. To prepare some preparatory work to design tests and test work needed to record the results and check whether the tests are complete.
Testing Principle:
- Testing shows the presence of bugs (testing does not show that software is error free)
- Exhaustive testing is impossible (for large and complex system not possible). An approach in which all possible data combinations are used. This includes implicit data combinations present in the state of software/data at the start of testing.
- Early testing. Tester does not have to wait until software is available to test. Works products are created throughout the software development life cycle. So as soon as these are ready we can test them. So requirement documents are the basis for acceptance testing. In this process to find errors and defects before they are passed to the next stage of the development process. If problems are solved at this stage in which it is introduced, this leads to what Kit calls ‘errors of migration’.
- Defect clustering. Defects are not uniform there are variety of reasons. Which are
i.e. system complexity, volatile code, the effects of change upon change, development staff experience, development staff inexperience.
- The pesticide paradox.
- Testing is context dependent.
- Absence of errors fallacy.
Fundamental of test process:
Time pressure can mean that we begin test execution before all tests have been designed.
- Test planning and control
The main activities of planning are as:
- Scope and objectives
- Test approach
- What is required for testing i.e. people, resource etc.
- Test strategy
- Scheduling of analysis and design tasks.
- Scheduling of implementation and execution and evaluation.
- Exit criteria
The main activities of controlling are as:
- Measures and analyzing results.
- Comparing expected and actual progress, test coverage and exit criteria.
- Making corrections, if things go wrong and deciding actions.
2. Test analysis and design
The main activities are as:
- Reviewing requirements, architecture, design, interfaces and others as collective test.
- Analyzing test items, specification, behaviour and structure to identify test conditions and test data required.
- Designing the tests.
- Determining test requirements are testable.
- What test environment look like and infrastructure and tools required.
3. Test implementation and execution
These activities have the following parts.
· Developing and prioritizing test cases, creating test data, writing procedures and preparing test harness and writing automated test scripts.
· Collecting test cases into test suits,
· Checking the test environment set-up is correct.
· Running test cases in the determined order using test execution tools.
· Keeping log of testing activities, including outcomes and version of software, data, tools and test ware.
· Comparing actual results with expected results
· Reported discrepancies as incident with as much information as possible,.
· Where necessary, repeating test activities when changes have been made following incident raised.
Confirmation testing: re-execution of the test that previously failed in order to confirm a fix.
Regressing testing: executing of a corrected test and execution of previously passed tests to check that defects have not been introduced.
4. Evaluating exit criteria and reporting
This process comprises the following things.
- Check whether previously a determined exit criterion has been met.
- Determining if more test is needed or specified criteria need amending.
- Writing up the result of the testing activities for the business sponsors and other stake holders.
5. Test closure activities.
- Ensuring that the documentation is in order ; what has been delivered is defined, closing incident and raising changes for future deliveries, documenting that the system has been accepted.
- Closing down and archiving the test environment, test infrastructure and test ware used.
- Passing over test ware to the maintenance team.
- Writing down the lessons learned from this project for later.
List of order of independence testing:
- Those who wrote the code
- Members of the same development team.
- Members of a different group (independent test team).
- Members of a different company (a testing consultancy/outsourcing).
Developers to test their own code have advantage because as soon as problems are discovered , they can fixed without the need for extensive logs, but also difficulties to find your own mistakes.
Way of avoiding confrontations>
- The aim to work together rather than be confrontational. Keeping focus of delivering a quality product.
- Results should be presented in a non-personal way.
- Attempt to understand how other feel.
- At the end of discussion, confirm that you have both understood and been understood.
Chapter 2: Life Cycles
Software development Models;
- Water fall model (Linear or sequential model)
- V-model
- Iterative Model (cyclical model).
The steps in this model are in sequences which are as:
- Requirement specification
- Functional specification
- Technical specification
- Program specification
- Coding
- Testing
In this model testing starting once the code has been developed. Then decision is made whether production can be released into the live environment. This is major draw back of this model.
Verification- checks that the work-product meets the requirement set out for it. An example of this would be to ensure that a website being built follows the guidelines for making websites usable by as many people as possible. Verification helps to ensure that we are building the product in the right way.
Validation- changes the focus of work product evaluation to evaluation against user needs. This means ensuring that the behaviour of the work product matches the customer needs as defined for the project. Fore example for the same website above the guidelines may have been written with people familiar with websites in mind. It may be that this website is also intended for novice users. Validation would include these users checking that they too can use the website easily. Validation helps to ensure that we are building the right product as far as the users are concerned.
V-Model.
- Requirement specification- user needs
- Functional specification- functions required to meet the user needs.
- Technical specification- technical design of the function identified in the functional specification.
- Program specification- detailed design of each module or unit to be built to meet required functionality.
Specification review:
· Conformance to the previous work product (in case of functional specification, verification would include a check against the requirement specification).
· Sufficient detail for subsequent work product (again, for the functional specification, this would included a check that there is sufficient information in order to create the technical specification).
· Testable- details are sufficient for testing.
Middle of v-model show the planning for testing should start with each work product.
The right side focuses on the testing activities for each work product.
- Acceptance testing: testing against requirements.
- System testing: against functional requirements.
- Integration testing: against technical specifications.
- Unit testing: against program specification.
Advantage: defects can be identified and resolved as early as possible.
But disadvantage is that you can not go next stage until complete first stage as a result changes in requirements are uncovered until the user testing is carried out.
Iterative Development Model
Requirements do not need fully defined before coding can start. Instead a working version of the product is built in a series of stages or iterations- each stage encompasses requirement definition, design, code and test. In this model each cycle has a define timescale and cost. And cycles are referred as time boxes. At the end of each time box a decision is made about functionality needs to be created for the next iteration. The advantage of this is involvement of user representative in testing. Which minimize the risk in development? But disadvantage is that the lack of formal documentation makes testing difficult. For this developer may use test –driven development approach. Developer makes change without documents which could lack of traceability so for this a change management process is needed. Furthermore need amount of testing for implementing any changes so that does not cause unintended changes to other parts of the software.
Iterative development includes prototyping, rapid application development (RAD) and agile software development. A proprietary methodology is Rational Unified Process (RUP).
TEST LEVELS
Characteristics of good testing are:
- Early test design- begins with specification documents.
- Each-work product is tested- each document in v-model on left is tested by an activity on the right. So specification document is test basis.
- Testers are involved in reviewing requirements before they are released.
The test stages in v-model are called test levels:
- Unit testing- code testing which build in isolation. Units are programs, module or components or units. For checking conformance to the program specification unit test would also verify that all of the code that has been written for the unit can be executed. Defects found and fixed during unit testing are often not recorded.
- Integration testing- is to put units together. The purpose of integration testing is to expose defects in the interfaces and in the interactions between integrated components or systems. There are three integration strategies.
- Big Bang integration- where all units are linked at once, results in a complete system. Difficult to isolate any error found because of not paying attention to each individual unit.
- Top down integration- system is built in stages starting with components which called other components. Calling component is at top and called components are at bottom. In it the interactions of each component must be tested when it is built. When a component is not integrated yet then stub (skeletal implementation of the component) is used which is a passive component, called by other components. So stub is commonplace in top down approach.
- Bottom up integration- opposite to top down approach. In it the components which are not in place are those which actively call other components. In bottom up approach of replacing these components with special components they called drivers because they are active and calling other components.
So there may be more than one level of integration testing.
-
-
- Component integration- intersection between software components and done after unit testing. This type of integration is carried out by developers.
- System integration – intersection between different system and done after system testing of each individual systems. This is usually carried out by testers.
System testing:
This is to check the functionally from end to end perspective. It determines the behaviour of the system/product defined in the scope documents. It is done by an independent team not development team base on the specification as written, and not the code. It check that system built meet the system requirements. Functional specification should contain definitions of both the functional and not functional requirements of the system.
Functional requirements- is a requirements that specifies a function that a system or system component must perform. It could be specific to a system. What application will do i.e. check your fund in bank accounts.
Non functional requirements- are important but not directly related to what the system performs.
Examples of not functional requirements are:
- Install ability- installation procedures
- Interoperability-operation of application in different environments.
- Maintainability- ability to introduce changes to the systems.
- Performance- expected normal behaviour.
- Load handling- behavior of the system under increasing load.
- Stress handling- behaviour at upper limits of the system capability.
- Portability- use on different operating platforms.
- Recovery- recovery procedures on failure.
- Reliability- ability of the software to perform its required functions.
- Usability- ease with which users are engage with the system.
But Security is a functional requirement.
Acceptance testing:
It provides confidence to end users that system will function according to their expectations. It is also base on the specification document. In other words it is the system conformance to the customer requirements. It assesses the system readiness and deployment and use. It is the responsibility of customers and users.
Forms of acceptance testing:
- User acceptance testing
- Operational acceptance testing- it include checking the
- Back up facilities
- Procedures for disaster recover
- Training for end users
- Maintenance procedures
- Security procedures.
- Contract and regulation acceptance testing
- contract acceptance testing-
- Regulation acceptance testing- governmental, legal or safety standards.
- Alpha and beta testing:
1. Alpha testing take place at developer site. Operational system testing.
2. Beta testing- at customer site. Operational system is tested by a group of customers at their own locations and provides feedback. It is also called field testing.
Confirmation testing: When a defect is detected and fixed then the changed software should be retested to confirm that the problem has been successfully removed.
Debugging: when a developer removed the defects, this is called debugging? But it is not testing activity.
Regression testing: when unchanged software should also be retested to ensure that no additional defects have been introduced as a result of changes to the software.
Maintenance testing: testing which take place on a system which is in operation in the live environment is called maintenance testing.
Impact analysis: an understanding of the parts of the system which could be affected by the changes could reduce the amount of regression testing required which is practically impossible and not cost effective. In other word the analysis of impact of changes on the system.
Chapter3: Static Testing
Static testing: are those techniques that test software without executing the code. This includes both the testing of work-products other than code, typically requirements or specification documents and testing of code without executing it. First is known as review and second is known as static analysis, review are completed manually and static analysis is normally completed automatically using tools.
Review and test process:
A review is a systematic examination of a document by one or more people with the main aim of finding and removing errors in requirement specification, design, code, test plans and test cases any thing written/typed in during the software development life cycle.
Benefits of review:
Important for cost and time saving and many others which are as?
- Development productivity can be improved and timescale reduced because the correction of defects in the early work-products will help to remove product ambiguities.
- Testing cost and time.
- Reductions in lifetime costs i.e. on-going support costs will be lower.
- Improved communication results.
The types of defects most typically found in review are.
- Deviations from standards.
- Requirements defects.
- Design defects. I.e. not matching requirements
- Insufficient maintainability i.e. too complex
- Incorrect interface specifications.
Review Process: reviews are formal or informal. The decision on the appropriate level of formality for a review usually based on combinations of the following factors:
- The maturity of development process.
- Legal or regulatory requirements.
- The need for an audit trail. The level of formality in the type of review used can help to raise the level of audit trail.
Reviews objectives:
The reviews have varieties of objects.
- Finding defects
- Gaining understanding.
- Generating discussion.
- Decision making by consensus.
Basic review process:
Formal and informal review both exhibit same elements of process.
- The document under review is studied by the reviewers.
- Reviews identify issues or problems and inform the author either verbally or documented form.
- The author decides on any action to take in response.
Phases of a formal review:
Characteristics of formal review:
-
- Selecting the personnel.
- Allocating roles
- Defining the entry and exit criteria
- Selecting the parts of documents to be reviewed (not always required).
- Kick-off:
- Individual preparation
- Review meeting
- Time available (short meeting may collect defects).
- Requirements of the author (to help in correction of defects).
- Type of review.(collection of defects not discussion)
- Rework (correction of defects by author).
- Follow up: the review leader checks that the agreed defects are corrected and how much time has been spent on it. He will also check the exit criteria.
Roles and responsibilities in review: (user, maintainer, tester or operations)
The specific roles and responsibilities that should be fulfilled for formal reviews are:
- Manager: what to reviewed and ensure the sufficient time allocated in the project plan for all the required review activities and determine that if the review objectives have been met.
- Moderator: this is who lead the review of document or set of documents including planning, the review, running the meeting and follow up the meetings, review leader or a person upon whom the success of review rests.
- Author: responsible for fixing any defects.
- Reviewers: have business or technical background also called checkers or inspectors.
- Scribe or recorder: attend meeting and document all defects, issues and problems.
Role of tester is not normally associated with reviewers.
Types of review:
There are four types of reviews.
- informal review (least formal) key characteristic:
- no formal process undertaking
- may or may not documented
- Reviewer does not have technical skill but to check and ensure that document makes a sense.
- Main purpose to find defects and it is inexpensive way.
- Review may be implemented by pair programming (one programmer reviews the code of the other pair programmer) or technical lead reviewing design and code.
- Walkthrough. Key characteristics.
· Meeting lead by author of document and attended by peer group.
· Review session is open ended and may vary in practice for quite informal to very formal.
· Preparation by reviewers before the walkthrough.
· Main purpose is to enable learning about content of document under review, to help team members gain an understanding.
· Walkthrough typically explore scenarios or conduct run of code or process.
- Technical review. Key characteristics:
· Include peers and technical experts.
· Led by trained moderator and performed as a peer review without participation of management.
· Prepare a review report about the list of finding.
· Vary in practice and have numbers of purposes i.e. Discussion, decision making, evaluation of alternatives, finding defects, solving problems and checking conformance to specifications and standards.
- Inspection (most formal). Key characteristics:
· led by moderator
· Inspection process is formal, base on rules and check lists and uses of entry and exit criteria.
· Pre-meeting preparation is essential to ensure document consistency.
· An inspection report with a list of findings to aid improvements including correcting defects under reviews.
· From follow up process is used to ensure corrective action in completed and timely.
· Main purpose is to find defects and process improvement may be a secondary purpose.
Success factors for reviews:
- Each review should have a clearly predefined and agreed objective and the right people should be involved to ensure the objective is met.
- Any defects found are welcomed, and expressed objectively, and the review leader has ensured that the people issues and psychological aspects are dealt with.
- Review techniques (both formal and informal) that are suitable to the type and level of software work products and reviewers (this is especially important for inspections).
- Checklist or roles should be used where appropriate, to increase effectiveness of defect identifications.
- Management support is essential for a good review process.
- Emphasis on learning and process improvement.
For quantitative success measurement could be as:
- How many defects found.
- Time taken to review/inspect.
- Percentage of project budget used/saved.
Fagan inspection method was developed by Michael Fagan in 1976 of IBM.
Static analysis by tools:
The purpose of static analysis is to find defects is software source code and software models without executing the code.
Source Code: a series of statements written in some human-readable computer programming language.
Software model: is an image of the final solution developed using techniques such as UML (unified modeling language) and normally generated by designer.
Static analysis finds defects which are hard to find during the execution.
The benefits or value of static analysis are:
- Early detection of defects prior to test execution.
- Early warning about suspicious aspects of code or design by using the calculation of metrics, such as high –complexity measure.
- Identification of defects not easily found by dynamic testing such as development standard breaches as well as detecting dependencies and inconsistencies in software models, such as links or interfaces that were either incorrect or unknown before static analysis was carried out.
- Improved maintainability of code and design.
- Prevention of defects (by identification early in the life cycle is easier to identify why it was there in the first place root cause analysis).
The defects discovered by static analysis tools include.
- Reference a variable with an undefined value i.e. using a variable as a part of calculation before the variable has been given a value.
- Inconsistent interface between modules and components.
- Variables that never used.
- Unreachable code (dead code) mean did not execute.
- Programming standards violations.
- Security vulnerabilities i.e. password structures that are not secure.
- Syntax violations of code and software models i.e. using wrong modeling language.
A static analysis tool run automatically and reports all defects it identifies either significant or not.
Chapter 3: Test Design Techniques.
The specification of the test case is the second case of test specification. The term specification and design are used interchangeably.
The design of test comprises three main steps.
- Identify test condition
- Specify the test cases.
- Specify test procedures.
Identify test condition: an item or event of a component or system that could be verified by one or more test cases e.g. A function, transaction, feature, quality attribute, or structural element.
A test case: a set of input values, execution preconditions, expected results and execution post conditions, developed for a particular objective or test condition , such as to exercise a particular program path or to verify compliance with a specific requirement.
A test procedure specification: a sequence of action for the execution of a test. It is also called test scripts to distinguish it from automated scripts that control test execution tools.
To test a whole system a test execution schedule which puts all the individual test procedures in the right sequence and sets up the system so that they can be run?
Test coverage idea: it provides a quantitative assessment of the extent and quality of testing. It is important for two reasons.
- Quantitative measure of the quality of testing that has been done by measuring what has been achieved.
- Estimating how much more testing needs to be done, using quantitative measure can set targets for test coverage and measure progress against them.
Categories of Test Case Design Techniques
There are three categories.
1. Those based on deriving test cases directly form a specification or a model of a system, known as specification-based or black-box techniques.
2. Those based on deriving test cases directly form the code written to implement a system, known as structure-based or white-box techniques.
3. Those based on deriving test cases from the tester’s experience of similar systems and general experience of testing, known as experience based techniques. It also called ‘ad hoc’ techniques. It is not a systematic technique.
Specification Based or Black Box Technique:
If there is no specification then tester has to build a model of the proposed system, by interviewing key stakeholders to understand what their expectations are. It provides the test basis to generate tests systematically.
Here are five specification based techniques which are:
- Equivalence partitioning
- Boundary values analysis
- Decision table testing
- State transition testing
- Use case testing.
Now have look on each one.
Equivalence Partitioning:
Input partitions
It is that in many cases the inputs to a program can be chunked into groups of similar inputs. This technique has advantage of the properties of equivalence partitions to reduce the number of test cases we need to write. In it both valid and non valid partitions are important to test.
See examples on book page no 81:
Out put partitions:
As input to a programmed can be partitioned, so can the output. So need a test case that generates each of these outputs as an alternative to generating input partitions.
Other partitions and examples on book page 82.
Boundary Value Analysis
Each component has a limited value e.g. 1 to 99 these are boundary of integer inputs. This is work well close to equivalence partition analysis because of partitions must have boundaries.
“As Close As we can get “means take the next value in sequence using the precision that has been applied to the partition.
For example see book page 83.
Decision Table Testing
It is derived from the specification because specification contains the business rules to define the function of system and the conditions under which each function operates. It is the mechanism used to capturing the logical decision is called a decision table. It inputs all conditions and actions that can arise from them. Conditions are on the tope and actions are in bottom. The business rules involve the combination of some condition to produce some combination of actions, are arranged across the top and in table these are in form of column. In reality there are large number of condition and actions but number of combinations producing specific actions is relatively small. The tables which have limited entries is called limited entry table and distinguish from others which have large inputs identified. From decision table we can determine test cases by setting values for the conditions and determining the expected output. So each column represents a test case.
State Transition Testing
In it output triggered by changes to the input conditions, or change state. In other word behaviour depends on current state and past state. So the main diagram used in this technique is called state transition diagram.
State Transition Diagram
It is a representation of the behaviour of a system. It is made up from just two symbols.
A state mean the system is static in a stable condition and it will only change if it is stimulated by an event of some kind. E.g. a TV stays on until you turn it off.
The second is arrow which is symbol of transition e.g. change from one state to another. The state change will be triggered by an event e.g. pressing button. The start state will have a double arrow head pointing to it.
Event
An event is anything that acts as a trigger for a change; it could be input to the system or inside the system e.g. data base field being updated. What happens for each event each time we want to initiate a test? We can take the intermediate step of creating what is known as a state table. It records all possible events and all the possible states. State table is the source from which we derive the test cases. For each combination of event and state it shows the outcome in terms of the new state and any outputs that are generated. ST has a row for each state in the state transition diagram and a column for every event. If the event has no effect we label the table entry with a symbol that indicates that nothing happens; this is sometimes called a ‘null’ transition or an ‘invalid ‘transition. We should also test negative test cases. Which mean those cases where the ST indicates there is no valid transition?
Use Case Testing
They capture the individual interactions between ‘actors’ and the system. An actor is user and use cases capture the interactions that each user takes part in order to produce some output that is of value. Test case based on use case, often called scenarios. This is high level view of requirements. It has major benefit that it relates to real user processes. The main thing is that first test the highest priority use cases.
Structure-based or white Box techniques
It is used to explore the program structures. It is used to ensure that each statement in the code is executed at least once. In this technique test cases generate from the code. It is mostly done by the specialist tools in reality but the knowledge of the techniques is still valuable. In this technique you can say that the code is the starting point. In it code is always mean pseudo code. It is limited language than any real language. But it enables the designer to create all the main control structure needed by programs. It is sometimes used to document designs.
Code can be of two types, executable and not executable. Executable code instructs the computer to take some action; non executable code is used to prepare the computer to do its calculation but it does not involve any actions. E.g. it is reserving space for some kind of calculation. In pseudo code non executable statement at beginning of the program and executable program start with Begin and End statement.
There are three ways that executable code can be structured which are:
- Sequence: statements execute one after another.
- Selection: computer has to decide if a condition is true or false (Boolean).
- Iteration: a chunk of code is executed more than once.
Flow Charts
Is the visual representation of the program structure? It consists of two symbols. Rectangles request sequential statements and diamonds represent decisions.
Control Flow Graphs
This provides a method of representing the decision points and the flow of control within a piece of code, it is similar to flow chart but it only shows decisions. It is produced looking only at the statements affecting the flow of control. It also made of two symbols: node and edges.
Node: represents any point where the flow of control can be modified (i.e. decision points), or the points where a control structure returns to the main flow (e.g. END WHILE OR ENDIF).
Edge: is a line connecting any two nodes.
Region: the area contained within a collection of nodes and edges.
We can draw sub graphs, to represent individual structures.
Techniques to draw flow graph:
- Analyse the component to identify all control structures i.e. all statements that can modify the flow of control, ignoring all sequential statements.
- Add a node for a decision statement.
- Expand the node by substituting the appropriate sub graph representing the structure at the decision point.
Any chuck of code can be represented by using these sub graphs.
Statement Testing and Coverage
Statement testing is testing aimed at exercising programming statements. It is used to test the executable statements. And if we test 100 percent then it is called 100 percent coverage. The flow chart is good because the rectangles and diamonds show the executable statements but a flow graphs is less cluttered, showing only the structural details, in particular where the program branches and rejoins. Do we need both diagrams? Well, neither has everything that we need.However we can produce a version of the flow graph that allows us to determine statement coverage. To do this we build a conventional control flow graph but then we add a node for every branch in which there are one or more statements.
To understand what happen with the control structure when program runs. So best way is to ‘dry run’ the program with some inputs; this means writing down the inputs and then stepping through the program logic noting what happens at each step and what values change. When you go to end you know the complete path.
Hybrid Flow Graph
An additional node that represent the edges with executable statements in them; they make it a little easier to identify what needs to be counted for statement coverage.
Flow charts, control flow graphs and hybrid flow graphs all show essentially the same information, but sometimes one format is more helpful than other. So hybrid flow graph is as a useful combination of the control flow graph and the control flow chart. To make it even more useful we can add label to indicate the paths that a program can follow through the code. All we need to do is to label each edge; paths are then made up from sequence of the labels see figure 4.12. And for more detail see book page 107.
Rules for statement coverage
- Exclude the non-executable statements that precede Begin.
- Ignore blank lines that have been inserted for clarity.
- Are you consistent about what you do or do not include in the count with respect to control structures.
Decision Testing and Coverage
The aim of it is to ensure that the decisions in a program are adequately exercised. Decision coverage is measured by counting the number of decisions exercised (i.e. both exits are exercised) divided by the total number of decisions in a given program. It is usually expressed as a percentage.
For testing the decision two test cases needed but for loops one test case is enough.
Experienced Based Techniques
This technique is used when there is no specification from which you derive specification-based test cases or no time to run full structured set of tests.
All type of testing tap knowledge and experience either ad hoc or error guessing through more sophisticated techniques such as exploratory testing.
Error Guessing
A special test to find errors that could not find by the formal techniques. It takes advantage of tester’s knowledge skill and experience with similar applications. Advantage of this technique is that when applied after systematic techniques; add value in identifying and exercising test cases. But main drawback is that its effectiveness can depends upon the tester experience but could be overcome by listing a number of possible errors by contributing the testers and or users of system.
Error guessing could become more structure by creating list of defects and failure points. This list could be used as a basis of a set of tests that are applied after the systematic technique has been used.
Exploratory Testing
Exploratory testing is a technique that combines the experience of testers with a structured approach to testing where specifications are either missing or inadequate and where there is severe time pressure. It exploits concurrent test design, test execution, test logging and learning within time-bones and is structured around a test charter containing test objectives.
Using objectives tester could focus on more important area and can maximize the amount of test achieved.
Systematic and Experienced –Based Techniques
Which one is best use the simple Rule of Thumb?
- Always use functional testing as the first priority but some time may be needed to use structured techniques for early code product testing.
- After functional testing think about the test coverage because when coverage is inadequate then extra tests will be needed.
- Structural method is supplement to functional methods even functional coverage is adequate. It is worth to checking the statement and decision coverage to ensure that enough of code has been exercised during testing.
- Once the systematic testing is complete then there is opportunity to use experience-based testing to ensure that the most important and more error prone areas of the software have been exercised? In some circumstance e.g. time pressure, poor specification, experienced based techniques is a viable option.
Choosing which technique is best does not only the one factor. There are number of factors to bear in mind which are;
- Type of system
- Regulatory standards
- Customer or contractual requirements
- Level of risk
- Type of risk
- Test objectives
- Documentation available
- Knowledge of the testers
- Time and budget
- Development life cycle
- Use case methods
- Experience of type of defects found.
Some leave no room for selection e.g. regulatory or contractual requirements leave the tester with no choice. Test objectives, where they relate to exit criteria such as test coverage, may also lead to mandatory techniques.
Chapter 5 Test Management
No need of testing if there is no chance of adverse future events in software or hardware development.
Risk
A factor that could result in future negative consequences, usually expressed as impact and likelihood.
Level of Risk = probability of the risk occurring* impact if it did happen.
Project Risks:
The project leader will use project risks to manage the capability to deliver.
a: failure to supply on time
b: acceptance criteria
a: skill and staff shortage.
b: personal & training issues.
c: political issues, change of management/restructuring that will affect the project resources.
- Problems that stop testers to communicating their needs and test results.
- Failure to follow up on low-level testing and reviews.
a: lack of appreciation of testing benefits.
a: problems in defining right requirements.
b: requirement met with existing project constraints.
c: the quality of design, development and test team.
Risks are analysed, manage and mitigate. These are recognized during test planning should be documented in the IEEE 829 test plan. Risk register should be maintained by test leader for the on-going management and control of existing and new project risks.
Product Risks:
Tester or test leader using the risk-based testing approach for managing product risks. Potential failure areas (adverse future events or hazards) are the product risks.
Product risks are:
- Error prone software delivered.
- Poor requirements leading to badly define and built software.
- Potential that a defect in the software/hardware could cause harm to an individual or company.
- Poor software quality characteristics (E.g. functionality, security, reliability, usability, performance) leading to poor user feedback.
- The software does not meet the requirements and delivers functionality that was not requested.
Using risks, leader decides from where to start testing in the software development i.e. poor requirements is the major risks.
Product risks also provide information regarding decisions, means how much testing should be carried out on specific components or systems.
Testing is used to reduce the adverse effects of risks.
So risk based approach provide the proactive opportunities to reduce level of risks.
In risk based approach the risks identified are:
- Will determine the test technique to be employed or extent of testing carried out e.g.. [MISRA] Motor Industry Software Reliability Association define which test techniques should be use for each level, more risk, more coverage required from test techniques.
- Priorities testing is an attempt to find the critical defects as early as possible i.e. areas having more defects or complexity.
- Non-test activities to reduce risk e.g. provide training to inexperienced designers.
Risk-based testing draws on collective knowledge and insights of project stakeholders, testers, designers etc. and any one with solution of knowledge to determine risks and level of testing to address those risks.
To minimize product failure risk management provide the disciplined approach.
- To assess continuously what can go wrong (risks). For these regular reviews of the existing and looking for new product risks should occur periodically throughout life cycle.
- Determine the priority of risks using probability and likelihood. So for reducing these risks mitigating activities should be carried out.
- To implements actions to deal with those risks (mitigating actions).
Testing is actually support the identification of new risks by continually reviewing risks of the project deliverables through out the life cycle.
To deal with risks identified different strategies, such as contingency plans.
Testing is a risk control activity that provides feedback about the residual risk in the product by measuring the effectiveness of critical defect removal and by reviewing the effective of contingency plans.
Test Organisation
Independent testing: is a testing carried out by someone other than the creator
(Developer) of the code being tested.
Developer: as the creator and owner of documents and code related to development, perceives these deliverables as being correct when they are delivered.
Tester: by contrast will take the view that any thing delivered for testing is likely to contain errors and will search diligently to identify and locate those errors.
The greater is the level of independence, the greater the likelihood of errors in testing arising from unfamiliarity. So level of independence depend on the size of the organisation.
For complex or safety critical projects, best to have multiple level of testing with some or all of the levels done by independence tester.
The ‘agile’ approach to development challenges the traditional approach to independence. In this approach every body takes on multiple roles and so maintaining total independence is not always possible.
Feature of independence testing see book page [131], software testing.
Tasks of a test leader and tester
Test leader role: could be a project manager or who have skill and experience of testing but depend upon the structure and organisation and complexity, size of the system. It is important to understand testing job and role. A role is an activity or a series of activities given to a person to fulfill e.g. the role of test leader. A person can have more than one role at any moment but depend upon their experience and level of understanding. A job is effectively what an individual is employed to do.
Tasks of a leader include:
- Coordination in development of strategy and planning with others.
- Writing and reviewing test strategies and policies produced for the project.
- Contributing the testing perspective to other project activities such as project deliverable schedule.
- Planning of required tests-which will include ensuring that the development uses the correct understanding of the risks, selecting the required test approaches (test levels, cycles, approach, objectives and incident management planning), estimating the time and effort and converting to the cost of testing and acquiring the right resources.
- Managing the specification, preparation, implementation and execution of tests, including the monitoring and control of all the specification and execution.
- Taking the required action, including adapting the planning, based on test results and progress (sometimes documented in status reports), and any action necessary to compensate for problems or delays.
- Ensuring that adequate configuration management of test ware is in place and that the test ware is fully traceable e.g. there is a hierarchical relationship establishes between the requirements and the detailed specification documents.
- Putting in place suitable metrics for measuring test progress and evaluating the quality of the testing delivered and the product.
- Agreeing what should be automated, to what degree, and how, ensuring it is implemented as planned.
- Where required, selecting tools to support testing and ensuring and any tool training requirements are met.
- Agreeing the structure and implementation of the test environment
- Scheduling all testing activity.
- At the end of the project, writing a test summary report based on the information gathered during testing.
Test leader tasks under taken by test leader very closely align with project manager and close with standard approaches to project management. The key is to ensure that everyone is aware of who is doing what tasks, that they are completed on time and within budget, and that they are tracked through to completion.
Tasks of Tester
The task under taken by tester may include.
- Reviewing and contributing to the development of test plans.
- Analyzing, reviewing and assessing user requirements, specifications and models for testability.
- Creating test specification from the test basis.
- Setting up the test environment (often coordinating with system administration and network management). In some organisations the setting up and management of the test environment of the test environment could be centrally controlled: in this situation, tester would directly liaise with the environment management to ensure the test environment is delivered on time and to specification.
- Preparing and acquiring/copying/creating test data.
- Implementing tests on all test levels, executing and logging the tests, evaluating the results and documenting the deviations from expected results as defects.
- Using test administration or management and test monitoring tools as required.
- Automating tests (may be supported by a developer or a test automation expert).
- Where required, running the test and measuring the performance of components and systems (if applicable).
- Reviewing tests developed by other testers.
If specialist testers are not available, then additional resources could be used at different test levels.
- For component and integration testing, any additional roles would typically be filled by someone from a development background.
- For system and user acceptance testing, any additional roles would typically be filled by someone from a business or user background.
- System operator (sometimes known as production support) would be responsible for operational acceptance testing.
Within test project a job may contain many roles and tasks.
Test Approaches (Test Strategies)
It defines how testing will be implemented. A strategy should be developed early in
- In the project life cycle to stop defects.
- Left until start of execution, also called reactive approach. Sometime follow waterfall model/using sequential approach mean next will not start until previous one has nearly finished.
There are many approaches/strategies that can be employed including:
- Analytical approaches: such as risk based testing where testing is directed to areas of greater risk.
- Model-based approaches: such as stochastic testing using statistical information about failure rates (reliability growth or usage (such as operational profiles).
- Methodical approaches: such as failure based (including error guessing and fault attacks), check list based and quality-characteristic based.
- Standard-compliant approaches, specified by industry compliance standard such as MIRSA.
- Process-compliant approaches, where various processes developed for use with the various agile methodologies or traditional waterfall approaches.
- Dynamic and heuristic approaches: such as exploratory testing where testing is more reactive than pre-planned, and where execution and evaluation are concurrent tasks.
- Consultative approaches: such as those where test coverage is driven primarily by the advice and guidance of technology and/or business domain experts outside or within the test team.
- Regression-averse approaches: such as those that include reuse of existing test material, extensive automation of functional regression tests, and standard test suites.
Different approaches can be combined but depend upon the circumstances prevalent in a project at the time.
Some time standard i.e. MIRSA may dictate to choose a strategy but following factors must be considered when define strategy/approach.
· Risk of failure of the project.
· Skills and experience of the people in the proposed techniques, tools and methods. There is no point of implementing the sophisticated component-level, technique-driven approach or strategy when the only resources available are business users with no technical grounding.
· The objective of the testing endeavour and the mission of the testing team. E.g. finding most serious defects.
· Regulatory aspects such as external and internal regulations for the development process e.g. MIRSA.
· The nature of the product and the business e.g. different approaches is required for testing mobile phone coverage than for testing an online banking operation.
Testing Planning and Estimation
Test Planning
Testing planning normally is under taken by the project leader. It contain a list of tasks and milestone against which the progress against. In other words define the shape of testing efforts in development and implementation project called ‘green field’ as well as maintenance (change and fix) activities. Main document produced in testing planning is called master test plan.
This document defines high level test activities being planned. The details of test-level activities are documented within test level plans e.g. the system test plan.
The factors influencing the testing process are:
- Risks, test plicy- constraints, scope of testing- resources, test objectives- critically, testability.
The content section of plan is normally identical to IEEE829 which is the standard for software test documentation. This standard identifies 16 sections present in the test plan.
Test is a continuity activity that spans over the life of the project. As risks and change occurs the plan and planning should be amended to recognize these and reflect the current position.
Changes are introduced through a change process. The 16 point section is acronym of SPACDIRT which is as:
- S= scope (testing items, features to be tested and feature not to be tested ).
- P= people (responsibilities, staff and training and approvals).
- A= approach
- C= criteria (including item pass/fail criteria and suspension and resumption requirements).
- E= environment needs
- D=deliverables (test)
- I= identifier and introduction (test plan)
- R= risks and contingencies
- T=testing tasks and schedule
Test- planning Estimation
There are number of testing activities:
- Putting together the overall testing approaches (sometimes called test strategy) ensuring that the test levels and entry and exit criteria are defined.
- Liaising with the project manager and making sure that the testing activities have been included within the software life-cycle activities such as: design, development, and implementation.
- Working with the project what to be tested, who plan, test, involved and how test activities should be done, how result will be evaluated and define exit criteria (stop testing).
- Finding and assigning resources for different tasks under taken during testing.
- Deciding what the documentation for the test project will be e.g. which plans, how test cases will be documented etc.
- Defining the management information, including the metrics required and putting in places the processes required to monitor and control test preparation and execution, defect resolution and risk issues.
- Ensuring that the test documentation generate repeatable test assets e.g. test cases.
Exit Criteria
Exit criteria mean when a test activity will be complete and when it should be stop. There should be a exit criteria for all test project activities e.g. planning, specification and execution as a whole or some specific level for specific specification as well as execution.
Exit criteria must be specified in the relevant plan. Some exit criteria might be:
- All test planned have been run.
- A certain level of requirement coverage has been achieved.
- No-high priority or severe defects are lefts outstanding.
- All high risk areas have been fully tested, with only minor residual risks left outstanding.
- Cost-when the budget has been spent.
- Schedule has been achieved e.g. release date has been reached and product has to go live.
It should have been agreed as early as possible in the life cycle, but some time subject to controlled changes but must be understand by those responsible for delivery.
Test Estimation
There are two approaches:-
- Metrics-based
- Expert-based
Metric-based approach:
It relies upon the data collected from previous or similar projects.
This kind of data include:-
- Number of test conditions
- Number of test cases written.
- Number of test cases execution
- Time taken to develop test cases.
- Time taken to run test cases.
- The number of defects found.
- The number of environment outages and how long an average each one tested.
Using this approach, quite easy to estimate the accurate time and cost required for similar project.
Expert-base approach:-
It is relies on the experience of owners of relevant tasks or expert to derive an estimate. It also kwon as Wide Band Delphi approach. In this context, ‘expert’ could be;
- Business expert
- Test process consultants.
- Developers
- Technical architects.
- Analysts and designers.
- Anyone with knowledge of the application to be tested or the tasks involved in the process.
There are many ways of using this approach e.g.
- Distribute a requirement specification to the task owners and get them to estimate for their task in isolation. Amalgamate the individual estimates when received; build in any required contingency, to arrive at the estimate.
- Distribute to known experts who develop their individual view of the overall estimate and then meet together to agree on and or debate the estimate that will go forward.
Many things affect the level of effort required to fulfill the test requirements of a project. These are split into three main categories as:
1: Product characteristics:
· Size of the test basis
· Complexity of the final product
· The amount of non-functional requirements
· The security requirements (perhaps meeting BS7799 (, the security standard).
· How much documentation is required e.g. some legislation –driven changes demand a certain level of documentation which may be more than an organisation would normally produce ).
· The availability and quality of the test basis e.g. requirements and specifications.
2: Development process characteristics:
· Timescales
· Amount of budget available
· Skills of those involved in the testing and development activity ( the lower the skill level in development, the more defects could be introduced, and the lower the skill level in testing, the more detailed the test documentation needs to be );
· Which tools are being used across the life cycle ( i.e. the amount of automated testing will affect the effort required ).
3: Expected outcome of testing such as:
- The amount of errors
- Test cases to be written.
Taking all of this into account, once the estimate is developed and agreed the test leader can set about identifying the required resources and building the detailed plan.
Test Progress Monitoring and Control
1: Test Progress monitoring
The activities and timescale determined within the test plan are constantly monitored, mean what is actually happened. The purpose of test progress monitoring is to provide feed back and visibility of the progress of test activities.
The data required to monitor progress can be collected manually e.g. test cases developed or by using test management tools or to collect the data as an automatic output from a tool either already formatted into a report or as a data file that can be manipulated to present the picture of progress. The progress data is also used to measure exit criteria such as test coverage e.g. 50percentage requirement coverage achieved.
Common test metrics include:
- Percentage of work done in test case preparation (percentage of planned test cases prepared).
- Percentage of work done in test environment preparation.
- Test case execution (e.g. number of test cases run/not run, and test cases passed/failed).
- Defect information (e.g. defect density, defects found and fixed, failure rate and retest results).
- Test coverage of requirements, risks or code.
- Subjective confidence of testers in the product.
- Dates of test milestones.
- Testing costs, including the cost compared with the benefits of finding the next defect or to run the next test.
These metrics are used to track progress towards the completion testing, which is determined by the exit criteria. So exit criteria are directly related to metrics.
There is a trend towards ‘dash board’ which reflects all of the relevant metrics on single screen or page ensuring maximum impact. For a dashboard, and generally when delivering metrics, it is best to use a relatively small but impact-worthy subset of various metric options available. Because reader does not want to wade through lots of data for the key information they are after, which invariably is ‘are we on target to complete on time?’
Metrics are displayed in graphical form which reflects progress on the running test cases and report on defects found.
2: Test Reporting
It is a process of reporting test metrics in summarized form updating the reader regarding the testing tasks undertaken.
The reported information can include:-
- What happened during a given period of time e.g. a week, a test level, whole test endeavour or when exit criteria have been met?
- Analysed information and metrics required to support recommendations and decisions about future actions. Such as :-
- an assessment of defects remaining;
- The economic benefits of continued testing e.g. additional testes are exponentially more expensive than the benefit of running.
- Outstanding risks.
- The level of confidence in tested software e.g. defects planned vs. actual defects found.
The IEEE 829 standard include outline of test reporting summary which could be used for test reporting.
- Test summary report identifier; e.g. TSRXYZ.
- Summary- documents the environment in which the test activity being reported on took place.
- Variances- test strategy, plan, specification, procedures etc.
- Comprehensive assessment.
- Summary results;- results from test activities, include detailed defects raised and fined and those that remain unresolved.
- Evaluating;-quality of test, pass/fail criteria.
- Summary of activities:- major testing activities, test environment, unavailability, successor weakness test specification process etc. should include resource usage data e.g. planned spend against the actual plan.
- Approvals;- identifies all approvers of the document.
This information’s could also be used to help to process improvement activities.
This information could be used to assess whether:-
- Goals for testing were correctly set (achieved, if not why?)
- Test approach/ strategy were adequate (e.g. did it ensure there was enough coverage?).
- Testing was effective in ensuring that the objectives of testing were met.
3: Test Control
Test control uses the information’s collected in the monitoring and reporting phases to decide on a course of action to ensure control of test activities is maintained and exit criteria are met. It needs only when planned test activities are behind the schedule.
The action taken could impact any of the test activities and may also affect other software life cycle activities.
Examples are:
- Reprioritizes tests when an identified project risk occurs (e.g. software delivered late).
- Change the test schedule due to availability of a test environment.
- Set an entry criterion requiring fixes to be retested by a developer before accepting them into a build (this is particularly useful when defect fixes continually fail again when retested).
- Review of product risks and perhaps changing the risk ratings to meet the target.
- Adjusting the scope of the testing (perhaps the amount of tests to be run) to manage the testing of late change requests.
The following are not the responsibility of the test leader but not stop him to making recommendation to the project manager.
- De-scoping functionality e.g. removing less important planned deliverable from the initial delivered solution to reduce the time and effort required to achieve that solution.
- Delaying release into the production environment until exit criteria have been met.
- Continuing testing after delivery into the production environment so that defects are found before they occur in production.
Incident Management
An unplanned event occurring that requires further investigation. In testing, anything where the actual result is different to the expected result.
Incidents can be reviewed at any time in the software life cycle e.g. from review of the test basis (requirements, specifications etc) to test specification to test execution.
Incident management, according to IEEE 1044 (standard specification for software anomalies), is ‘the process of recognizing, investigation, testing action and disposing of incidents. It involves recording incidents, classifying them and identifying the impact.
The process of incident management ensures that incidents are tracked ensures that incidents are tracked from reorganization to correction, and finally through retest and closure. It is important that organisations documents their incident management process and ensure they have appointed someone (often called an incident manager/coordinator) to mange/police the process.
Incidents are raised an incident reports via electronically or on paper.
Incidents are raised on incident reports have the following objectives.
· To provide developers and other parties (involved in testing) with feedback on the problem to enable identification, isolation and correction as necessary. The more information’s provided the better.
· Provide test leader to track quality of system under test and progress of testing. One of the key metrics used to measure progress is a view of how many incidents are raised; their priority and finally those they have been corrected and signed off.
· To provide ideas for test improvement. For each incident the point of injection should be documented e.g. a defect in requirements or code and subsequent process improvement can focus on that particular area to stop the same defect occurring again.
The detail that normally included on an incident is:-
- Date of issue, issuing organisation, author, approvals and status.
- Scope, severity and priority of the incident.
- References, including the identity of the test case specification that revealed the problem.
- Expected and actual results.
- Date the incident was discovered.
- Identification or configuration item of the software or system.
- Software or system life-cycle process in which the incident was observed.
- Description of the anomaly to enable resolution.
- Degree of impact on the stakeholder(s) interests.
- Severity of the impact on the system.
- Urgency/priority to fix.
- Status of the incident (e.g. open, deferred, duplicate, waiting to be fixed, fixed awaiting confirmation test or closed).
- Conclusions and recommendations.
- Global issues, such as other areas that may be affected by a change resulting from the incident.
- Change history, such as the sequence of actions taken by project team members with respect to the incident to isolate, repair and confirm it as fixed.
The IEEE829 contains the outline of a test incident report. Which are as:-
- Test incident report identifier: the unique identifier assigned to this test incident report.
- Summary: a summary of the incident, detailing where expected and actual results differ, identifying at a high level the items that are affected, and the steps leading up to the recognition of the incident e.g. if in test execution, which test was executed and the actual key strokes and data loaded.
- Incident description: a detailed description of the incident, which should include:
- Inputs. 2. Expected results 3: actual results 4: anomalies 5: date and time.
6: procedure step 7: environment 8: attempts to repeat 9: tester’s comments
10: Observer’s comments should also include any information regarding possible causes and solutions.
- Impact: if known, document what impact the incident has on progress.
Configuration Management
Configuration management is the process of managing products, facilities and processes by managing the information about them; including changes and ensuring they are what they are supposed to be in every case.
In testing configuration management will involve controlling both the versions of code to be tested and the documents used during the development process e.g. requirements design and plans.
Configuration management should ensure traceability throughout the test process e.g. a requirement should be a traceable through to the test cases that are run to test its level of quality and vice versa.
Effective configuration management is important for the test process as the contents of each release of software into a test environment must be understood and at the correct version, otherwise testers could end up wasting time because either they are testing an invalid release of the software or the release does not integrate successfully, leading to the failure of many test.
A good configuration management system will ensure that the testers can identify exactly what code they are testing, as well as have control over the test documentation such as test plans, test specification, defect logs etc.
Chapter 6:
What is a test tool?
A software product that supports one or more test activities, such as planning and control, specification, building initial files and data, test execution and test analysis.
It is a piece of software that is used to make the testing process more effective or efficient, meaning making testing easier, quicker, more accurate etc.
Benefits and Risks
Test tools need to be thought of as long-term investment that need maintenance to provide long-term benefits, mean upfront investment and ongoing maintenance and support can provide substantial income in return. Moreover cost needs to be less than the cost of performing testing activities without the tool if the investment is to be worthwhile.
Comparator Tool
It can be used virtually straight out of the box. A comparator can check whether one large test file is the same as another, if different report upon the difference but manually doing so is different.
In addition, incident management tools are fairly intuitive and easy for both experience and novice testers to use. They are likely to provide a ‘quick win’.
Other tools built in house by developers e.g. test harnesses, test oracle or test data preparation tools relatively easy for developers to produce with a good understanding of the tool management and the system and databases in the test environment.
Benefits
The amount of time and effort spent performing routine, mundane, repetitive tasks is greatly reduced. And a tester can spend this time to other key tasks, such as planning, analysis and design.
Using tools provide more predictable and consistent results as human failing such as manual keying errors, misunderstandings, incorrect assumptions, forgetfulness etc are eliminated.
The widespread use of databases to hold the data input, processed or captured by the test tool, means that it is generally much easier and quicker to obtain and present accurate test management information, such as test progress, incidents found/fixed etc.
Risks
Risks associated with test tools are:
- Over-optimistic expectations
- Lack of appreciation of effort required to implement and benefits that the tool can bring.
More over test environment is not designed and built with the test tools in mind, for a vendor to demonstrate the shortcomings it may encounter in a typical test environment.
Test Tools
Type of tool
There are several ways to classified then
- Fundamental test process (Primarily associated).
- Type of testing that supports
- Who use them?
In ISTQB foundation level test tools will be classified according to type of activity they support.
Tool support for management of Testing and Tests
Test Management Tools
- Provide various activities and tasks through out the development life cycle.
- Some activities to interface to other more specialist tools
E.g. Requirement Management tool, text execution tool, incident management tool and configuration management tool.
Test management tools provide a framework for creating, storing and editing test procedures.
These may be linked or traced to requirements, test condition and risks.
Such test procedures can be prioritized or grouped together and scheduled so that they are run in the most effective and efficient ways or order. Some test management tools allow decencies to be recorded so that test will fail owning to known defect can be highlighted and left unexecuted.
Tests can be recorded as passed or failed so that an incident be raised if the actual and expected results differ.
Test management tools generally ho;d data imn a database. This allows a large amount of reports and metrics to be produce. The metrics are as;
- Test and project management to control the current project.
- Estimated for future projects.
- Identifying weaknesses or inefficiencies in the development or test process that can be subsequently investigated with the aim of improving them.
Test management information’s reports should be designed to meet the needs of project managers and other key users.
It can also enable reuse of existing test ware in future test projects.
Incident Management Tools
It is also known as defect management tool. At the basic level provide two critical activities:-
- Creation of the incident report.
- Maintenance of details about the incident as it progresses through the incident life cycle.
Sometime lots of mandatory information is required in order to comply with industry or generic standard IEEE1044 to ensure agreed incident life cycle is strictly applied.
Incidents are only able to be assigned to particular teams or users.
It also use database to store and mange details of incidents. Incidents to be categorized according to values stored in appropriated fields. Such values change during the incident life cycle as incident analysed, debugged, fixed and confirmation tested. It also views the history for management information about incidents. And also search using SQL-type questions.
It can also be used in conjunction with data held in test management tools when planning and estimating for future projects. It also analysed to provide input to test process improvements projects. The field in database structure normally include:-
- Priority (H, M, L).
- Severity
- Assignee (to whom incident assign).
- Status in the incident life cycle (New, Open, Fixed, Reopen, Closed).
These tools can be integrated with test management, requirements management and or test execution tools which enable incidents to be input and related to test cases and requirements. It used to raise defects during regression testing.
Requirement Management Tools
It is used by business analyst to record, manage and prioritize requirements of a system and also used to manage changes to requirements.
If requirements change after tests have been written then test cases may also need t change then these changes issues to developers.
The tractability function within which this too enable links and references to be made between requirements, functions, test conditions and other test ware items. This mean, as requirements change, it is easy to identify which other items may need to change.
Requirement tools also enable requirements coverage metrics to be calculated easily as tractability enable test cases to be mapped to requirement.
As seen, traceability can create a lot of maintenance work but it does highlight those areas that are undergoing change.
Tester obtain requirement from the tool and begin to devise test conditions and test cases. Tractability within requirements tool will also highlight the test conditions affected by the changed requirement.
Configuration Management Tools
Mainly developed for management: the versions of different software (hardware) components that comprise a complete build of the systems; and various complete builds of systems that exit for various software platforms over a period of time;
What is build: a development activity where a complete system is compiled and linked (typically daily) so that a consistent system is available at any time including all latest changes?
Benefits of Configuration Management Tools
The benefits largely depend upon the following:
- Complexity of system architecture.
- Number and frequency of builds of the integrated systems.
- Number of choice available to customer (internal or external )
CM tools allow traceability between test ware and builds of an integrated system and versions of subsystems and modules. Traceability is useful for:
- Identification of correct versions of test procedure to be used.
- Determination of test procedures and test ware reused, updated and maintained.
- Assistance of the debugging process in case of failure of running a test procedure could be traced back to the appropriate version of a subsystem.
Tool support for static testing
Review Process Support Tools
These tools provide framework or inspections. This can include:
- Maintaining information: rules and checklists.
- Record, communicate and retain review comments and defects.
- Ability to amend and reissue the deliverable under review whilst retaining a history or log of the changes made.
- Traceability functions to enable changes to deliverable under review and other deliverable that may be affected by the change.
- Use web technology to provide access to this information’s any where.
This tool interfaces the configuration management tool to control version number of a document under review. It is good for technical/peer review but not for walk through and informal reviews. Management buy in if it benefits are long run.
Static Analysis Tool
It enables developers to analyse code before its execution or find defects early as possible. These tools generate lot of errors and warning message about the code. Used for medium to large changes good but not for small changes in code. Could also used for messages that are not relevant and finds defects that are hard to find during dynamic testing. Used to understanding code and to calculate complexity and other metrics. Some are integrated dynamic tools and coverage management tools and are language specific. E.g. code written in c needs to have static analysis tool that specific to c.
Types of defects that can be found using a static analysis tools can include:
- Syntax error e.g. spelling or missing punctuation
- Variance from programming standard e.g. too difficult to maintain.
- Invalid code structures. (Missing ENDIF statements).
- Modules or code may not be executed or unreachable code or invalid code dependencies may point to errors in code structure.
- Portability (e.g. code compiles on window but not on UNIX).
- Security vulnerabilities.
- Inconsistence interfaces between components (e.g. XML message produced by a component is not of the correct format to be read by component B).
- References to variables that have a null value or variables declared but never used.
Modelling Tools
It is used by developers for
- Analysis
- Design stages of development cycles.
- Benefits: cost effective at finding defects at early stage of development life cycle. Benefits similar to reviews and inspections.
- Allow omissions and inconsistencies to be identified and fined early so that detailed design and programming can begin from a consistent and robust model.
- Designers use UML to build a model of the software specification.
- It can map business processes to the system architecture model. This enable programmer and testers to have a better and common understanding of what program should do and testing is required.
States object models, databases can help to identify what testing is required and assist in checking whether tests cover all necessary transactions useful for complex system.
Tool support for test specification
Test design tools
- Use for creating test cases (For which test basis needs to be input and maintain).
- integrated with
- modeling tools
- requirement management tools
- static analysis tools
- Test management tools.
- Some tools allow a GUI model of the test basis to be created and then allow tests to be generated from this model.
- Some tool (Test frames) generated template from requirement specification held in narrative form.
- Test designed from state, object, database can be used to verify that model built correctly and used to derive test cases.
- Test oracle is a type of test design tool that automatically generate expected results.
- Test oracle replacement systems, migrations and regression testing.
- Test design tools useful for safety critical, high risk software when coverage level is higher and industry, defence or government standards need to be adhered to.
Test Data Preparation Tools
These are used by developers to manipulate data so that environment is in appropriate state for test to be run.
Tools Support for test execution and logging
Test Comparators
It compares the contents of files, data bases, XML messages, object and other electronic data formats. Also compare the actual result and expected results. Highlight difference and provide assistance to developers when localizing and debugging code.
Provide function to ignored specified section of file, screen or object or masked out. Used date or time stamped technique for this. It is useful for regression testing because the output or interface files should usually be the same. This is the onley benefits of this tool and included in the execution tools.
Test Execution Tools
This allows test scripts to run automatically or semi automatically.
Test Script: written in programming language or scripting language and used to navigate through system under test and to compare predefined expected outcomes with actual outcomes. At last result of test run written in test log. Test script can then be amended and reused to run different scenarios through the same system.
1. Record tools (capture playback): used to record test script and then play back exactly as it was executed. The recordings of tests are useful during exploratory testing for reproducing a defect or for documenting how to execute a test. These tools can be used to capture user actions so that navigation through a system can be recorded. In both cases the script can then be made more robust by the technical expert so that it handles valid system behaviours depending upon the inputs and the state of the system under test.
2. Data-driven testing: robust test script that deal with various inputs can be converted indo data driven test. This is where hard-coded inputs in the test script are replaced with variables that point to data in a data-table.
3. Key-driven testing: it is further enhancement of data driven testing or action word. Include extra column in the data table. The script read the keyword and takes the appropriate actions and subsequent path through the system under test. E.g. IF ELSE OR SELECT CASE statements are required in the test script for keyword-driven testing.
4. Technical skills: programming skills and standard are required to use the tool effectively. manual testing also relies on the robust and well driven test scripts that are easy to maintain
5. Maintenance: for maintenance of test script time and budget is required. And change to a system can mean that the test scripts need to be update. So relevant skills and knowledge is also required to do this.
6. Effective and Efficient Use: the following are benefits:
a: cost savings as a result of the time saved by running automated tests rather manual tests.
b: accuracy benefits from avoiding manual errors in execution and comparison.
c: the ability and flexibility to use skilled testers on more useful and interesting tasks than running repetitive manual tests.
d: the speed with which the results of the regression pack can be obtained. They are generally more useful for regression testing.
The automated test scripts to produce the amended documents would need to be analysed and updated as required. Automated scripts for new documents could be added to the regression pack after this release is complete.
At start the manual regression testing is cheaper but for long term benefits of automated regression testing is more beneficial and cost effective.
Test Harnesses
Used by developers and written in house by developers to perform component or integration testing for specific purpose. It is also used to test various objects ranging from a middleware system to a single or small group of components.
Easy to control and manage the small environment. Enable tester/developers to localize defects quicker and turn-around time to fix those defects.
It is based on the similar principle of data-driven testing using test execution tools, for test harness different kind of test cases to be designed and run without the time-consuming process of keying them manually.
The benefits of which one is best between the test execution tool and test harness depends upon the, task, environment, risk, purpose and level of the testing being performed.
Coverage Measurement Tools
It is used to measure the percentage of code structure covered across the white box measurement techniques such as statement coverage and branch or decision coverage. More over it is used to measure the modules and functions calls. It is often integrated with static and dynamic analysis tools and are primarily used by developers. It can measure the code written in any language but not all tools. Good for reporting coverage measurement and can therefore be used to assess test completion criteria and or exit criteria. Coverage measurement of requirements and test cases/scripts run can usually be obtained from test management tools. More over is used to instrumenting the code with extra statements, extra statement written into log and finally remove before execution goes into production. It is generally used on high risk and safety critical systems.
Security Tools
These tools are used to detect security threats, support execution of test procedures to confirm that different security issues handled (defect) i.e. viruses, denial of services and any other attacks.
They are useful for testing e-commerce, e-business and websites. A person of security tools expertise must be select for security like testing.
Tool support for performance and monitoring
Dynamic analysis tools
These are used to find defects that hard to find during static testing, used by developers, during component testing and component integration testing to:-
· Report on the state of software during its execution;
· Monitor the allocation, use and reallocation of memory.
· Identify memory leaks
· Detect time dependencies
· Identify unassigned pointers
· Check pointer arithmetic
Further more they are used for safety critical and high risk software where reliability is critical. Moreover these are integrated with static analysis and coverage measurement tools.
Developers used static analysis to find defects before component test execution. The integrated tool may allow;
- The code to be analysed statically
- The code to be instrumented
- The code to be executed (dynamically)
Dynamic analysis tool usually language specific. The tool could then;
- Report static defects
- Report dynamic defects
- Provide coverage measurement figures
- Report upon the code being (dynamically)\executed at various instrumentation points.
Performance testing/load testing/stress testing tools
Performance testing tools have been developed to carry out both loading and stress testing. It is difficult to do accurately and in a repeatable way without using test tools;
Load Testing
This testing reports upon the performance of a system under test, under various loads and usage patterns. A test driver can be used to simulate the load or usage pattern. Alternatively, response time or transaction times can be measured under various levels of usage by running automated repetitive test scripts via the user interface of the system under test. In both cases output will be written to a log. Reports or graphs can be generated from the contents of the log to monitor the level of performance.
Performance Testing
This tool is also be used for stress testing. In this case load on the system under test is increased gradually in order to identify the usage pattern or load at which the system under test fails e.g. if an air traffic control system support 200 concurrent aircraft in the defined air space , the entry of the 201st or 205th aircraft should not cause the whole system to fail. It can also be used against the whole systems but they can also be used during system integration test to test an integrated group of systems, one or more servers, one or more databases or a whole environment.
If the risk analysis finds that the likelihood of performance degradation is low then it is likely that no performance testing will be carried out. E.g. a small enhancement in the existing mainframe system not necessary for formal performance testing, For this normal manual testing may be considered sufficient.
There are similarities between performance testing tools and test execution tools in that they both use test scripts and data driven testing.
Performance testing tools can find the defects such as;
- General performance problems
- Performance bottlenecks
- Memory leakage (e.g. if system running under heavy load for some time).
- Record-tracking problems
- Concurrency problems
- Excess usage of system resources
- Exhaustion of disk space
The cost of some performance tools is high and the implementation and training costs are also high.
Monitoring Tools
These tools are used to check whether;
· Systems are available
· Their performance is acceptable
· Used for monitoring e-commerce, e-business, websites because their affects consequences are severe.
· Less important for internal system because due to failure, in organisation there is contingency plan to resolve it.
It is not actually a testing tool but good for performance testing.
Other Tools
Not assigned for testers/developers but could be used to support one or more test activities;
- Spreadsheets
- Word processors
- E-mail
- Backup and restore utilities
- SQL and other database query tools
- Project planning tools
- Debugging tools
In absence of test management and incident management tools, spreadsheets are normally used for recording, tracking and maintaining. Test pass or fail could be recorded on spreadsheets.
- Spreadsheets- for producing decision tables, different test scenario required, manipulate test management information.
- Word-Processor- writing test strategies, plans, weekly reports and test deliverable.
- E-mail- communication with developers about defects, distributing reports and other deliverables.
- Back-up and restore utilities- restore a consistent set of a data into the test environment for regression testing.
- SQL- for analyzing data to obtain actual or expected results.
- Project- estimating resources, timescales, monitor progress.
- Debugging-by developers to localize and fix defects.
Introducing a tool into an organisation