Software testing used to be one of the most underrated area in software development process. Indeed, it is not even covered at the undergraduate level at university (thanks BCS). This article provides an introduction into software testing. It covers: what software testing is, types of testing as well as what and when you should test? Let's start with a simple 'Hello Name' program:
|
1 2 3 4 5 6 7 8 9 10 11 |
import java.util.Scanner; public final class HelloName { public static void main(String[] args) { Scanner sc = new Scanner(System.in); System.out.print("Please enter your name:\t"); System.out.println("Hello " + sc.nextLong() + "!"); } } |
There are various approaches one could take to verify the 'correctness' of the program.
- Compiler checks syntax
- Certain common problems can be detected using code analysers e.g. Lint4j
- Hire an experienced test consultant to do (manual) exploratory testing
- Run with exhaustive input strings up to e.g. 128 chars
- Boundary cases e.g. empty string, max length string
- Special inputs e.g. ctrl-C, ctrl-D, ctrl-Z
- 6-person multidisciplinary team allocated two days to do Fagan inspection
- Java Pathfinder exhaustively explores space of possible executions
- Screw it all: find an academic collaborator and apply for funds to develop a general theory of a hello World correctness!
We can clearly see, that this is getting a little bit out of hand. Obviously, "no testing at all is bad", so how far do we need to go before it is too much?
What is software testing?
Software engineering is a relatively new field and it seems that certain definitions are not yet in place e.g. software testing is not particularly well defined. The best definiton of testing goes something along the lines of:
Testing consists of evaluating software in order to derive an estimate of whether or not it meets some criteria
It still leaves a lot of unanswered questions: What is evaluated? When is it evaluated? How is it evaluated? What criteria must it meet? How do we know if it meets the criteria?
In addition, testing itself has quite a lot of limitations. Dijkstra famously said:
Program testing can be used to show the presence of bugs but never to show their absence!
Testing can:
- show software works and (perhaps) sometimes does what we want
- show software doesn't work i.e. it doesn't do what it is supposed to
- reduce the chance that software doesn't work i.e. a bit more confidence that key properties hold
Types of Testing
The world is littered with examples of poor testing: Ariana 5 and Mars Polar Lander for a start. Clearly, software can go horribly wrong and needs to be tested. There are different kinds of tests and they reveal different kind of bugs.
Code evaluation
The most basic distinction we can make is between dynamic testing and static testing: the first one is done by executing the code and the latter without executing the code.
Code Visibility
Both of them assume visibility of the code, but this is not always the case. If the code is visible, this kind of testing is referred to as white-box testing.
Alternatively, we could consider testing by exploiting the knowledge of the system to design tests e.g. closed source COTS or you are building tests for many possible implementations.
Finally, there is black-box testing where you have no knowledge of the software's implementation.
Meeting The Specification
We can also categorise tests according to specification requirements which divided between functional and non-functional. The functional testing checks whether the program provides some specific functionality whilst non-functional testing assesses the program on all other aspects such as dependability, usability etc.
Sir, we have a failure...
In QA, there are 3 key words which describe the problems with software:
- Fault
- Error
- Failure
A failure occurs when the software does something observably bad e.g. crashes with a NullPointerException. This usually occurs because the system got in to a bad state - an error e.g. the string containing the program's data file name is null. Errors are caused by faults which are bugs in the program that can cause a failure to occur e.g. program starts with a configuration which doesn't set the data file name.
What and when should you test?
Option 1 - Unit Testing
The first option is to start with the smallest meaningful unit and test it:
|
1 2 3 |
int increment(int i) { return i + 1; } |
Tested by:
|
1 2 3 4 5 |
@Test public void test_increment() { int j = 0; assertEquals(1, increment(j)); } |
This is commonly known as Unit Testing. Unit testing allows very precisely identify errors in the software and tend to be easily implementable by the programmer (if one knows what they are doing).
In Java, C# and other OOP, we tend to unit test methods and (occasionally) classes, but there isn't a precise definition as to what exactly the 'unit' is.
Many problems are not found at the unit level as they arise because of communication from interaction between units i.e. one unit has an error and passes that error across to another.
Option 2 - System Tests
In addition to the unit tests, we can test the system as a whole - these are known as system tests. System tests are typically black box tests.
There is a number of advantages of using system tests. They can potentially find any bug with regards to your design intent. They can also test both functional and non-functional requirements. Especially, the non-functional tests are really hard (if not impossible) to pin down on unit level e.g. performance.
They can also be used to understand non-explicit requirements - some behaviour of the system may obviously wrong. Tests on this level can usually be performed by a specialist tester team.
In system testing, even if a bug is found, it might be difficult to pin down.
Option 3 - Integration Tests
The 3 and the final level are integration tests. They test interacting classes, modules and/or subsystems. It usually involves grouping together some units and testing them together using a black-box approach.
Integration tests are sort of the middle ground between system tests and unit tests. They tend to be considerably more traceable than system tests and can be performed by engineers who are quite close to the components being integrated.
Approaches
There are 3 main kind of approaches to integration tests:
- Big Bang - put everything together than test!
- Top-down - Modules tested from the entry points and integrated progressively
- Bottom-up - Modules are progressively integrated and tested from the most elementary ones
So far so good, but how do we convince our customer that those tests are meaningful for them?
Option 4 - Acceptance Testing
They are the ones designed by the domain expert (someone who knows the stuff). Their main intent is not to discover failing scenarios, but rather to discover that the product will work (and how well) in a production environment. This kind of testing is performed by potential users usually before the software is built (e.g. paper prototypes).
The advantages of using acceptance testing are pretty obvious: they verify validity of the product. On a more 'human' level, they help build better relationship with the client as well as better understand the customers, domain and the market.
Regression Testing
Regression testing is used to ensure that software still works after new features have been added or changes have been made. This is where test automation kicks in. All automated tests become a regression test suit which apply to any cope (unit, inegration, system ...).
Example
This is our initial code:
|
1 2 3 4 5 6 7 8 9 |
int increment(int i) { return i + 1; } @Test public void test_increment() { int j = 0; assertTrue(increment(j) == 1); } |
Now this code changes to the following:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
int increment(int i) throws Exception { if (i < Integer.MAX_VALUE) { return i+1; } else { throw new ArithmeticException(); } } @Test public void test_increment() { int j = 0; assertTrue(increment(j) == 1); } |