The Art of Testing
I have worked for 5+ years as a tester,
and, if I do say so myself, have gotten quite good at it. I have developed a
general philosophy of testing that I believe drives my success. Below is this
First of all, so many test engineers
seem embarrassed by their profession. Instead of calling themselves a “test
engineer,” depending on what kind of testing they do, they may call themselves
a “computer engineer” or a “systems engineer” or one person called himself a
“design (!?) engineer.” While there is systems engineering involved in testing
(if you do it right anyway), the term “systems engineer” is ambiguous. A
systems engineer could deal with computer systems, cars, or even the “system”
of the human body. I have no problem stating what I am. I am a TEST engineer.
Specifically I am a software test engineer. I have no problem with it; I am
proud of the work I do and strive to do the best possible job I can. If I do my
job correctly, I am absolutely integral to the success of the product.
Personally, I like testing. Testing is
when the rubber hits the road. When engineers and developers design a product,
they often sit in a room, void from any reality, and debate fiercely about how
their idea will surely work. As a tester, I don’t have to debate with
anybody over if something works. I can say, “Your idea sounds nice. Now let me
test it out.” I take it, test it, and let reality talk for me.
There are of course problems inherit to
testing, like there are probably anything else. Testing is at the end of the
product cycle, which usually goes: design, development, test. Whatever
technical or budget problems there were at earlier stages in the cycle will
only snowball when they get to test. If the product was poorly designed, poorly
documented, or budgets were already blown, it is felt that much more heavily by
test than anyone else.
Perhaps the hardest part about being a
test engineer is you are the person who delivers bad news to people. No matter
how rational people are or how much it is recognized that finding problems
in-house is better than on-site, people do not like hearing this bad news. When
I would tell development about problems, they would sometimes play “You’re a
mean one, Mr. Grinch” for me. I would calmly wait for the song to be over,
smile, then give them a detailed report on a problem.
It requires a certain personality to be
an effective tester. I was in a meeting once and the head of development said,
“We think the product is almost ready. Even Amber agrees.” The room
erupted in laughter. The head of quality assurance said, “Now that’s saying
something!” Apparently that did not happen a lot, LOL.
Probably the most political part about
testing is that it is often the target for manipulation. A tester gives a pass
or fail status. Certainly there are many people who want the passing status
regardless if it is earned. A good tester must be an honest broker. It is
not a job for pushovers.
Ultimately, the tester is responsible
for quality. It is a rewarding feeling to be handed a bad product, work
diligently on testing it, then see the product become better. Products need not
be confusing to install, inefficient, or buggy. It is possible to make usable,
intelligible, foolproof, working products. The use and operation of products
should not be a headache. If you have good testing, you will have a good
I am absolutely opposed to anyone who
says “testing does not add value to a product but is necessary.” As a tester,
my goal is to indeed add value / alter the product. When I write a report, I
expect the product to be altered. My hands may not touch the software code but
my effort in finding and hunting down what are often complicated problems
certainly lead to the improvement of the product. If my efforts don’t lead to
altering the product somehow, then what exactly is it that I am doing? Testing
is more than just handing out a grade.
You will always hear me defending the
need for testing. Any product or service can be tested. Testing brings objectivity
to assessing the quality of a product. I consider testing to bring “eyes” to a
product. I imagine it like an inspector in a manufacturing shop looking at a
piece through a magnifying glass. He knows every important measurement about
that piece and if it passes or fails. There is often an attitude that testing
is beneath people; that some products or skills are just so sophisticated they
are not subject to objective metrics. This attitude is adopted, for instance,
when assessing students with standardized tests. This is nothing more than
arrogance and unwarranted ego. I face this very attitude over even technical
products. “We don’t need a test for that it will surely work.” This is
the bane of my existence and I laugh at it! Every single time someone has said
this, there has been a problem with the product. Every single time. I’ve never
made a test that didn’t end up being valuable. I think people are opposed to
testing as they don’t want “eyes” on them for what those eyes will find. If you
want a quality product, you must have testing. Period.
My specialty is in in-house testing:
testing a product before it is delivered on-site, before even on-site testing.
It is my goal to find every problem before the product goes on-site.
While I will never outright promise perfection right off of the bat, as there
are often if not always blind spots, it is still my goal.
I have watched speeches delivered by
testers at popular software companies that make fairly non-critical products
like email for regular commercial use. Their philosophy is that testing is a
waste of money and to let customers, the best tester possible, report problems.
That may work for products such as simple email. But if you work on a product
that is more critical to get right the first time, say perhaps medical
software, this is a losing philosophy. I always tell new testers, “Imagine you
are the poor shmuck on site who has to explain to a customer why the product is
not working.” I encourage them to test thoroughly in-house. On-site testing is expensive
and difficult. You are often without all of the tools you need; if something
goes wrong, it is a pain to gather all information and tools you need to fix
it; and it is more difficult to communicate with engineers or developers from
on site. Ever since adopting the philosophies described in this document, I
have come close to delivering quality products for the first on-site test most
times, and if not, by the second time. Whenever the product does not pass the
first time around, I learn something, and update my philosophy. The following
are my principles that have helped achieve this happy end.
Test in the Operational Configuration
The most important advice I can ever
give to a tester is always test in the operational configuration. Two
things: First, you will sometimes be faced with the question, “Should we test
in this configuration or that one?” The answer is: which one will be used at
the customer site? That is how you should test. Second: Think of all situations
that will be encountered on the customer site. If they are not mimicked in your
test lab with the tools you already have, think of a way, if possible, to mimic
the situation in-house.
I will tell the very stories that led me
to this philosophy. We were designing performance tests as directed. We were
unsure if the performance test should include a new filter or not. (Old
performance tests did not have it.) The answer we got was that we should not
test with the filter because “then more data would flow” and this would be a
“more difficult” test.
So, that is how we tested. We tested the
filter itself, exclusively, by testing if small amounts of data passed or
didn’t pass as expected but never with the filter on and a full-fledged
scenario with all data flowing.
When we got on site, which naturally had
the filter on, we found that the filter was unexpectedly filtering out data
unrelated to what the filter was supposed to filter out. The filter, which was
only supposed to stop data going one way, was also inadvertently stopping data
going the other way. Data not flowing at all is a critical error! From then on,
it was adopted, by people involved on that project anyway, that there should be
one test with all features turned on and all data flowing, as close to the
operational configuration as possible.
Further, the tools you have in-house to
test are usually inferior models of what will be encountered on site. Sometimes
they do not produce all data that would be encountered on site. If this is the
case, try to think of a way to generate the data and include it in the “every
bell and whistle” test.
I came to this conclusion after using an
in-house model to test particular data. It did not produce all data necessary.
We did generate the data but only sent small packets of data to see if it passed
correctly. But on site, when large amounts of this data were generated, it
produced a “memory leak.” This is why I advocate to understand the on-site
setup and mimic it as best as possible in-house with one of your tests having
every feature possible included.
Performance testing is one of the most
misunderstood aspects of testing thus it deserves its own section. The usual
thought process is, “Pound the [object under test] with a lot of stress.” But the
correct performance test should be, “Pound the [object under test] with a
variety of stress.”
A very well-known performance test can
be used to explain this. There is a manufacturer of toilets that markets that
toilet as being able to flush a bucket of golf balls. It is a wildly popular
toilet. The problem is that what a toilet flushes is not comparable to a bucket
of golf balls. A better test, mimicking what actually gets flushed, is flushing
a bucket of gooey wadded up tissue paper. Golf balls are smooth, firm and
probably just flush in a single line. Any toilet could probably flush a million
with no problem, save the problem of actual physical space. Plumbers often
complain about these toilets; they are not, in actual operation, any better.
Indeed, it does not test in the actual
operational configuration. Further, for this performance test, it only tests
one type of stress. If a particular type of stress produces no wear on the
object under test, you can pound the object a million times with the stress and
it would continue to perform well. Zero times one is zero. Zero times one
million is still 0.
Instead, consider every type of stress
the object may actually go under. Test all of them. If testing software, try to
test data such that every piece of software code is executed. Indeed, in the
above example, where the performance of the product was tested with a filter
on, it is just as likely that there would be a performance problem with the
filter on as off, as it goes down different code. If you see a weakness, narrow
down which type of stress caused it. Once you identify the stress, then indeed,
more of that stress will cause more wear. But otherwise, something that
produces zero wear with just a little stress will produce zero wear with a lot
Test the Differences that Matter
Designing tests for complicated features
is somewhat of an art and, I have found, cannot safely be handed off to a new
tester. The most sophisticated skill to develop is to know what variables are
different enough from each other such that a separate test is warranted.
Imagine you were testing cars. To test
for speed, it would hardly matter if the cars were blue, black, or red. But if
you wanted to test how well the cars stand up to the sun, perhaps the color of
paint would matter.
I have seen way too many tests that test
each “different color car.” To eliminate this, if testing software, a good rule
of thumb may be, “Will the data in the two different tests I am proposing go
through different code?” If the answer is yes, then two different tests are
Tests for complicated features are
probably going to require a matrix filled in with every single logical
condition possible. Take the time to understand the design of the product.
Plotting it out like this will help select what tests are necessary and what
ones are not.
Requirements Should Drive Testing
To ensure comprehensive testing,
requirements should drive testing. A good process, generically described, has
specific requirements, all enumerated, from customer requests. Not only does
each requirement need a solution developed for it, each requirement needs a
test. A good tracking system should be in place to ensure every requirement has
a test. The test team should be involved in the product cycle from the very
beginning when requirements are generated. Testing should not be an
afterthought. This ensures that the test team has an accurate understanding of
the requirements and can begin to make tests as soon as possible.
When creating test procedures and test
scripts, they should reference back to the requirement under test. This creates
traceability. If there is ever a question on why a test is created or what its
“intent” is, with good documentation like this, the requirement can be
referenced. It can be decided if the requirement is still relevant if the test
is under question to be deprecated.
Also, if there is ever a problem report
written about the product, a test should be developed to test that problem. It
should be added to the suite of tests such that every new version of the
product is tested for it. In the test procedure or test script developed for
it, the problem report number should be referenced in the same way that
requirements are referenced. Before closing the problem report, it should be
verified both that the problem was fixed and that a test was developed.
I always insist on being involved in the
requirements generation process. I make sure the requirements are written
intelligently as to be testable, don’t contradict each other, among other
things. As noted above, testing is at the end of the cycle and the end gets
dumped with all of the problems from earlier in the cycle. Inserting your voice
early in the cycle can alleviate some of those problems.
Comprehensive Testing / Standardized
In addition to testing every
customer-derived requirement, all features and functions should be tested in a
thorough manner, even if not an explicit requirement. In my opinion, that all
features or functions of a product work is always an implicit requirement. This
should be a standard requirement for all products. Also, certain situations
that have caused problems in the past should become standard requirements. This
latter list will be built with experience about the product.
For instance, if testing a Graphical
User Interface (GUI), to test it thoroughly, every button from left to right,
top to bottom, of the feature should be tested. Or, if you have a piece of
software that has different configuration options, every single configuration
option should be tested.
Further, through experience, I have
found certain situations should be tested as problems may arise. For instance,
stop your product in the middle of use then start it again. Everything should
return to a normal state of operation. Any product under test will have
different “standard” requirements necessary. I encourage standard requirements
with every project, built based off wisdom from past projects. Here are some
generic recommended standard requirements:
performance problems after product is put under stress
configuration options work
configuration options are documented (tester should verify this and ensure
accuracy of document)
can be restarted
does not break or “crash” during use
of product is smooth and all sub-products necessary for installation are
included (or documentation states need for sub-product)
statements are intelligible and meaningful
spelling errors on labels for product
formatting throughout all of product
When reporting a problem, the problem
should be described as specifically as possible. You should always know the
version number of every product you are working with and put them in the
report. You should know as much about everything as possible. Do not take a
single thing for granted. Effort should be made to narrow the problem down to
as simple of circumstances as possible.
One problem should get one report. It is
sometimes argued that 3 or 4 symptoms are actually the same problem and should
only get one report. That is merely theory until it is tested. All symptoms
should have their own report so that all symptoms are verified before closing
the problem report. If there is more than one symptom in one problem report,
extra symptoms usually get ignored.
The tracking software used to organize
problem reports should be sophisticated enough to group them based on what test
and/or project the problem is associated with. A query can then find all
problems associated with test procedure X or project Y.
Problems should always be documented.
This is the only way they can be effectively tracked. There is no way that an
undocumented problem—and rationalizations abound for why problems should not be
documented—will be tracked properly. When problems are documented, there is an
easy list that can be referenced of all problems tied to a product or project.
In a report, the problem report ID numbers need only be referenced, otherwise a
book is written in the report. It was sometimes said where I work, “A problem
should not be documented until the solution for it is known.” If the solution
was known, it would not be a problem! Some investigation should be done to
narrow a problem down, engineers and developers should be notified before
writing a problem report (to ensure it is not user error or working as indeed
expected), but once it is agreed upon that it is a problem, it should be
documented. Even if more analysis needs done (and testers or developers can do
the analysis) it should still be documented. At least the problem is noted. Otherwise,
if not noted, inevitably it is forgotten about, and whatever work had been done
on it will vanish in the air.
Test the Tools You Test With
If you buy an off the shelf tool to test
with, you may perhaps assume that it has been tested and works as expected. But
if you build tools to test with in-house, they should be tested. These tests
should produce false results—the tool is capable of accurately failing tests
that should fail. If you don’t test the tools you test with, it is similar to
stepping on an uncalibrated scale. It will cause a lot of unnecessary
Over Testing is Under Testing
There is often a sentiment that “just in
case” the product should be over tested. For instance, test it longer than
necessary, test variables that aren’t that much different, etc. There is always
a cost to doing anything and, in my opinion, when you over test, you under
test. The main problem is that by over testing, you are taking up precious
time. When a test is made, it should be run for most product releases. That
test, locked in to routine tests, will get run over and over. This can get very
bloated. For instance, we had an automated suite of tests that ballooned to
taking 14 hours to complete. This meant that it could not be run
overnight—indeed, meant we couldn’t test it regularly, which meant under
I also believe that, when possible,
tests should be fit into one test procedure instead of several. Every test
procedure needs tracked, there is always set up and take down time, and each
test procedure needs a report. Instead, when you have brought up all systems,
do all the testing with it up, then take it all down and write one report. Most
tests do not take long as to justify dividing tests up for time reasons. A
reason to divide test procedures is if they have different set ups.
Also, as noted above, if you test in
anything but the operational configuration “to make it a more difficult test,”
you are then not testing in a particular configuration and losing some value in
“What’s different?” (Systems
Testing is more than just running a test
and then giving a pass or fail status. A monkey could probably be trained to do
this. If a test fails, a good tester will take it upon themselves to
investigate further as to why it is failing. This generates much intelligence
to give to development in solving the problem.
The most important question a tester
should learn to ask is, “What’s different?” If two different tests produce two
different results, it is not some wild mystery, something is different.
For instance, a test passed in-house but
not on-site once. After much investigation, it came down to a different version
of the operating system that was used. I tell this example to point out that
the seeming smallest of variables do indeed matter.
If you do have different tests that
produce different problems, don’t see it as a frustrating mystery, but a very
valuable investigation tool. If you find a problem and you only have one system
to work on, you have no point of reference to work off of to narrow down the
problem. But if a test on a different system produces results that pass, you
are sitting on a gold mine. The goal is to compare the two systems until you
find what is different. You greatly narrow down what the potential problem is.
Reduce Complexity (Systems
Too often when a problem arises, a new
engineer on my team will have all data up, all features on, and will be trying
to solve the problem like this. Perhaps it is inexperience or perhaps it is
ego, but they think they can solve a problem while standing on one foot,
juggling bowling pins, with an eye patch over their eye.
I am always telling them to reduce the
complexity. Turn off all data and features such that you have the simplest
scenario possible to recreate the problem.
One method to do this, if having trouble
narrowing down the problem, is to turn off half of the things you have on. Does
the problem still exist? If it went away, turn off everything currently on, and
turn on the stuff you just took down. You just narrowed down the probably to ½
of the previous suspects. Don’t have anything extraneous on that does not need
to be on.
Or, turn off every feature and return
the product to its most simple state possible. Start turning on features to see
how it reacts. Go to a known state of things then start deviating from that.
I think of it as building a tower.
First, you build the first floor. You build it, bang on it, make sure it works.
Then build the next one, bang on it, make sure it works. You do this until it
All systems are built from smaller
parts. If you reduce the system down to the relevant parts, it is easier to
deal with. The ability to solve most problems, I have found, no matter how
complex they seem, come down to understanding a few simple things. There is
virtually no problem, even in a technical world, that is unsolvable by a person
willing to think.
Have Something to Show for Your Work
This is a general philosophy of mine and
also one that applies to testing. If ever I do work, I want there to be some
object at the end of the day that shows what work I did. If I learned something
new, I want a document with either my notes or a full blown documentation of
what I learned. If I worked on some test scripts, I want to put them on a
shared server so that my work is seen and can be used by other people. If I
found a problem, I want a problem report written. Otherwise, if I vanished one
night, it would be as if I had done nothing. I casually call it
"stamping" my work but "stamping" is not the best word for
Similarly, as a test engineer, your work
should “touch” the product. You probably won’t write the code or engineer the
product yourself. But when you test it, you alter the product. If you aren’t
altering the product, you aren’t doing any work.
Testing, though often unappreciated, is
vital to the quality of a product. Testing should not be considered a necessary
evil but integral to the success of a company. Invest in good testing and you
invest in a good product.
July 22, 2012