"Testing brings eyes to a product." Amber Pawlik



Amber Pawlik

Amber's Blog

Facebook Page

Amber Pawlik Books

Islam on Trial

Objectivist Sexuality

On Demand Side Economics

Didactic Method to Teach Economics

The Art of Testing

I have worked for 5+ years as a tester, and, if I do say so myself, have gotten quite good at it. I have developed a general philosophy of testing that I believe drives my success. Below is this philosophy.

First of all, so many test engineers seem embarrassed by their profession. Instead of calling themselves a “test engineer,” depending on what kind of testing they do, they may call themselves a “computer engineer” or a “systems engineer” or one person called himself a “design (!?) engineer.” While there is systems engineering involved in testing (if you do it right anyway), the term “systems engineer” is ambiguous. A systems engineer could deal with computer systems, cars, or even the “system” of the human body. I have no problem stating what I am. I am a TEST engineer. Specifically I am a software test engineer. I have no problem with it; I am proud of the work I do and strive to do the best possible job I can. If I do my job correctly, I am absolutely integral to the success of the product.

Personally, I like testing. Testing is when the rubber hits the road. When engineers and developers design a product, they often sit in a room, void from any reality, and debate fiercely about how their idea will surely work.  As a tester, I don’t have to debate with anybody over if something works. I can say, “Your idea sounds nice. Now let me test it out.” I take it, test it, and let reality talk for me.

There are of course problems inherit to testing, like there are probably anything else. Testing is at the end of the product cycle, which usually goes: design, development, test. Whatever technical or budget problems there were at earlier stages in the cycle will only snowball when they get to test. If the product was poorly designed, poorly documented, or budgets were already blown, it is felt that much more heavily by test than anyone else.

Perhaps the hardest part about being a test engineer is you are the person who delivers bad news to people. No matter how rational people are or how much it is recognized that finding problems in-house is better than on-site, people do not like hearing this bad news. When I would tell development about problems, they would sometimes play “You’re a mean one, Mr. Grinch” for me. I would calmly wait for the song to be over, smile, then give them a detailed report on a problem.

It requires a certain personality to be an effective tester. I was in a meeting once and the head of development said, “We think the product is almost ready. Even Amber agrees.” The room erupted in laughter. The head of quality assurance said, “Now that’s saying something!” Apparently that did not happen a lot, LOL.

Probably the most political part about testing is that it is often the target for manipulation. A tester gives a pass or fail status. Certainly there are many people who want the passing status regardless if it is earned.  A good tester must be an honest broker. It is not a job for pushovers.

Ultimately, the tester is responsible for quality. It is a rewarding feeling to be handed a bad product, work diligently on testing it, then see the product become better. Products need not be confusing to install, inefficient, or buggy. It is possible to make usable, intelligible, foolproof, working products. The use and operation of products should not be a headache. If you have good testing, you will have a good product.

I am absolutely opposed to anyone who says “testing does not add value to a product but is necessary.” As a tester, my goal is to indeed add value / alter the product. When I write a report, I expect the product to be altered. My hands may not touch the software code but my effort in finding and hunting down what are often complicated problems certainly lead to the improvement of the product. If my efforts don’t lead to altering the product somehow, then what exactly is it that I am doing? Testing is more than just handing out a grade.

You will always hear me defending the need for testing. Any product or service can be tested. Testing brings objectivity to assessing the quality of a product. I consider testing to bring “eyes” to a product. I imagine it like an inspector in a manufacturing shop looking at a piece through a magnifying glass. He knows every important measurement about that piece and if it passes or fails. There is often an attitude that testing is beneath people; that some products or skills are just so sophisticated they are not subject to objective metrics. This attitude is adopted, for instance, when assessing students with standardized tests. This is nothing more than arrogance and unwarranted ego. I face this very attitude over even technical products. “We don’t need a test for that it will surely work.” This is the bane of my existence and I laugh at it! Every single time someone has said this, there has been a problem with the product. Every single time. I’ve never made a test that didn’t end up being valuable. I think people are opposed to testing as they don’t want “eyes” on them for what those eyes will find. If you want a quality product, you must have testing. Period.

My specialty is in in-house testing: testing a product before it is delivered on-site, before even on-site testing. It is my goal to find every problem before the product goes on-site. While I will never outright promise perfection right off of the bat, as there are often if not always blind spots, it is still my goal.

I have watched speeches delivered by testers at popular software companies that make fairly non-critical products like email for regular commercial use. Their philosophy is that testing is a waste of money and to let customers, the best tester possible, report problems. That may work for products such as simple email. But if you work on a product that is more critical to get right the first time, say perhaps medical software, this is a losing philosophy. I always tell new testers, “Imagine you are the poor shmuck on site who has to explain to a customer why the product is not working.” I encourage them to test thoroughly in-house. On-site testing is expensive and difficult. You are often without all of the tools you need; if something goes wrong, it is a pain to gather all information and tools you need to fix it; and it is more difficult to communicate with engineers or developers from on site. Ever since adopting the philosophies described in this document, I have come close to delivering quality products for the first on-site test most times, and if not, by the second time. Whenever the product does not pass the first time around, I learn something, and update my philosophy. The following are my principles that have helped achieve this happy end.

Test in the Operational Configuration

The most important advice I can ever give to a tester is always test in the operational configuration. Two things: First, you will sometimes be faced with the question, “Should we test in this configuration or that one?” The answer is: which one will be used at the customer site? That is how you should test. Second: Think of all situations that will be encountered on the customer site. If they are not mimicked in your test lab with the tools you already have, think of a way, if possible, to mimic the situation in-house.

I will tell the very stories that led me to this philosophy. We were designing performance tests as directed. We were unsure if the performance test should include a new filter or not. (Old performance tests did not have it.) The answer we got was that we should not test with the filter because “then more data would flow” and this would be a “more difficult” test.

So, that is how we tested. We tested the filter itself, exclusively, by testing if small amounts of data passed or didn’t pass as expected but never with the filter on and a full-fledged scenario with all data flowing.

When we got on site, which naturally had the filter on, we found that the filter was unexpectedly filtering out data unrelated to what the filter was supposed to filter out. The filter, which was only supposed to stop data going one way, was also inadvertently stopping data going the other way. Data not flowing at all is a critical error! From then on, it was adopted, by people involved on that project anyway, that there should be one test with all features turned on and all data flowing, as close to the operational configuration as possible. 

Further, the tools you have in-house to test are usually inferior models of what will be encountered on site. Sometimes they do not produce all data that would be encountered on site. If this is the case, try to think of a way to generate the data and include it in the “every bell and whistle” test.

I came to this conclusion after using an in-house model to test particular data. It did not produce all data necessary. We did generate the data but only sent small packets of data to see if it passed correctly. But on site, when large amounts of this data were generated, it produced a “memory leak.” This is why I advocate to understand the on-site setup and mimic it as best as possible in-house with one of your tests having every feature possible included.

Performance Testing

Performance testing is one of the most misunderstood aspects of testing thus it deserves its own section. The usual thought process is, “Pound the [object under test] with a lot of stress.” But the correct performance test should be, “Pound the [object under test] with a variety of stress.”

A very well-known performance test can be used to explain this. There is a manufacturer of toilets that markets that toilet as being able to flush a bucket of golf balls. It is a wildly popular toilet. The problem is that what a toilet flushes is not comparable to a bucket of golf balls. A better test, mimicking what actually gets flushed, is flushing a bucket of gooey wadded up tissue paper. Golf balls are smooth, firm and probably just flush in a single line. Any toilet could probably flush a million with no problem, save the problem of actual physical space. Plumbers often complain about these toilets; they are not, in actual operation, any better.

Indeed, it does not test in the actual operational configuration. Further, for this performance test, it only tests one type of stress. If a particular type of stress produces no wear on the object under test, you can pound the object a million times with the stress and it would continue to perform well. Zero times one is zero. Zero times one million is still 0.

Instead, consider every type of stress the object may actually go under. Test all of them. If testing software, try to test data such that every piece of software code is executed. Indeed, in the above example, where the performance of the product was tested with a filter on, it is just as likely that there would be a performance problem with the filter on as off, as it goes down different code. If you see a weakness, narrow down which type of stress caused it. Once you identify the stress, then indeed, more of that stress will cause more wear. But otherwise, something that produces zero wear with just a little stress will produce zero wear with a lot of stress.

Test the Differences that Matter

Designing tests for complicated features is somewhat of an art and, I have found, cannot safely be handed off to a new tester. The most sophisticated skill to develop is to know what variables are different enough from each other such that a separate test is warranted.

Imagine you were testing cars. To test for speed, it would hardly matter if the cars were blue, black, or red. But if you wanted to test how well the cars stand up to the sun, perhaps the color of paint would matter.

I have seen way too many tests that test each “different color car.” To eliminate this, if testing software, a good rule of thumb may be, “Will the data in the two different tests I am proposing go through different code?” If the answer is yes, then two different tests are warranted. 

Tests for complicated features are probably going to require a matrix filled in with every single logical condition possible. Take the time to understand the design of the product. Plotting it out like this will help select what tests are necessary and what ones are not.

Requirements Should Drive Testing

To ensure comprehensive testing, requirements should drive testing. A good process, generically described, has specific requirements, all enumerated, from customer requests. Not only does each requirement need a solution developed for it, each requirement needs a test. A good tracking system should be in place to ensure every requirement has a test. The test team should be involved in the product cycle from the very beginning when requirements are generated. Testing should not be an afterthought. This ensures that the test team has an accurate understanding of the requirements and can begin to make tests as soon as possible.

When creating test procedures and test scripts, they should reference back to the requirement under test. This creates traceability. If there is ever a question on why a test is created or what its “intent” is, with good documentation like this, the requirement can be referenced. It can be decided if the requirement is still relevant if the test is under question to be deprecated.

Also, if there is ever a problem report written about the product, a test should be developed to test that problem. It should be added to the suite of tests such that every new version of the product is tested for it. In the test procedure or test script developed for it, the problem report number should be referenced in the same way that requirements are referenced. Before closing the problem report, it should be verified both that the problem was fixed and that a test was developed. 

I always insist on being involved in the requirements generation process. I make sure the requirements are written intelligently as to be testable, don’t contradict each other, among other things. As noted above, testing is at the end of the cycle and the end gets dumped with all of the problems from earlier in the cycle. Inserting your voice early in the cycle can alleviate some of those problems.

Comprehensive Testing / Standardized Requirements

In addition to testing every customer-derived requirement, all features and functions should be tested in a thorough manner, even if not an explicit requirement. In my opinion, that all features or functions of a product work is always an implicit requirement. This should be a standard requirement for all products. Also, certain situations that have caused problems in the past should become standard requirements. This latter list will be built with experience about the product.

For instance, if testing a Graphical User Interface (GUI), to test it thoroughly, every button from left to right, top to bottom, of the feature should be tested. Or, if you have a piece of software that has different configuration options, every single configuration option should be tested.

Further, through experience, I have found certain situations should be tested as problems may arise. For instance, stop your product in the middle of use then start it again. Everything should return to a normal state of operation. Any product under test will have different “standard” requirements necessary. I encourage standard requirements with every project, built based off wisdom from past projects. Here are some generic recommended standard requirements:

·         No performance problems after product is put under stress

·         All configuration options work

·         All configuration options are documented (tester should verify this and ensure accuracy of document)

·         Product can be restarted

·         Product does not break or “crash” during use

·         Installation of product is smooth and all sub-products necessary for installation are included (or documentation states need for sub-product)

·         Error statements are intelligible and meaningful

·         No spelling errors on labels for product

·         Consistent formatting throughout all of product

Problem reporting

When reporting a problem, the problem should be described as specifically as possible. You should always know the version number of every product you are working with and put them in the report. You should know as much about everything as possible. Do not take a single thing for granted. Effort should be made to narrow the problem down to as simple of circumstances as possible. 

One problem should get one report. It is sometimes argued that 3 or 4 symptoms are actually the same problem and should only get one report. That is merely theory until it is tested. All symptoms should have their own report so that all symptoms are verified before closing the problem report. If there is more than one symptom in one problem report, extra symptoms usually get ignored.

The tracking software used to organize problem reports should be sophisticated enough to group them based on what test and/or project the problem is associated with. A query can then find all problems associated with test procedure X or project Y.

Problems should always be documented. This is the only way they can be effectively tracked. There is no way that an undocumented problem—and rationalizations abound for why problems should not be documented—will be tracked properly. When problems are documented, there is an easy list that can be referenced of all problems tied to a product or project. In a report, the problem report ID numbers need only be referenced, otherwise a book is written in the report. It was sometimes said where I work, “A problem should not be documented until the solution for it is known.” If the solution was known, it would not be a problem! Some investigation should be done to narrow a problem down, engineers and developers should be notified before writing a problem report (to ensure it is not user error or working as indeed expected), but once it is agreed upon that it is a problem, it should be documented. Even if more analysis needs done (and testers or developers can do the analysis) it should still be documented. At least the problem is noted. Otherwise, if not noted, inevitably it is forgotten about, and whatever work had been done on it will vanish in the air.   

Test the Tools You Test With

If you buy an off the shelf tool to test with, you may perhaps assume that it has been tested and works as expected. But if you build tools to test with in-house, they should be tested. These tests should produce false results—the tool is capable of accurately failing tests that should fail. If you don’t test the tools you test with, it is similar to stepping on an uncalibrated scale. It will cause a lot of unnecessary headaches.

Over Testing is Under Testing

There is often a sentiment that “just in case” the product should be over tested. For instance, test it longer than necessary, test variables that aren’t that much different, etc. There is always a cost to doing anything and, in my opinion, when you over test, you under test. The main problem is that by over testing, you are taking up precious time. When a test is made, it should be run for most product releases. That test, locked in to routine tests, will get run over and over. This can get very bloated. For instance, we had an automated suite of tests that ballooned to taking 14 hours to complete. This meant that it could not be run overnight—indeed, meant we couldn’t test it regularly, which meant under testing.

I also believe that, when possible, tests should be fit into one test procedure instead of several. Every test procedure needs tracked, there is always set up and take down time, and each test procedure needs a report. Instead, when you have brought up all systems, do all the testing with it up, then take it all down and write one report. Most tests do not take long as to justify dividing tests up for time reasons. A reason to divide test procedures is if they have different set ups.

Also, as noted above, if you test in anything but the operational configuration “to make it a more difficult test,” you are then not testing in a particular configuration and losing some value in testing.

“What’s different?” (Systems engineering)

Testing is more than just running a test and then giving a pass or fail status. A monkey could probably be trained to do this. If a test fails, a good tester will take it upon themselves to investigate further as to why it is failing. This generates much intelligence to give to development in solving the problem.

The most important question a tester should learn to ask is, “What’s different?” If two different tests produce two different results, it is not some wild mystery, something is different.

For instance, a test passed in-house but not on-site once. After much investigation, it came down to a different version of the operating system that was used. I tell this example to point out that the seeming smallest of variables do indeed matter.

If you do have different tests that produce different problems, don’t see it as a frustrating mystery, but a very valuable investigation tool. If you find a problem and you only have one system to work on, you have no point of reference to work off of to narrow down the problem. But if a test on a different system produces results that pass, you are sitting on a gold mine. The goal is to compare the two systems until you find what is different. You greatly narrow down what the potential problem is.

Reduce Complexity (Systems engineering)

Too often when a problem arises, a new engineer on my team will have all data up, all features on, and will be trying to solve the problem like this. Perhaps it is inexperience or perhaps it is ego, but they think they can solve a problem while standing on one foot, juggling bowling pins, with an eye patch over their eye.

I am always telling them to reduce the complexity. Turn off all data and features such that you have the simplest scenario possible to recreate the problem.

One method to do this, if having trouble narrowing down the problem, is to turn off half of the things you have on. Does the problem still exist? If it went away, turn off everything currently on, and turn on the stuff you just took down. You just narrowed down the probably to ½ of the previous suspects. Don’t have anything extraneous on that does not need to be on.

Or, turn off every feature and return the product to its most simple state possible. Start turning on features to see how it reacts. Go to a known state of things then start deviating from that.

I think of it as building a tower. First, you build the first floor. You build it, bang on it, make sure it works. Then build the next one, bang on it, make sure it works. You do this until it breaks.

All systems are built from smaller parts. If you reduce the system down to the relevant parts, it is easier to deal with. The ability to solve most problems, I have found, no matter how complex they seem, come down to understanding a few simple things. There is virtually no problem, even in a technical world, that is unsolvable by a person willing to think.

Have Something to Show for Your Work

This is a general philosophy of mine and also one that applies to testing. If ever I do work, I want there to be some object at the end of the day that shows what work I did. If I learned something new, I want a document with either my notes or a full blown documentation of what I learned. If I worked on some test scripts, I want to put them on a shared server so that my work is seen and can be used by other people. If I found a problem, I want a problem report written. Otherwise, if I vanished one night, it would be as if I had done nothing. I casually call it "stamping" my work but "stamping" is not the best word for it.

Similarly, as a test engineer, your work should “touch” the product. You probably won’t write the code or engineer the product yourself. But when you test it, you alter the product. If you aren’t altering the product, you aren’t doing any work.

Testing, though often unappreciated, is vital to the quality of a product. Testing should not be considered a necessary evil but integral to the success of a company. Invest in good testing and you invest in a good product.

Amber Pawlik
July 22, 2012

This article is protected under the US Copyright Act of 1976. No part may be copied.

Home / About Me
Email: amber - at - amberpawlik - dot - com