Test my thoughts

Testing metrics - still a problem

2008-12-30T11:04:00.000Z

My problem is not with the actual metrics. I think I’ll be able to define the right ones for me, which will answer my particular needs and not some “proved”/”best practice” programme/approach.
My concern is with the people, how they would react to applying the metrics. My biggest concern is that when metrics are applied target shifts from finding software problems to satisfying the metrics. The bigger the company – the bigger the issue.

You can’t completely forget about the metrics, they are needed for going forward!

"Certification zombies"

2007-11-24T21:56:00.002Z

It's been about a year since I graduated. I still remember the “most important exam” of my life (until the next one).

I didn't get the best or the worse grade in my group, I was an average student, actually a little bit below overage :) - yes I admit it! I got my grade – does it show what I know? Does it show how good I am? Unfortunately – NO!
And is not just for me, but for the most of us. It showed how close or far were our answers from the expectation of our professors. There were people with great grades, but they have no idea about what they were talking about. Not just on the subject but they have no “technical” mind at all. Is just not for them!

After university comes grownup live. But here we have same problems – certifications. Everyone wants to get one. If you have some time on you hands, you get some books and in a couple of months you will get your certification. You don't need to actually understand what is it about just need to answer as examples from the book. DO NOT try to think, you might come up up with a different answer – and there it goes your certification. Don't get me wrong there are people who actually know what they are talking about. But there is that small percentage who drag down the rest of “certification holders”. As James Bach says on his blog:
“There are now thousands of people who might be called “certification zombies” lurching around in an ISEB or ISTQB-induced fog, trying to apply what they learned in a few days of memorizing to the complex reality of testing.”

Unfortunately for the managers this is the best proof of how good you are. You just don't get appreciated if you don't have a certification. So I'll have to go with the horde and get one. I hope this will not abolish my thinking, put my brain in a box unable to see behind the “book rules” and make me one of those zombies James Bach was referring to.

Risk-Driven Testing / Risk Based Testing

2007-11-21T17:55:00.001Z

Why?
Whenever there's too much to do and not enough time to do it, we have to prioritize so that at least the most important things get done. In testing, there's never enough time or resources. In addition, the consequences of skipping something important are severe. Classify features or application unit with severity ranking, defining severity by measuring the negative impact of an eventual issues that could have on the work of that product.

Example of severity: High - meaning the customer cannot work with the software and wastes time and money, medium - meaning the user has to use some alternative workaround to achieve the aimed goal and low - meaning the customer can STILL work with the software.

This is classic theoretical relation between Risk and Testing effort:

It is known that usually 20% of the total features of an application will allow user to do 80% of his work (Pareto Principle in software products). So the ideal relation between Testing effort and risk will look something this:

where most of the problem has been discovered within first 20% of testing effort.

To make it real, we have to make sure we test what's most prior first and only after that, if we have time, we can attend to things that are less important. This is why prioritization has received a lot of attention.
This type of testing is called "Risk Driven Testing", where Risk has very specific meaning.

How?
Take the pieces of your system, whatever you use - modules, functions, section of the requirements even test cases and scenarios - and rate each piece on two variables, Impact and Likelihood.
Impact - is what would happen if this piece somehow malfunctioned. Would it destroy the customer database? Or would it just mean that the column headings in a report didn't quite line up?
Likelihood - is an estimate of how probable it is that this piece would fail.
Together, Impact and Likelihood determine Risk for the piece.
Placing all pieces of product on virtual axes of likelihood and impact, you will obtain something like this:
Having this diagram we will separate our testing effort in 4 big steps.

Example:
You received from the development team a new application; this application was never been tested before. And you have been told that you have to test it, they also say they don't know how much time you have to test it. You can have a month or you can have a day, every day may be last day for testing.
This is a simple application that enables you to save some objects and their proprieties. Also user can edit an existing record and can delete an existing record. Application enables you to delete and edit multiple records. All changes that have been made to the objects including deleting and adding a new object are logged. Objects are displayed to the user; user can select to clipboard their proprieties and can sort the object list and search objects by different criteria. Multiple selection is done by dragging, imagine cells in excel.
Think of the application as a desktop application that connects to the server that has the database.
Please make abstraction of the other details that I didn't mention, or if you think you can't continue without some details, make your own assumptions.

First step

The pieces in the upper right hand quadrant, which have both high Impact and high Likelihood, need the most of our attention. That's why these pieces will bi tested first.

Example:
- stores objects and their proprieties;
- delete objects and their proprieties, one by one;
- displays existing records;

Second step

Each piece of the product which is considered to have a higher Impact require our attention, even if the Likelihood value is low.

Example:
- edit existing records, one by one;
- multiple delete of the objects and their proprieties;

Third step

In this step we attend to the things that are most likely to happen, regardless its impact.

Example:
- changes logging;
- search objects;

Fourth step

In this final step, in case you have enough time for it, you will be testing the rest of the application. Those modules/pieces that have a smaller impact on the product overall and are very unlikely to be encountered by the user.
Here goes those 80% of the test effort that I've mentioned earlier.

Example:
- sort objects;
- multiple edit;
- clipboard selection;
...

Bug, Defect, Issue or just a Problem

2007-11-21T17:52:00.001Z

Bug, Defect, Issue, Problem - most of us are using this terms interchangeable. Is it right to do so? Do all these words describe the same thing? When do we use one or another?

Lets go to the root of these words, to find out their meaning.

Bug - Usage of the term bug dates back to 1842. It was first use by Ada Byron in her notes. Of course in her case it was an actual bug which was causing problems. Since then it became as a part of the IT jargon. It is used to describe inexplicable defects.

Defect - An imperfection in a bodily system.

Issue - An important question that is in dispute and must be settled.

Problem - a state of difficulty that needs to be resolved.

As a tester I very often come across an “unusual behaviour” or “unexpected results”. This could be as a result of a fault but it as well could be a wrong expectation from my side. Until we found out what is the cause we still need to track, log or just make a note of it. So what do you call it? First I would say this is an issue or a problem, it something wrong and not necessarily with the system. And only after we are convinced that it is a problem with our product and it requires changes to be made, only then I will address it as a defect.
It seems ok to reject an issue, since it may be a result of a wrong expectation, but I think one should not be able to reject a defect. A defect should be acknowledged and archived, but not rejected.

Ok we figured when we have an issue/problem and when we have a defect. But when we use bug? Do we have to look for small insects in our systems?

When I’m having a beer with my friends (which happens to be in IT industry as well) and someone brings up to discussion our work. Then we say bug/bugs. You have bags in your head!

Of course in the end all this don't really matters much. It’s all about the quality, regardless of how you call it.