Testing World is a Hebrew magazine for software testers (http://www.testingworld.co.il).
“Looking for Trouble” is my column in the magazine. In the column I write on a variety of subjects related to software testing.
The column is of course in Hebrew, but I translated many of the pieces; some were then published on stickyminds.com and a copy is available here.
Pick up a Book!
Download Article ( Hebrew)
Learn a bit about me, how I got to work in SW testing and why I think reading books about the profession – not just blogs and website articles, is important for your career. The article includes a link to recommended reading list.
Less is More
Picking Your Test Automation Language
Download Article (English / Hebrew)
It’s a classic dispute: Two test automation engineers can’t agree on which programming language to use. In some contexts, the strong points of a certain language definitively make it the right choice, but what do you do when either language could work well for a project? That’s when it becomes a managerial decision.
Tradition
Download Article (English / Hebrew)
When we try to implement new processes, there is often resistance from the team. People get so used to their typical habits that it doesn’t occur to them that there could be a better way to do things. To get buy-in from everyone, you need to understand the current traditions, then think about how you can set an example to start making the processes a new tradition
Overcome Your Hatred of Test Documentation
Download Article (English / Hebrew)
Writing test documents is a good practice to have: It enforces an orderly thought process, explains what you’re planning, and improves the test strategy. But knowing it’s useful doesn’t make it any more fun. This article suggests five tips to help make the idea of test documentation a little easier—or at least a little more difficult to hate.
Exit Criteria, Software Quality, and Gut Feelings
Download Article ( English / Hebrew)
Bug counts and trends don’t cover all the quality aspects of a product. A good exit criteria list provides an orderly list of attributes that research and experience showed to have impact on product quality, so you can monitor the product quality at any given time and forecast the expected status at release. That’s how you improve your product.
Who is to Blame?
Why does SW Testing has low prestige?
Download Article (Hebrew)
Software testing does not enjoy the high prestige that software development enjoys. In fact, it is seen as a second-level job in the development hierarchy. Why is this so? Is this a lost case or can we do something to change this stigma?
Tools Rules
Download Article (English / Hebrew)
When we speak about Test Automation, our first association is of a major integrated system: An execution controller, scripts, monitors, logs, reports, a user-interface and a database that supports the whole thing. Something big and complex.
But if we use James Bach’s definition for test automation (“any use of tools to aid testing”), we quickly realize that we write many of such tools every year. As tool writing is a major part of our daily work, it makes sense to think about the correct management of this activity.
Consistency
The hidden requirement
It’s easy to see that style consistency is important when discussing the user interface. But there are other areas where being consistent is just as important, even though they are not as visible. Consistency is one of the quality attributes of a product—any product—even if it is not stated clearly in the requirements documents, and testers have a responsibility to check for it.
Same Old Story
Developing and using bug taxonomy
In software testing, bug taxonomy involves defining feature categories and collecting lists of possible bugs in each category. These lists can be used to give inexperienced testers some starting points, to help experienced testers brainstorm new ideas, and to evaluate the completeness of a test case. Using an existing bug taxonomy can be useful, but creating your own is even better.
Double or two?
When is a bug a Duplicate?
When can a bug report be considered redundant because it is already reported in the bug management system? If you ask the developers, if two bugs are caused by the same mistake in the code, it’s enough to report one of them. But from a tester’s perspective it’s better to err on the side of over-reporting bugs.
Ethics
When we speak of “test ethics,” the given examples usually are trivial dilemmas. Do we avoid reporting a bug? Do we report that testing is progressing as planned, even though it’s definitely late? These questions are kids’ stuff: easy because the situation is so black-and-white. But life will present you with complicated cases where the answer is not that obvious.
Opportunity Cost
How Much of Debugging Software Is a Tester’s Responsibility?
Everyone knows a tester’s job is to help improve the quality of the software under test. But it gets a little murky when you try to define the boundary between testing and debugging. There’s no clear delineation: Some testers would state how to reproduce the bug, write the report, and hand it off, while others learn the code, find the root cause, and even create builds to fix the bugs. How much is useful, and how much is too much?
Reporting Performance Test Results
Reporting the results of functional tests is relatively simple because these tests have a clear pass or fail outcome. Reporting the results of performance testing is much more nuanced, and there are many ways of displaying these values—but Michael Stahl felt none of these ways was particularly effective. He proposes a reporting method that makes performance test results easy to read at a glance.
To Boot or Not to Boot?
Continuous operation tests find important bugs, partly as a result of their long operation and partly by increasing the probability of finding statistical bugs. However, CO tests have their own downsides. Mandating a periodic reset or reboot can work around these issues, as well as save time and cost for testing, reproduction, debugging, and fix verification.
A License to Err
There are many metrics to measure the effectiveness of a testing team. One is the rejected defect ratio (RDR), or the number of rejected bug reports divided by the total submitted bug reports. You may think you want zero rejected bugs, but there are several reasons that’s not the case. Let’s look at types of rejected bugs, see how they contribute to the rejected defect ratio, and explore the right ratio for your team.
Test Automation Bottlenecks
The test team uses the test automation system to execute thousands of test cases because … why not? The tests are running automatically, for free, so there is no incentive to improve test efficiency. Just run them all! But eventually, as more and more tests are added, the system becomes overloaded. Test runs are delayed and you get a bottleneck. Don’t throw more money—or new systems—at the problem; do this instead.
Cats and Dogs
Testers and developers often have a strained relationship. Each side has a certain level of expectations as to what the other side should know and do, while there is little understanding of the constraints, conditions, and requirements that the other team has to work within. But it does not have to be this way. A little effort in giving more specific and helpful feedback can go a long way toward improving attitudes.
Test Data Collection and Management
There is much published about the data we generate to assess product quality. There is less discussion about the data testers generate for our own use that can help us improve our work—and even less is said about recommended practices for data collection. Test data collection, management, and use all call for upfront planning and ongoing maintenance. Here’s how testers can improve these practices.
Spontaneous Fix
After finding and reporting a bug, a tester may get this response from a developer: “Please rerun the test on the latest version of the code and check if the bug still reproduces.” This seems like a rational request; just as a change can cause a bug to appear, it can also fix a bug. But is following up the responsibility of the tester or the developer? And if the bug is no longer there, how do you classify and close it?
Research Ideas
There are many established ideas for ways to test software, but the industry is changing every day, and there’s plenty of room for growth of new ideas—or challenges to traditional ones. Here are three ideas for “wish-list” research to conduct in order to shake up some of the conventional notions you may have about software testing techniques.
Metamorphic Tests
Metamorphic test technique was first suggested more than 20 years ago, yet it is pretty much unknown and unused. With the increasing ubiquity of AI-based applications, this technique, which solves some of the inherent problems in testing AI, will probably gain more popularity.
Metaphors and Allegories
The right metaphor or allegory can convey a concept in an effective way. Popcorn, screws-tightening, round matzos and safety belts are testing-relevant metaphors.
Security Testing
Security testing is seen by many as the domain of white-hat hackers and experts in penetration testing. In reality, security testing is, to a large degree, a broadening of the scope of functional tests. It is not a completely new domain and often it’s just a matter of paying more attention to the way the program behaves while running tests that functional testers would anyway execute.
The Iceberg Principle
Large parts of any non-trivial system are not clearly visible or apparent to the user. These parts, while hidden, play a very significant role in the workings of the system and without understanding “what’s below the waterline” one does not have a complete understanding of the system. Testers should pay attention to these parts, which tend, because of being hidden, to be under-tested.
A Deeper look at Equivalence Partitioning
When taught at courses or in textbooks, the equivalent partitioning test technique looks trivial and straight forward. But when we try to implement it on a real product, we often find that life is not as simple as the textbook examples. A deeper look at the technique teaches us about its pitfalls and how to use it when partitioning is not trivial.
A Deeper look at Boundary Values
When discussing the boundary values test technique, it is common to talk about two-point and three-point boundary values. What do three values add to the two values? Or maybe we need (spoiler alert!) four values? Other topics covered are the use of boundary values for testing multiple parameters and how to cope with cases where the boundaries are not well defined.
The One Line Challenge
Install Testing
