Dealing with Exponential Explosion – All Pairs and PICT tool
Download Presentation
As the number of inputs that a software system accepts grows, the number of different combinations of these inputs grows exponentially. It does not take a lot of input parameters and values to have so many combinations that it is not practical to cover all the possible combinations by tests. One technique to deal with this problem is the All-Pairs test technique.
This lecture explains the All-Pair test techniques and demonstrates how to apply it, using PICT tool (freeware from Microsoft).
This paper was presented in 2009, in a SIGiST event in Tel Aviv, Israel.
Right and Wrong in Writing Test Cases
Download Presentation
Test Case design is a common and well covered topic in SW test publications. On the other hand, very little is said about the correct writing style to use when writing Test Cases.
Using test-case examples, this presentation helps understand how better writing style can simplify test cases and improve their clarity. Common mistakes in test cases writing and organization, as well as practical proposals for improvements are covered as well.
This paper was first presented in SIGiST 2009, Tel Aviv, Israel and later in QA&Test 2010, Bilbao, Spain
The Pathologies of failed Test Automation Projects
Download Presentation / EuroSTAR eBook
Most test automation projects never die — they just become a mess and are redone. Initial solutions that start well and are full of promise often end up as brittle and unmaintainable monsters consuming more effort than they save. Political feuds can flourish as different automation solutions compete for attention and dominance. Tests become inefficient in both execution time and resource usage. Disillusionment ensues, projects are redefined, and the cycle begins again. Surely we can learn how to avoid such trouble on the next project. In this presentation, I describe failure patterns I have encountered during my years in software testing. The patterns are described and I give some suggestions how to detect them early, avoid or mitigate them.
This paper first appeared in the fomrat of the “Patterns in Test Automation project failure – and Recovery” paper created with Alon Linetzki (www.best-testing.com). I later added other failure patterns and wrote this one.
This paper was first presented in STAREast 2013, Orlando FL, USA
Roadblocks to Bug Reporting
Download Presentation / Paper
Testers sometimes find a bug, yet do not report it. The reasons vary and include technical and psychological reasons. This paper describes a list of such situations and proposes strategies to overcome them. By making testers aware of these cases, they will be able to recognize when they take place, and apply the proposed solutions to avoid them.
This paper was first presented in SIGiST 2008, Tel Aviv, Israel, and later in Conquest 2008, Potsdam, Germany
The Home-grown Tools Syndrome
Download Article
Test management is a generic process, yet much effort goes into developing tools in house to do this work. Learn the reasons for this phenomenon and suggestions for avoiding it.
This article was first published on www.stickyminds.com “Weekly column” on August 2012 ( http://www.stickyminds.com/s.asp?F=S17601_COL_2 )
More Important than Techniques
Download Presentation
When discussing Test design, a lot of the focus is on Test Techniques: How to select an effective set of tests out of the practically infinite list of possible tests.
However, even before test-techniques, it is critical that testers are familiar with a number of principles and rules about testing; principles that deal with the question on “how does one Test” rather than “how to design the tests”.
In this paper, I will discuss the very basic principles of the testing profession. Knowing these principles will help testers create better tests and avoid tests that don’t really test anything or are insensitive to some problems.
This paper was first presented in SIGiST 2012, Tel Aviv, Israel.
Effective Bug Management
Download Presentation
Effective bug management is a critical activity in a SW project. When bug management is ineffective, the project as a whole suffers: time, effort and energy are spent not on fixing bugs, but on arguments and delay tactics.
The parameters involved in effective bug management span the range of purely technical issues, to human behavior and to organizational politics. In many cases, these parameters are conflicting – satisfying one will result in neglecting the other. Finding the best solution for the conflicts is not easy. Components of the process, such as the bug triage, can be improved when the process owner learns to work with or around the technical, organizational and personal aspects.
In this paper I share the experience collected over many years of involvement in bug management activities. The talk includes description of challenges that are part of the process, and the solutions used by my organization to deal with them.
This paper was first presented in EuroSTAR 2009, Stockholm, Sweden.
The Fab Experience:
How I Stopped Whining and Started to Appreciate Process
Download Presentation / Paper
Many software teams struggle to implement and comply with quality processes. Process adoption and strict adherence is seen as stifling, blocking innovation and redundant.
Contrast this with semiconductor fabrication plants (fabs) where adherence to process is the everyday norm and no fab engineer feels compliance with quality processes is optional.
Studying the top reasons why the fab world follows process so diligently reveals some underlying ideas that can be used in the software development world. Applying these ideas, software companies can improve the ability to implement and adhere to quality processes.
This paper was first presented in PNSQC in Portland, OR, USA (October, 2014)
Is What You See What You Get?
Download Presentation
Computer Vision is gaining foot in everyday commercial systems. Face recognition is now part of Windows 10; virtual reality games and hand gestures control of the PC are just some examples of the coming flood of applications.
Intel is one of the companies leading this trend with its depth-camera offerings and the RealSense SDK.
This is great… but how do you test these applications? It turns out a lot of the standard approaches don’t work well and testers need to constantly invent new ways to test. This presentation describe the very difficult challenges facing testers of computer vision applications and some of the directions taken.
Body of Evidence: The Challenge of Measuring People
Download Presentation
Body models play an increasing role in the virtual world. The market offers a plethora of full-body scanners (including RealSense-based), but is struggling in defining the quality metrics for these models. How to validate a body-scanning system? Learn about this difficult problem, the emerging solutions and the unresolved challenges in this presentation.
99%
Download Presentation
Are you wasting your organization’s resources with Innovation?
Are you sure your Innovation effort is not misdirected, irrelevant and redundant?
Over-focus on Innovation is a mistake that is repeated frequently in the software test profession, where it is promoted for the wrong reasons. While effective in the short term, such innovation may yield frustration and disillusionment more than innovative solutions.
Using examples from various disciplines – from art to engineering – the lecture discusses the problem of over-focus on Innovation and presents a balanced approach of encouraging Innovation while using exiting solutions.
Testing of ISO 26262 Configurable Software
Download Presentation / Paper
ISO 26262 (Part 6, Annex C) specifies the test requirements for a Configurable Software. A Configurable Software’s feature set and behavior is dependent on its Configuration data (parameters that influence the compilation results) and Calibration data (parameters that influence the compiled software behavior). Most developed software ends up fitting this definition, so most organizations that develop software for the automotive domain need to comply with the directives of Annex C.
The standard proposes two ways to validate such software, but the text is hard to understand and needs explaining. Once it is understood, it becomes clear that meeting the standard’s requirements may call for testing an unrealistic number of SW/HW combinations. Avoiding this huge test effort is possible, if one can present a justifiable strategy for SW/HW combinations selection.
The lecture first explains the standard: what the various terms mean and what options the standard’s suggests for validation of the configurable software. It then propose a well thought out, yet quite generic strategy for the selection of justifiable sub-set of HW/SW combinations to be used in testing.
This paper was presented in QATest Safety&Security, Madrid, March’2019.
Protocol Fuzzer on Embedded Firmware
Download Presentation
Fuzzing is a test technique where large number of automatically-generated tests are executed, with the goal of finding memory leaks, buffer and heap overflows, use-after-free and and other issues, some of which pose a security risk. Fuzzing an embedded system pose a few challenges that do not exist in regular applications (applications that run on a full-fledged OS).
In this talk we start with a short review of fuzzing, then continue to describe the challenges of fuzzing on an embedded system. We then review a fuzzing project implemented a communication protocol in an embedded system. The details of this project provide an example how the challenges of fuzzing in an embedded system can be overcome.
Testability
Download Presentation
When it comes to testing, “observability” and “controllability” are two critically important product attributes. What do these terms mean? How can they be improved?
This presentation outlines the basic terms related to testability and Design for Test. The material covers a number of design patterns that improve testability. While the implementation of these patterns is frequently in the hands of the developers, knowing what to ask for improves the chances of testers to get these features. Apart from theory, the presentation contains a large number of examples from real life situations where testability hooks and features are used.
Writing Good Requirements
This is a workshop that I conduct internally at Intel, but have also taught externally. It is based on material that was first written by Erik Simmons (Intel), with a lot of changes, modification and additions done by myself over the years I taught it.
This workshop is targeted for engineers who read, review, write or use functional and non-functional requirements at product level and at feature level.
The workshop covers:
– Why requirements are important
– What makes a good requirement
– Reviews: why reviews are important; type of reviews; how to review a document
– Pitfalls in writing requirements
– Techniques for writing good requirements
The workshop is a mix of theoretical material and hands-on sessions where students write and review requirements.
Upon completion of this 1 day workshop students are able to: analyze functional and non-functional requirements, write good requirements and identify requirements that are not well written.
There was an attempt to video-tape this course, and it was stopped for various reasons before completion. The videotaped version was planned to be a significantly shorter version of the workshop. You can see a short excerpt here: