When do you know youíve tested enough?
When do you stop looking for a better price?
When do you stop looking for a better candidate to hire?
When do you stop researching other solutions?
When do you know you have enough data?
describes a situation when someone, because of concern about discovering something new, continues to look for and analyze information. They donít know when to stop.
Here are three steps
to help you determine if you are done. Classify Discoveries; Measure Frequency and Set Thresholds.
A discovery is uncovering something that you didnít know. Classify Discoveries into three categories; (A) Game Changer (B) Worthy and (C) Noted
(A) Game Changer.
A discovery that significantly changes a conclusion. For example, if youíre price shopping and you see a similar item at one-half the price, you might completely change your mind about which item to buy and where to buy it. If youíre in the manufacturing business and you discover a defect in a crucial part, or you create software and you discover a defect that will wipe out a customerís data, youíll push out a release date or re-engineer something.
Not a game changer, but somewhat impacting. Could affect customer satisfaction, or a schedule, or quality of product. You might end up addresses this by disabling some functionality, or requiring maintenance, or resetting expectations. It might be discovering that your candidate isnít exactly what you thought you were getting or a forecast or a price that is 10% different. You probably wonít change course, but youíll want to know if this is an anomaly, or just the tip of the iceberg.
Nice to know, but little to no consequences. Possibly require a work around, or a little, but not needed.
How long does it take to discover something in category A, B and C. Measuring frequency is a way to determine how much effort it takes to discover an issue. At the beginning of an analysis, research or test, you might discover several category Aís that alter the course of a decision. Youíll discover category Bís that require further review, and lots of category Cís. Over time youíll discover fewer Aís , Bís, and Cís. Your discoveries will take longer and longer, and over a period of time, the majority of discoveries will shift into category Bís and Cís, and then eventually, mostly just category C, and far fewer as well.
When the Frequency of each of the categories of Discovery falls below a certain value, i.e. Threshold, you are done. Here is where Risk comes into play. For example, whatís the risk if a category A or B is discovered. In some cases, such as software, the risk is pretty low because you can just download the new version from the internet. In the case of safety, a category ďAĒ might end up being a risk of a car crash. Based on the risk, you set a Threshold for Frequency per Category found. When the Frequency falls below the Threshold, for each Category, you are done.
Despite all the testing, analysis, or research in the world, there is no guarantee that a defect, perhaps severe, might still exist and be discovered. However, following the above methodology; by thinking about the categories, tracking the issues discovered, tracking the frequency to find these, and setting thresholds, you will greatly increase the probability that a decision to move forward is the right course to take, and prevent analysis paralysis.
Create Categories of Discoveries, Track the Frequency (time and effort) to uncover a discovery, set a threshold related to risk. When you meet the threshold, youíre done, itís time to stop.