Asking the Right Question
Before diving into statistics to reach a conclusion, determine what you want to know
by Matthew Barsalou
I started thinking about accident rates while riding in a car when I was visiting another country. Is there a higher probability of getting into an accident there or at home? I know enough about statistics to realize that the answer would depend on the exact question I was asking.
Obviously, a location with 100 accidents is not safer than a place that has 10,000 if the first location only has 101 vehicles on the road and the second location has 10 million. That's a rate of 99% versus 0.1%. This example may be oversimplified, but it clearly shows that it’s necessary to ask the right question before using statistics to reach a conclusion.
Let’s examine this by using two hypothetical countries: countries A and B. Suppose country A has more traffic accidents but also has a larger population. You could quickly make a comparison using a percentage. But now you must determine what the percentage will be:
- A percentage of vehicles on the road that are involved in an accident?
- A percentage of fatalities per 100,000 people in each country?
Neither of these actually would have told me what I originally wanted to know, which was specifically, “Which country has better drivers?” Suddenly, things have gotten much more complicated. What variable would be used to represent a good driver? Even trickier would be determining how to make a proper comparison.
What if country A actually has very good drivers, but the country’s infrastructure is in poor shape and the roads are often covered in ice? These conditions would seem to contribute to a high number of accidents. Drivers in country B, on the other hand, frequently perform unsafe acts, but have far superior driving conditions.
For those wondering which two countries I originally wanted to compare, it was Germany and the United States. I have had driver’s licenses in both countries, and I’ve attended motorcycle driving school in both countries as well. My two and a half days of driving instruction was optional in the United States; attending classes with people half my age during a period of months was awkward, but not optional in Germany.
This led me to suspect Germany has better drivers than the United States. I don’t have hard data on driving ability, but a quick look showed the traffic fatality rate in Germany is indeed lower than in the United States.1 I now have a quick-and-easy answer to my question, but it fails to consider infrastructure, traffic laws and other variables that may cause accidents—even when the drivers are highly skilled and drive in a safe manner.
Using an operational definition makes it easier to ask the right question when planning a statistical study. Kaoru Ishikawa gives an example of sampling in which iron ore delivered by ships needed to be evaluated.2 This was complicated because the iron ore was a mixture of lumps and powder, and other things—such as sand—were mixed in with the iron ore.
Suppose you wanted to determine if one ship contained more iron than the other. Weighing the cargo of each ship would be an expensive way to get an answer that may be wrong because one cargo may weigh more while containing less actual iron ore. An operational definition should be used to clearly define what you’re studying.
An operational definition helps to ensure that the subject of the investigation is clearly understood because “An operational definition of safe, round, reliable, or any other quality must be communicable, with the same meaning to vendor as to purchaser, same meaning yesterday and today to the production worker,” wrote W. Edwards Deming in Out of the Crisis.3 This applies in situations such as testing coal for purchasing price purposes or simply performing a statistical analysis to determine which country has better drivers.
An operational definition is:
- “A criterion to be applied to an object or a group of objects.
- A test of compliance for the object or group.
- A decision rule for interpreting the test results as to whether the object or group is, or is not, in compliance.”4
Deming explained it as “a procedure agreed upon for translation of a concept into measurement of some kind.”5 Deming gives the example of a specification for a cloth that is 50% cotton. Without an operational definition of 50% cotton, one half of the material may be 100% cotton, and the other half made of 100% of some other material.6
There are three questions that can be used as a framework for creating an operational definition:7
- What do you want to accomplish?
- What method will you use to accomplish your objective?
- How will you know your objective has been accomplished?
Suppose a study is to be performed to determine the time it takes for somebody to enter an order into a computer system. When exactly does the clock start? Without a clear operational definition, the person performing the time study may start the clock when the clerk starts typing data into the program and stop the clock after the clerk saves the data.
But what if the clerk frequently needs to call people for clarification before entering the data? Does it take two minutes or 20 minutes to enter one order? Neither answer is wrong, but the wrong conclusion could be reached if the person using the study to make a decision has a different understanding of order entry time than the person who performed the study.
A manager could mistakenly conclude the clerks must type faster because it should not take 20 minutes to enter the data into the system. Here, a proper operational definition should consider the objective of the study, which is determine how long the process takes to seek improvement opportunities.
An operational definition may be: “The average time to enter an order as determined by using a clock capable of measuring one-tenth-of-a-second intervals. The time will start when the clerk first picks up the order from the inbox, and the time should stop after the clerk has closed the order entry screen for the current order. Four clerks will be observed during a two-week period, and the average of all times shall be used as order entry time.”
Often, operational definitions are built into specification such as the specification describing a type of steel or cast-iron material. Other times, you’re not so fortunate and should clarify one as one of the first steps in performing a statistical study.
Otherwise, your statistics may give you a result that is correct related to the data that were analyzed, but any conclusions drawn may be incorrect regarding the question you really wanted to answer.
- Wikipedia, “List of Countries by Traffic-Related Death Rate,” https://en.wikipedia.org/wiki/List_of_countries_by_traffic-related_death_rate.
- Kaoru Ishikawa, The Man and Quality Control: Chapter 13 Research and Standardization of Sampling Method and Analytical Testing, www.juse.jp/ishikawa/e/man/Ch13_Ver2_150717.pdf.
- W. Edwards Deming, Out of the Crisis, Massachusetts Institute of Technology Press, 1989.
- Donald J. Wheeler, “Manuscript 248: Shewhart, Deming and Six Sigma,” 2009, www.spcpress.com/pdf/DJW248.pdf.
- Ishikawa, The Man and Quality Control: Chapter 13 Research and Standardization of Sampling Method and Analytical Testing, see reference 2.
- Donald J. Wheeler, Twenty Things You Need to Know, SPC Press, 2009.
Matthew Barsalou is a certified Six Sigma Master Black Belt located in Germany. He has a master’s degree in business administration and engineering from Wilhelm Büchner Hochschule in Darmstadt, Germany, and a master of liberal studies from Fort Hays State University in Hays, KS. Barsalou is an ASQ senior member and holds several certifications.