Similarly, in a clinical trial, there are a lot of other factors, such as patients’ age, general health, exercise regimens, and blood pressure, that can make it hard to see whether the results of the experiment can really be attributed to the drug as opposed to some other factor. You want to know which drill is better, but other factors, such as the size of the well, its depth, and what you’re digging through, will also affect how efficiently you drill the well and complicate your evaluation of the new drill. “Usually in an experiment, you’re trying to learn something about one, or at most a few, independent variables, but a lot of other factors can get in the way,” says Redman.
But there are also lots of independent variables - factors you suspect have an impact on your dependent variable. In an experiment, the variable of interest is called your dependent variable (note that you might have multiple dependent variables, but for the sake of simplicity here, I’ll refer to one dependent variable). But you also have to be realistic about the cost of your experiment, and given that it costs millions of dollars to drill an oil well, you’re likely to run this experiment on a smaller number of wells. The bigger your sample size, the more likely you’ll have results that are statistically significant. Note that the number of wells here is pretty small compared with an experiment, for example, where you’re showing 1,000 potential customers a new marketing campaign. That’s your experiment, and your variable of interest might be how efficiently you drilled the well. You select 30 wells and drill 15 of them with the old bit and 15 with the new. You want to know how this new, more expensive bit compares to the bit you’re currently using, so you conduct an experiment comparing your existing drill bit with the new one. Let’s say you’re in the business of drilling oil wells, and you have a new drill bit that is operated by an artificial intelligence program that adjusts the pressure and speed with which you’re turning the bit. He gives the example of two-year-olds, who are constantly running experiments: “They think, ‘If I scream, mom will come running.’ They are gathering data about the world, and while it’s not controlled, they are doing it purposively.” “An experiment is a planned activity whose purpose is to learn something about the world,” explains Redman. All kinds of businesses can conduct these experiments, and they necessarily don’t need to be costly or time consuming - they just need to be “controlled” and include an element of “randomization.”
When people hear the term, they most often think of clinical trials, where one group is given a treatment and another a placebo, but pharmaceutical companies and medical scientists aren’t the only ones using these types of experiments. What is a randomized controlled experiment? He also advises organizations on their data and data quality programs. To better understand what a randomized controlled experiment is and how businesses use them, I talked with Tom Redman, author of Data Driven: Profiting from Your Most Important Business Asset. One of the more structured experiments is the randomized controlled experiment. There is a spectrum of experiments that managers can do from quick, informal ones, to pilot studies, to field experiments, and to lab research. Where that data comes from and how we analyze it depends on a lot of factors - for example, what we’re trying to do with the results, how accurate we need the findings to be, and how much of a budget we have. In order to make smart decisions at work, we need data.