A key strength of Marketing Mix Models as an approach to measuring and planning marketing budgets is that the model integrates information from many different business drivers. Several longtime MMM consumers of my acquaintance often spend more time thinking about and discussing these non-marketing estimates with their modelers than the marketing effects!
Since I know you, dear reader, have committed to memory part 1 of this series, I am confident you are aware that including these non-marketing drivers can alleviate Omitted Variable Bias. It can also, if chosen carefully, reduce or remove selection bias, so a marketing mix analyst should include these drivers for improving the measurement of marketing – not only for the benefits of understanding all the key drivers of a business.
In this post, we will look at one of ways this strength of marketing mix is also a key weakness. If Omitted Variable Bias was the only known bias a Marketing Mix Model could suffer from, endlessly adding variables would be the best possible approach. But there are other well-known biases and Suppression Bias is probably the most direct ‘opposite’ of Omitted Variable Bias and certainly one of the reasons to be careful about including all known drivers in a model.
Suppression Bias
Suppression Bias occurs when including an independent variable in a regression model reduces the estimated effect for another independent variable. Better yet, it only occurs when the two drivers are correlated.
That is to say, exactly when Omitted Variable Bias also occurs, but with the opposite analyst action. Omitted Variable Bias occurs when we leave a variable out, but Suppression Bias occurs from including the variable!
Reviewing a synthetic example might be helpful.
The data and the analyses
Our example dataset is synthetic data for a single market (let’s call it “National”) over 42 weeks with 5 marketing variables. Here’s a plot (although potentially a difficult one to interpret):
Figure 1
As this is synthetic data, we know the true relationship between marketing activity and Y for all variables. In case you might try to eyeball this, you should know that the true relationship here is between adstocked marketing drivers and sales. The table of true coefficients (and the retention rates for adstock) looks like this:
Table 1
The known correct model here is one including these 5 drivers, plus a base driver (left out for clarity), and an error term. In case, dear reader, you are unsure what we mean when we talk about synthetic data and the known correct model, I mean that I first made up the driver time series, then I adstocked them with the retention rates in the table, then I multiplied the results by the coefficients in the table, summed those products and added a moderate amount of gaussian noise (σ = 20% of the average Y). So, we definitely know the true (and causal, no less!) effect of the drivers and we know the correct model to estimate. Table 2 shows the outcome of that model:
Table 2
Most of us would be pretty alarmed to see TV as a negative driver of sales – even if our job isn’t to manage the TV buy.
Now here is a reduced model, where we remove Radio:
Table 3
Well, that’s better! (except for that darn Meta coefficient, but this story is about the relationship of TV to Radio).
But why isn’t TV suffering from Omitted Variable Bias?
First off, maybe it is! Afterall, we left out Radio and the estimated coefficient for TV is less than half of the true value . . . but if we reflect a minute on OVB, it requires a correlation between the two drivers and that both drivers be correlated to the response variable.
Maybe Radio isn’t correlated to TV or to Y? Let’s review the Pearson correlation coefficients:
Table 4: Correlation to Y
Table 5: Correlation between Drivers
In Table 4 we see that Radio is correlated to Y (and, moreover, in this synthetic data case we know that this linear relationship is real). In Table 5 we see that Radio and TV are quite correlated as well.
So, this has met our two criteria for an omitted variable bias– removing Radio is _supposed_ to bias our TV estimate!
And it is!
But there is a second, (and in this case) stronger effect: Suppression Bias.
The key additional information is that TV is much _less_ correlated to our Y than Radio is (0.02 vs 0.22). In this case, with Radio and TV being strongly related to each other, the relationship of Radio to Y can swamp the relationship of RV to Y, and the best fit coefficients fail to show the correct effect for TV!
But then, what do I do?
The difficulty is clear to you, of course, but for your friends I will state it outright:
Given Omitted Variable Bias, I know I need to include all variables correlated to both a marketing driver and my outcome to recover a true effect of marketing. Given Suppression Bias, I know I have to exclude drivers that are too correlated with Y or my marketing driver effect will be biased. In a real Marketing Mix Problem, I don’t know which bias is driving my estimated further from the truth!
The simple answer is that you think about your model carefully, you use outside information (expert domain knowledge, if you want to call it what the cool kids call it) to choose which variables are most essential to include in your model, and you check your model results against the sensibilities of yourself and stakeholders as you consider possible models.
Perhaps we HAVE to include Radio because the promo code shared on the Radio is the most entered promo code on new customer purchases and so fine tuning the amount of Radio investment is the biggest business question. Perhaps there is no business value in including Radio because the Radio buy was locked in for the full year, but we might still change the back half of the year TV plan. Perhaps we have an amazing geo-test result we can use as our estimate of TV and Radio, and so we can partial out those effects and fit a model on the remainder to get response curves for all of our marketing?
Marketing Mix Modeling is a powerful tool, in large part because it simultaneously estimates the effects of many drivers. As with many powerful tools, it takes care and thought to get the best possible results. Suppression Bias is a reason not to include every known driver, just as Omitted Variable Bias is a reason not to exclude any driver correlated to marketing and the outcome. Knowing that the tension between the two is well established, the marketing mix analyst has the burden of carefully considering prior knowledge when choosing variables to include in a model search. Additionally, the analyst has the responsibility to consider the model results from different variable sets in comparison to each other to see which models are most useful to stakeholders. All of the models are likely to include some biased estimates; some of those models will still be useful for guiding marketing buys.
Some further reading