Koehn AI Company Logo

Causal Impact: Analyzing the impact of events

7 February 2023

Actio et Reactio

As the Earth keeps on turning and all of us keep on pursuing our daily work and life, things will continue to change and changes cause further reaching effects. Understanding the effects of specific impacts is essential for business success. If you saw increasing material prices after an extreme weather event, e.g., you may want to quantify this to better prepare for similar upcoming events. When you change the advertising strategies for your company, you want to measure the extra profit gained via these efforts.

Causal Impact

With Google’s Causal Impact package (tfcausalimpact for Python), there is an easily applicable tool to do this. We will illustrate how it works, but also where you have to be careful.

As an exemplary dataset, we use the development of the real wages index of Germany, published quarterly [1]:

Fig. 1: Time curve of the real wages index, nominal wages index and the consumers' price index.
Fig. 1: Time curve of the real wages index, nominal wages index and the consumers' price index.
The real wages index (black line, “Reallohnindex”) is mainly described by the nominal wages index (orange line, “Nominallohnindex”) and the consumers’ price index (blue line, “Verbraucherpreisindex”). We directly see that the curves behave quite differently from 2020 onwards, compared to the more steady behaviour pre-COVID. Note that this article is not supposed to be an analysis or even a critique of COVID politics. It illustrates an analysis about the effect the measures had – and it is of course far from obvious how the effect would have been had other measures been chosen instead.

Causal Impact on the Data Itself

Running the Causal Impact framework on a dataset, we only have to tell the algorithm when the impact happened and whether to use additional inputs next to the actual target column. While it offers more ways to customise the analysis, we may only use the target (i.e. the real wages index) for a first run and set the impact on the first quarter of 2020:

Fig. 2: Causal Impact result of run on real wages index only.
Fig. 2: Causal Impact result of run on real wages index only.

Let us use this plot to explain the working principle of Causal Impact [2]. Based on the data before the impact, Causal Impact fits a Bayesian structural time-series model and uses it to predict how the target value (black curve, labelled “y”) would be expected to continue had the impact not happened. This is the so-called counterfactual: If this prediction suffices to explain the behaviour post-impact, there may have well been no impact after all. In the plot above, this is the orange band, with average predictions as the dashed red line. The broader the band, the more uncertainty lies behind the predictions. As we have not provided any further information, the uncertainty is rather large in the above first run. Still, the results make it clear that the provided impact did indeed happen: The second and third plot show the effect of the impact, pointwise and cumulative, i.e. how much the true values exceed the predictions.

Causal Impact with Covariates – How Not to Do it

We already stated that we can also include additional data in the analysis, so-called control time series or covariates. Their purpose is to improve the predictability of the target, but they should not be affected by the impact themselves. This is a very important requirement. If the provided control time series are indeed also influenced by the impact event, the Causal Impact algorithm can no longer reliably quantify the effect of the impact – or even whether there was one in the first place.

We can illustrate this by doing what we shouldn’t: we add the nominal wages index as a control time series, which is a valuable input for determining the real wages index but also clearly experiences the impact itself (confer Fig. 1). The results then look like this:

Fig. 3: Causal Impact result on run with nominal wages index as control time series.
Fig. 3: Causal Impact result on run with nominal wages index as control time series.

First of all, we see that the orange band is much narrower, even following the trend of the true values. For the pre-impact era, this is very much desirable. However, the nominal wages index is not a suitable control time series and this is reflected by the post-impact predictions continuing to follow the true values. As a result, the second and third plots show much less effect of an impact: the amount is smaller and the abscissa (i.e. no effect after all) lies within the uncertainty of the predicted effect. Adding also the consumers’ price index, we even get a symmetric uncertainty band, implying that the model can not quantify the effect of the impact:

Fig. 4: Causal Impact result on run with nominal wages index and consumers' price index as control time series.
Fig. 4: Causal Impact result on run with nominal wages index and consumers' price index as control time series.

To extract more quantitative results, Causal Impact offers the summary functionality. For the first run, this returns:

Tab. 1: Results of the summary function of Causal Impact.
The last lines tell whether Causal Impact can confirm the impact and the table quantifies the amount of the effect, stating the respective values with standard deviations (“s.d.”) and 95% confidence intervals (“CI”). In this case, Causal Impact is certain the impact did indeed happen and that it had a considerable effect. This already changes when we include the nominal wages index as control time series:
Tab. 2: Results of the summary function of Causal Impact including nominal wages.
While Causal Impact still confirms the impact happened, the effect is considerably reduced (do not be confused by the large numbers for the relative effect: These are computed with respect to the predicted value, which is close to zero here.) Including now also the consumers’ price index as control time series, Causal Impact no longer confirms the impact happened and, consequently, the effect is expected to be vanishing.
Tab. 3: Results of the summary function of Causal Impact including price index.

Another requirement for the control time series is that it must have data also before the impact. In the above example, you may of course think that the effect is larger when there were more strict COVID measures (or with some time delay). Comparing the stringency index [3] with the real wages index, we find:

Fig. 5: real wages, net income and consumers’ price index vs. stringency level of COVID measures (min, max, mean per quarter).
Fig. 5: real wages, net income and consumers’ price index vs. stringency level of COVID measures (min, max, mean per quarter).

However, using the stringency level of the COVID measures as control time series, Causal Impact can not extract any information from it as the measures did not exist pre-impact. What is more, if the data is precisely 0 in the whole of the pre-impact era, we run into division by 0 errors. This can be circumvented by replacing the zeroes by small random values, but then – as one can expect – the predictions blow up as the stringency value increases to the actual values:

Fig. 6: Causal Impact result on run with the stringency level of the COVID measures as control time series.
Fig. 6: Causal Impact result on run with the stringency level of the COVID measures as control time series.

The fact that Causal Impact could not gain any information from the stringency level data is also reflected by the symmetric shape of the prediction band: It could not even deduce whether an increase in the stringency level translates to an in- or decrease in the target value.

CI with Covariates – How to Do it

We illustrated two reasons that make a control time series meaningless as such. So what could be a useful covariate in this case? Clearly, it should be one that is linked to the data of interest, the real wages index, but did not experience the impact. In the example of marketing, the data of interest could be a market where you changed your marketing strategy, while the covariates are markets without changes.

It’s not easy to find a similarly good example in the case of COVID, as it impacted pretty much all aspects of life. However, looking at the net cold rent index (“Nettokaltmietindex”, where 100 reflects the level of 2015) [4], we find that it was rather unaffected – and it does make sense, too, as you can’t increase rent arbitrarily in Germany.

Fig. 7: real wages, nominal wages and consumers' price index vs. net cold rent index.
Fig. 7: real wages, nominal wages and consumers' price index vs. net cold rent index.

Running Causal Impact on the real wages index with the net cold rent index as control time series results in:

Fig. 8: Causal Impact result on run with the net cold rent index as control time series.
Fig. 8: Causal Impact result on run with the net cold rent index as control time series.
We see that the net cold rent index helps increasing prediction accuracy at least a bit. This leads to a higher estimated effect of the impact, as can be seen also by the metrics:
Tab. 4: Summary of Causal Impact results.
Note that the smaller relative effect (compared to the run without any control time series) is due to the predictions being higher.

A short final note on the most important modification of the routine: Instead of running it with default parameters, it is possible to choose i.a. a different kind of method to find the Bayesian posterior, the so-called Hamiltonian Monte Carlo method, via model_args={'fit_method': 'hmc'}.

The default way relies on TensorFlow Probability’s Variational Inference, which has the advantage of a considerable speed-up: While hmc took 56s in the above example, the default variatonal inference took 19s. The benefit of hmc is, however, precision:

Fig. 9: Causal Impact result on run with the net cold rent index as control time series and using Hamiltonian Monte Carlo.
Fig. 9: Causal Impact result on run with the net cold rent index as control time series and using Hamiltonian Monte Carlo.

The predictions gained again in accuracy and the estimated effect of the impact increased as well:

Tab. 5: Summary of Causal Impact results.

We hope these examples motivate why you should be careful when choosing your control time series. It is also advisable to run the algorithm several times to get a notion of how stable the results are. Lastly, please consider to run the algorithm on randomly chosen moments of impact that lie before the actual one. If Causal Impact does not provide you with further impacts, it is a good sign that your setup is adequate. Note that further impacts before the one that is examined reduce the quality of the predictions and therefore the reliability of all results altogether.

[1] https://www.destatis.de/DE/Presse/Pressemitteilungen/2022/03/PD22_127_62321.html
[2] https://research.google/pubs/pub41854/
[3] https://github.com/owid/covid-19-data/tree/master/public/data
[4] https://www-genesis.destatis.de/genesis/online?sequenz=tabelleErgebnis&selectionname=61111-0021



Stay up to date with our free newsletter

0 comments

Leave a comment