Table 1 Summary of the key analytical methods used to assess health interventions and their relative trade-offs

Interrupted Time Series (ITS)

A before-after comparison in the level and trend of outcomes pre and post intervention [17, 21, 22]

Straightforward methodological approach without reliance on simplifying assumptions [17, 21, 22]

Influenced by simultaneous events occurring at the time of intervention [17, 21, 22]

No control group to compare intervention effects against a group exposed to the intervention which can bias estimated intervention effects [23]

Difference-in-differences (DiD)

A contrast of outcome changes pre and post intervention using a naturally occurring control group and treatment group subject to the intervention change [18, 24]

Using the intervention itself as a naturally occurring experiment, allows to difference out any exogenous effects from events occurring simultaneously [18, 24]

The parallel trends assumption is based on counter-factual intervention trends which cannot be tested [18, 24]

Use of a naturally occurring control group to compare intervention effects naturally isolates group differences from intervention effects. No statistical test to verify the parallel trends assumption can bias estimated effects [18, 24]

Synthetic Control (SC)

Comparison of treatment effects between a treatment group and a constructed control i.e. a synthetic control using weights similar to treatment outcomes pre-intervention [25, 26]

Can complement other analytical methods particularly when a naturally occurring control group cannot be established and/or when simplification assumptions do not hold e.g. the parallel trends assumption in DiD [25, 26]

Requirement of sufficient data pre and post intervention containing sufficient detail of control weights similar to the treatment group [19]

Can overcome parallel trends assumption required for DiD. Cannot test for similarity of controls used to construct the synthetic control which may bias estimated intervention effects. Heavy data requirement pre and post intervention [19, 25]

Matching

A comparison of outcomes between treatment and control groups pre and post intervention post matching groups with similar observable factors [18, 27]

Reduction of biases within groups is eliminated due to matching [18, 27]

Requirement of sufficient data pre and post intervention for matching similar observable characteristics between treatment and control groups. No statistical means to testing ‘similarity’ [27]

Heavy data requirement to match similar characteristics. Matching is limited to observable factors and does not account for non-observable factors. ‘Similarity’ determined using subjective judgment and cannot be statistically measured and can bias estimates [27].

Instrumental Variables (IV)

An observable variable i.e. the instrument is selected to randomise the estimation of treatment effects [18, 20, 28]

Introduction of randomness when estimating treatment effects to reflect similarity to a RCT [18]

Dependence on choosing the most appropriate instrument to satisfy the assumption of no relationship between the outcome and assuming outcome is affected only via intervention exposure [18, 29]

Imposed randomisation using an instrument useful for estimating intervention effects. Randomisation is imposed and not naturally occurring like with DiD and can bias estimated effects [18, 20, 28, 29]