Table 3 Quality assessment of economic evaluations—overview
Brown et al. (2007) [23] | Kesztyüs et al. (2011) [26] | Krauth et al. (2013) [28] | McAuley et al. (2010) [29] | Moodie et al. (2009) [22] | Moodie et al. (2010) [34] | Moodie et al. (2011) [35] | Moodie et al. (2013) [30] | Peterson et al. (2008) [21] | Pringle et al. (2010) [24] | Rush et al. (2014) [37] | Wang et al. (2003) [36] | Wang et al. (2008) [27] | Wang et al. (2011) [25] | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Economic evaluation | Is the research question posed in terms of costs and consequences? | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Is a specific type of economic analysis technique performed? | 1 | 1 | 0.5 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | |
Domain Score | 2 | 2 | 1.5 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | |
Comparators | Is there a rationale for choosing the intervention(s) being investigated? | 1 | 1 | 1 | 1 | 0.5 | 0.5 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Is there a rationale for choosing the alternative program(s) or intervention(s) used for comparison? | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | |
Does the report describe the alternatives in adequate detail? | 0.5 | 1 | 0.5 | 1 | 0.5 | 0.5 | 0.5 | 0.5 | 0 | 0 | 0.5 | 0.5 | 0.5 | 0.5 | |
Is a description of the event pathway provided? | 1 | N.A. | N.A. | N.A. | 0.5 | 0 | 0.5 | 0.5 | N.A. | 0 | 0.5 | 1 | N.A. | 0.5 | |
Is a formal decision analysis performed? | 0.5 | N.A. | N.A. | N.A. | 0 | 0 | 0 | 0.5 | N.A. | 0 | 0 | 0.5 | N.A. | 0 | |
Domain Score | 3.5 | 2.5 | 2 | 2.5 | 2 | 1.5 | 2.5 | 3 | 1 | 1.5 | 2.5 | 3.5 | 2 | 2.5 | |
Target population | Is the target population for the intervention identified? | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0.5 | 0.5 | 1 | 1 | 1 | 1 |
Are the subjects representative of the population to which the intervention is targeted? | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0.5 | 0.5 | 1 | 1 | 1 | 1 | |
Domain Score | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 1 | 1 | 2 | 2 | 2 | 2 | |
Time horizon | Is there a time horizon for both costs and outcomes? | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0.5 | 1 | 1 | 1 | 1 |
Do the authors justify the time horizon selected? | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 1 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | |
Domain Score | 1.5 | 1.5 | 1.5 | 1.5 | 1.5 | 1.5 | 1.5 | 2 | 1.5 | 1 | 1.5 | 1.5 | 1.5 | 1.5 | |
Perspective | Is a perspective for the analysis given? | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 1 |
Is a societal perspective taken, either alone or in addition to other perspectives? | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | |
When there is more than one perspective, are the results of each perspective presented separately? | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | |
Domain Score | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 0 | 0 | 1 | 2 | 2 | 2 | |
Costs and resource use | Are all relevant costs for each alternative included? | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Are opportunity costs of lost time (productivity costs) for parents and informal caregivers measured when required? | N.A. | N.A. | 1 | N.A. | N.A. | N.A. | N.A. | 1 | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | |
Do cost item identification and valuation extend beyond the health-care system to include school and community resources when necessary? | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0.5 | 1 | 0.5 | 1 | 1 | 1 | |
Are future salary and productivity changes of the child taken into consideration when appropriate? | 1 | N.A. | N.A. | N.A. | 0 | 0 | 0 | 0 | N.A. | 0 | N.A. | 1 | N.A. | 0 | |
Are all of the sources for estimating the volume of resource use described? | 1 | 1 | 0.5 | 0.5 | 1 | 1 | 1 | 1 | 0 | 0 | 0.5 | 1 | 1 | 0.5 | |
Are all the sources for estimating all of the unit costs described? | 1 | 1 | 0.5 | 0.5 | 1 | 1 | 1 | 1 | 0 | 0.5 | 0.5 | 0.5 | 1 | 0.5 | |
Domain Score | 5 | 4 | 4 | 3 | 4 | 4 | 4 | 5 | 1.5 | 2.5 | 2.5 | 4.5 | 4 | 3 | |
Outcomes | Is a primary health outcome given? | 0.5 | 1 | 1 | 1 | 0.5 | 0.5 | 0.5 | 1 | 0 | 0 | 1 | 0.5 | 1 | 0.5 |
Do the authors justify the health outcome(s) selected? | 1 | 1 | 0.5 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0.5 | 1 | 1 | 1 | |
Is effectiveness, rather than efficacy, assessed? | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | |
Are the details of the design of the effectiveness/efficacy study(s) provided? | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0.5 | 0.5 | 1 | 1 | 0.5 | 0.5 | |
Are the results of the efficacy/effectiveness of alternatives reported? | 1 | 1 | 1 | 1 | 0.5 | 0.5 | 1 | 1 | 1 | 0.5 | 0.5 | 1 | 1 | 0.5 | |
Are school/day-care absences taken into consideration? | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | N.A. | |
If intermediate outcome variables are used, are they linked by evidence or reference to the end benefit? | 1 | 1 | N.A. | N.A. | 1 | 1 | 0.5 | 0.5 | N.A. | 0.5 | 0.5 | 0.5 | 1 | 0.5 | |
Domain Score | 5.5 | 6 | 4.5 | 5 | 5 | 5 | 5 | 5.5 | 3.5 | 3.5 | 4.5 | 5 | 5.5 | 4 | |
Analysis | Are costs AND outcomes measured in units appropriate for the indicated analytic technique? | 1 | 1 | 0.5 | 1 | 1 | 1 | 1 | 1 | 0.5 | 1 | 1 | 1 | 1 | 1 |
Are costs valued appropriately? | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | |
Is the valuation of outcomes appropriate for the type of analysis? | 1 | 1 | 0.5 | 1 | 1 | 1 | 1 | 1 | 0.5 | 0.5 | 1 | 1 | 1 | 1 | |
Are quantities of resources used reported separately from their unit costs? | 1 | 1 | 0.5 | 1 | 1 | 0.5 | 0.5 | 0.5 | 0 | 0.5 | 0 | 1 | 1 | 1 | |
Are the costs aggregated correctly? | 1 | 1 | 0.5 | 1 | 0.5 | 0.5 | 0.5 | 0.5 | 0 | 0.5 | 0.5 | 0.5 | 1 | 1 | |
Are details of statistical tests and confidence intervals given for stochastic data? | 1 | 1 | 0 | 1 | 0.5 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 0.5 | 1 | |
Domain Score | 6 | 6 | 3 | 6 | 5 | 5 | 5 | 5 | 2 | 3.5 | 4.5 | 5.5 | 5.5 | 6 | |
Discounting | When required, are costs and consequences that occur over more than 1 year discounted to their present values? | 1 | N.A. | N.A. | 0.5 | 1 | 1 | 1 | 1 | N.A. | 0 | 1 | 1 | N.A. | 1 |
If costs or benefits are not discounted when the time horizon exceeds 1 year, is an explanation provided? | N.A. | N.A. | N.A. | 0 | N.A. | N.A. | N.A. | N.A. | N.A. | 0 | N.A. | 0 | N.A. | N.A. | |
Domain Score | 1 | N.A. | N.A. | 0.5 | 1 | 1 | 1 | 1 | N.A. | 0 | 1 | 1 | N.A. | 1 | |
Incremental analysis | Are incremental estimates of costs and outcomes presented? | 1 | 1 | 0.5 | 0.5 | 1 | 1 | 1 | 1 | 0.5 | 0.5 | 1 | 1 | 1 | 1 |
Are the incremental estimates summarized as incremental ratios? | 1 | 1 | N.A. | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 0.5 | 1 | |
Are confidence intervals/limits calculated for incremental ratios or incremental estimates of costs and outcomes? | 0 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 1 | 0.5 | 0 | 1 | |
Domain Score | 2 | 3 | 0.5 | 1.5 | 3 | 3 | 3 | 3 | 0.5 | 0.5 | 3 | 2.5 | 1.5 | 3 | |
Sensitivity analysis | Are all important assumptions given? | 1 | 0.5 | 1 | 0.5 | 1 | 1 | 1 | 1 | 0 | 0 | 0.5 | 1 | 1 | 1 |
Is a sensitivity analysis performed? | 1 | 1 | 0.5 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | |
Do the authors justify the alternative values or ranges for sensitivity analysis? | 1 | 0 | 0.5 | 0.5 | 1 | 1 | 1 | 0.5 | N.A. | 0 | 0.5 | 1 | 1 | 0.5 | |
Domain Score | 3 | 1.5 | 2 | 2 | 3 | 3 | 3 | 2.5 | 0 | 1 | 2 | 3 | 3 | 2.5 | |
Conflict of interest | Does the article present the relationship with the sponsor of the study? | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0.5 | 1 | 1 | 0 | 1 |
Does the article indicate that the authors had independent control over the methods and right to publish? | 0.5 | 0.5 | 1 | 0.5 | 1 | 1 | 1 | 1 | 0.5 | 1 | 0 | 0.5 | 0.5 | 1 | |
Domain Score | 1.5 | 1.5 | 2 | 1.5 | 2 | 2 | 2 | 2 | 0.5 | 1.5 | 1 | 1.5 | 0.5 | 2 | |
Conclusions | Is the answer to the study question provided? | 1 | 1 | 1 | 0.5 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Are the most important limitations of the study discussed? | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | |
Do the authors generalize the conclusions to other settings or patient/client groups? | 0.5 | 1 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 1 | 0.5 | 0.5 | |
Domain Score | 2.5 | 3 | 2.5 | 2 | 2.5 | 2.5 | 2.5 | 2.5 | 2.5 | 2.5 | 2.5 | 3 | 2.5 | 2.5 | |
Overall score for quality assessment in percent | 89 | 92 | 74 | 81 | 83 | 82 | 85 | 87 | 43 | 48 | 73 | 86 | 84 | 81 | |
Global impression of the quality of the article | Very good | Excellent | Good | Very good | Very good | Very good | Very good | Very good | Poor | Fair | Good | Very good | Very good | Very good | |