Which indicator should be used? A comparison between the incidence and intensity of catastrophic health expenditure: using difference-in-difference analysis

Abstract

Background

Catastrophic health expenditure (CHE) represents out-of-pocket payment as a share of household income. Most previous studies have focused on incidence aspects when assessing health policy effects. However, because CHE incidence is a binary variable, the effect of the health policy could not accurately be evaluated. On the contrary, the intensity of CHE is a continuous variable that can yield completely different results from previous studies. This study reassesses the coverage expansion plan for four serious diseases using the intensity of CHE in Korea.

Methods

We used the Korea Health Panel Study from 2013 to 2015 to conduct the analysis. The study population is households with chronic diseases patients. We divided the population into two groups: the policy beneficiary group, i.e., households with a patient of any of the four serious diseases, and the non-beneficiary group. A difference-in-difference model was employed to compare the variation in the intensity and incidence of CHE between the two groups. We defined the incidence of CHE as when the ratio of out-of-pocket medical expenses to household income is more than a threshold of 10%, and the intensity of CHE is the height of the ratio subtracting the threshold 10%.

Results

The increased rate of CHE intensity in households with four serious diseases was lower than that in households with other chronic diseases. The interaction term, which represents the effect of the policy, has a significant impact on the intensity but not on the incidence of CHE.

Conclusions

CHE indicators should be applied differently according to the purpose of policy evaluation. The incidence of CHE should be used as the final achievement indicator, and the intensity of CHE should be used as the process indicator. Furthermore, because CHE has an inherent characteristic that is measured by the ratio of household income to medical expenses, to lower this, a differential out-of-pocket maximum policy for each income class is more appropriate than a policy for strengthening the coverage for specific diseases.

Background

Catastrophic Health Expenditure (CHE) incidence is the proportion of households in which the ratio of out-of-pocket (OOP) medical expenses to their payment ability exceeds a certain threshold in the whole population [1,2,3,4]. Because the indicator has the advantage of collectively measuring the medical cost burden of households, it can be a useful tool for comparing healthcare coverage among countries. Thus, even the World Health Organization and the World Bank have adopted it as an indicator of financial risk protection [1]. Many researchers have used this indicator to evaluate the effect of health policy or systems that aim to lower the medical cost burden of patients [5,6,7,8,9,10,11,12].

However, recent studies have pointed out the significant inherent limitations of the indicator. Jung and Lee [13] conducted a systematic review of CHE, reviewing the time series trends of reported incidence and intensity of CHE in 34 countries. Intensity is a paired indicator with incidence [2], although it has not been used as much as incidence in previous studies. It measures the ratio of OOP medical expenses to the ability to pay (more accurately, it subtracts a certain threshold from the ratio). Jung and Lee [13] confirmed that CHE incidence in most countries does not significantly fluctuate in the time series, whereas the intensity is flexible. Furthermore, they reviewed four studies that employed difference-in-difference (DID) analysis to evaluate the healthcare system’s newly launched coverage expansion plan in different countries and confirmed that only one study found a significant decrease in CHE. In fact, many studies in Korea that assessed the effect of the benefit expansion policy of National Health Insurance (NHI) using the DID test concluded that there was a statistically significant effect on medical expenditure but no effect on CHE incidence [14,15,16,17,18]. Consequently, Jung and Lee [13] argued that incidence has an inherent limitation—it does not change and cannot sensitively reflect the policy effect. Additionally, they proposed intensity as an alternative indicator.

However, because Jung and Lee [13] conducted a systematic review, they could not demonstrate whether the reason why most DID analysis results did not reveal statistically significant policy effects was due to the intrinsic characteristics of CHE incidence. Moreover, they could not confirm whether the intensity of CHE can lead to opposite results when CHE incidence is used. Therefore, this study theoretically explains why CHE incidence inherently does not vary in the time series but intensity does.

CHE incidence is a dichotomous variable. Therefore, even if the policy reduces the OOP medical expenses, households could still incur CHE when the proportion of OOP expenditure in household income exceeds the threshold. The reduced medical spending could not be reflected in the result, except for households that are close to the threshold. Figure 1 depicts that if policy effectiveness is determined only based on CHE incidence, the results of the analysis could be completely reversed, although Policy A is more effective than Policy B. However, because the intensity of CHE is a continuous variable, it is free from this problem.

Fig. 1

Limitation of using the incidence of CHE to analyze the policy effect

This study contributes to the literature by demonstrating the arguments suggested by Jung and Lee [13] that results can vary depending on whether incidence or intensity is used when analyzing policy effect by conducting the DID test.

Many studies conducted propensity score matching (PSM) and DID tests to analyze the benefit expansion plan of NHI in Korea. They all reported no policy effect based on the incidence of CHE [16,17,18,19,20]. Since 2005, the Korean government has tried to strengthen the coverage of NHI for cancer and gradually extended the coverage by including other diseases (cardiovascular, cerebrovascular, and rare/intractable diseases). Coverage strengthening started by lowering the statutory copayment rate for patients with cancer from 20 to 10% in September 2005 and further reducing it to 5% in 2009 [21]. Additionally, the NHI reduced the copayment rate to 5–10% for rare and intractable, heart, and cerebrovascular diseases [21].

The problem was that the services not covered by the NHI were not captured in the statutory copayment rate. These services, such as magnetic resonance imaging, computerized tomography scans, chiropractic, and genetic testing for cancer treatment, are out of the NHI’s scope [22]. Such services are not subject to the fee regulation of the NHI but are set by medical providers, so they are likely to become expensive.

Therefore, the Korean government announced a comprehensive benefit expansion policy, “the benefit enhancement plan for four major diseases,” in June 2013 and implemented it in October [22, 23]. The policy allowed the NHI to cover the previously non-covered 25 items, including ultrasonic diagnostic tests and magnetic resonance imaging for heart diseases, in 2013, 100 in 2014, and 258 in 2015 [22].

This study evaluates the NHI’s benefit expansion policy for four serious diseases in Korea by using the incidence and intensity of the CHE indices and demonstrates that the results can be different between the two indicators. If the policy is found to be effective when the intensity of the CHE index is used, we can argue that the consistent negative results of the existing studies that analyzed the policy effect are due to the insufficiency of measurement methods.

Methods

Data sources

This study uses the Korea Health Panel Study (KHPS) version 1.5 database, which is jointly conducted by the Korea Institute for Health and Social Affairs and the National Health Insurance Service. Since 2008, the study has been conducted annually to analyze medical use behavior and expenditure in Korea. The KHPS is an official statistical database with a nationally representative sample extracted through a two-stage stratified cluster sampling process with probability proportionality based on the Korea Population and Housing Census. The KHPS includes healthcare utilization and expenditure based on the receipts of healthcare expenses and housekeeping books. Each healthcare expenditure is divided into three parts: copayment, insurance benefit for insured services, and uncovered services. In 2008, the KHPS started with 7866 households (24,616 individuals), and in 2012, the retention rate of the original sample was 70.6%. In 2013, 2222 households (6454 individuals) were added to the original sample, and 6983 households (18,834 individuals) were surveyed in 2015.

Study population

This study analyzes the policy effect of the benefit expansion for four serious diseases in 2014. Some studies in Korea consider 2013 as the year when the policy was introduced and compare the period before 2012 and after 2014 [16,17,18,19,20,21]; however, the policy was implemented in October 2013, after the third quarter. Moreover, only 25 items were newly covered. Therefore, we believe it is insufficient to analyze the policy effect with 2013 as the base year. In fact, studies that used 2013 as the base year had inconsistent results [24]. Conversely, studies that used 2014 as the base year reported significant policy effects [25].

Therefore, we selected households with patients of chronic diseases in 2013 and 2015 as the study population. The policy beneficiary group (treatment group) is defined as a household with a member who has used medical care (emergency room, outpatient, or inpatient) for any of the four serious diseases at least once a year in 2013 and 2015. The four types of diseases were announced by the Ministry of Health and Welfare in Korea. We converted them into the international classification of diseases and presented the results as supplementary material. Figure 2 depicts the specific sampling process.

Fig. 2

Sampling process

Independent variables

The independent variables are based on Andersen’s health care utilization model [26], and we separate householder and household levels. The KHPS defines the head of a household as a householder and otherwise as a nonhouseholder. The householders’ characteristics include gender, elderly, education level, spouse, and whether they are economically active. The households’ characteristics include living alone, equivalized income, private health insurance, NHI, presence of disabled persons, and presence of any of the four serious diseases. The independent variables are shown in Table 1.

Table 1 Variable lists

Dependent variable

The dependent variables are the intensity and incidence of CHE, which is incurred when the proportion of OOP medical expenses in household income exceeds a certain threshold. In this study, we use a household’s total income to measure their ability to pay. The definition of OOP medical expenses varies depending on the characteristics of the data sources. In the KHPS, it is common to include only direct medical expenses, which are the minimum expenses incurred based on doctors’ opinions [27]. OOP medical expenses include medical and drug costs incurred on utilizing healthcare in the emergency room as well as outpatient and inpatient services. We set the CHE threshold level at 10%, which is the lowest value suggested by Wagstaff and van Doorslaer [2]. They suggested threshold values ranging from 10 to 40% when defining the ability to pay as total household income. Nevertheless, we chose 10% because many recent empirical and review studies suggested it as a representative threshold [28]. Moreover, if the threshold is set too high, the height of intensity and number of households with CHE will be too small, thereby making it insufficient for analysis. we calculated the incidence and intensity of CHE based on the above definitions of OOP medical expenses, the household’s ability to pay, and threshold level of CHE. The calculation is presented in the Table 1.

Statistical analysis

We used the paired t-test and DID analysis to identify the effectiveness of the policy. Particularly, we performed the DID test to confirm the health policy effects by estimating two models. One model used CHE incidence as the dependent variable, and the other used CHE intensity. The two models were set to compare the characteristics of the two indicators. Both models include the time before and after the policy was implemented and whether the group is a policy beneficiary as follows:

$${Y}_i={\beta}_0+{\beta}_1T+{\beta}_2{P}_i+{\beta}_3T{P}_i+{\beta}_4{X}_i+{\varepsilon}_i$$

Y _i could be the incidence of CHE or refer to the intensity of CHE; T, the time variable comprises 0 and 1, representing 2013 and 2015, respectively. P_i, the policy variable also comprises 0 and 1, indicating whether a group is a policy beneficiary (households with patients of four serious diseases) or not, respectively. X_i is the covariate variable. Before testing the policy effectiveness, we conducted a PSM and balancing test of the covariates to determine the similarity between the treatment and control groups. It uses the conditional probability of participating in the program under certain observable variables [29]. For its beneficial practical implications, PSM has gained popularity in evaluating healthcare programs [30, 31], health finance and insurance schemes [32], and many other areas of evaluation research. Rosenbaum and Rubin [29] suggested the covariate balance test because matching is based on propensity scores but not on covariates themselves. All analyses were performed using the Stata/SE version 16.0 software (Stata Corp., Texas, USA).

Results

General characteristics

Table 2 shows the general characteristics of the treatment and control groups in 2013. We found significant relationships between treatment and all variables, except living alone, through the bivariate analyses.

Table 2 General characteristics of households before treatment

PSM and covariate balance test

We conducted PSM with replacement and checked the balance of covariates. We used the Stata command “teffects psmatch” and option “nn(2).” A balance between the treatment and control groups for each variable was tested using absolute standardized differences. Although there is no officially agreed-upon threshold level, an absolute standardized difference of < 0.1 has been considered a good balance of covariates [33]. The result of the balance test reveals that all covariates’ absolute standardized differences are less than 0.1 (Table 3).

Table 3 Covariate balance test after propensity score matching

Paired t-test

Table 4 presents the results of the paired t-test of OOP medical expenses and household income after PSM. The analysis results reveal that there was an increase in OOP medical expenses in the non-beneficiary group from 2013 to 2015.

Table 4 Paired t-test

Analyzing policy effects on CHE using DID

As we mentioned above, there are two dependent variables in this study (intensity of CHE and incidence of CHE). Table 5 presents the results of the DID analyses for each dependent variable. After analyzing the effectiveness of the policy intervention while controlling the characteristics of households and householders, we found that the results of each group are completely different. The interaction term, which reveals the pure effect of the policy, has a significant impact on the intensity of CHE but not on its incidence.

Table 5 Difference-in-difference analyses

Discussion

This study aims to demonstrate that the policy effect result of the DID test can differ between the incidence and intensity indicators. For empirical evidence, we analyzed the case of the Korean policy—the 2014 benefit enhancement plan for four serious diseases.

Table 5 indicates that the absolute value of CHE intensity in the policy beneficiary group is higher than in the non-beneficiary group; however, after the policy, the gap in the intensity of CHE between the two groups has decreased, thereby implying that policy effects exist. This result differs from that of previous studies, which used the incidence of CHE as a dependent variable [16,17,18, 21, 25]. Furthermore, CHE incidence reveals no policy effect as expected. This finding supports Jung and Lee [13] that CHE incidence does not change in nature and cannot sensitively reflect the policy effect, and intensity can be an alternative indicator.

We can infer that the intensity of CHE is less affected by predisposition factors than the incidence of CHE (Table 5). These results reveal that intensity is a more appropriate indicator than incidence when estimating the level of financial burden of households. Because the unit of incidence is the ratio of the number of households to the whole population, it varies when more people are sick, even without any change in medical prices. Conversely, intensity is the ratio of OOP medical expenses to payment ability among households that incurred CHE incidence. Therefore, choosing intensity when conducting a microscopic study that analyzes the effect of a specific policy and/or considering the time series variation is appropriate.

Notably, assessing the effectiveness of the policy using only the intensity of CHE is insufficient because it is a sensitive indicator that can be interpreted as “effective” even with a slight reduction in OOP medical expenses. Because the intensity is a continuous variable, there is no standard for how much intensity should be reduced to be determined as having a policy effect. Conversely, because the incidence of CHE is measured as a dichotomous variable based on a specified threshold, we can determine whether the policy is effective using the threshold. Therefore, it is essential to distinguish between the variables when assessing the policy to expand health insurance coverage. When the NHI aims to eliminate households with CHE, the reduction in CHE incidence can be a final performance indicator, and the reduction in CHE intensity can be used as an intermediate indicator that determines proximity to the goal.

It is necessary to grasp the policy effect from various perspectives, apart from determining whether the policy intended to strengthen the coverage of some diseases is desirable. Thus, it is important to not only focus on CHE incidence but also analyze how much burden of medical expenses has been reduced even though CHE is already incurred. After considering the intensity of CHE, it is possible to interpret that the burden of medical costs is alleviated.

Furthermore, we discovered two unexpected essential results. First, focusing only on CHE would ignore the context of the policy. The benefit enhancement policy we analyzed aims to reduce the OOP burden of households with a patient of any of the four serious diseases in Korea. To confirm the policy effect in this context, the medical cost of the beneficiary group should be less than that of the non-beneficiary group. However, as Table 4 indicates, the OOP medical expenses in the treatment group did not decrease. Instead, it increased in the non-beneficiary group. Therefore, the results of the DID test in Table 5 can be due to the increase in the OOP medical expenses of the non-beneficiary group and not due to the decrease in the OOP medical expenses of the beneficiary group. It can be interpreted that the OOP medical expenses burden of households with a patient of any of the four serious diseases was transferred to households with other diseases. If the government tries to control the medical costs of some specific diseases, medical institutions can increase the cost of other diseases. Thus, the basic statistics for medical expenses are important when using CHE indicators.

Second, regardless of incidence or intensity, CHE is a combination of two variables—medical expenses and payment ability. Although there was no statistically significant variance in household income between 2013 and 2015, that of the control group decreased substantially (Table 4), which can affect the value of CHE. Jung and Lee [13] suggested that reducing CHE is only achieved by implementing policies that consider income levels, such as medical aid for the poor or a system that differentially charges medical expenses according to income levels.

The concept of CHE originated with the cost of serious diseases (catastrophic illness), which significantly interfere with life [34]. Havighurst, Blumstein, and Bovberjerg [35] argued that the policy should be altered to reduce the high medical costs that persist because economically burdensome diseases can be chronic diseases that require continuous treatment and rehabilitation. This argument has led to a trend that emphasizes the relative household income and medical expenses since the late 1970s [36, 37]. Feldstein [38] argued that the NHI should be designed to cover medical expenses of not more than 10% of annual household income, and the U.S. Congressional Budget Office proposed that this should be 15% [39]. Due to the inherent nature of CHE, which is measured by household income and medical expenses, a differential medical cost burden system based on the level of household income should be considered because it is difficult to reduce households with CHE under specific disease-related policies.

This study has a few limitations. First, we define the ability to pay as a household’s total income, which has the largest value among the options suggested by Wagstaff and van Doorslaer [2]. They defined the ability to pay as a household’s total income, household’s total income minus food expenses, total expenditure, or total expenditure minus food expenses. Therefore, the CHE might have been underestimated. Second, the PSM has limitations in that it splits the samples and reduces the size, thereby making it difficult to secure the sample size. Moreover, it makes room for sample bias. However, these limitations are manageable because the sample size after PSM is appropriate for this study and the sample bias was controlled again in the DID regression analysis. Third, the policy to strengthen the coverage of the four serious diseases began in 2005, but the KHPS data became available from 2008. Therefore, the period of the first stage of 2005–2008 was unavailable. Additionally, to analyze the exact policy effect using the DID method, the analysis period should be as short as possible so that the influence of other exogenous factors can be excluded to the maximum possible extent. Therefore, we set the analysis period as 2013, the last year of the second phase, and 2015, the year immediately following the third phase. Lastly, another limitation is that the four diseases were integrated and analyzed to divide the treatment and control groups.

Conclusion

CHE has been widely used as a performance indicator to analyze the effectiveness of health policies in reducing the burden of medical expenses. However, incidence indicators are highly likely to be robust in policy assessment and are not sensitive to the effects of a reduction in OOP medical expense because they only detect changes around the threshold. However, previous studies usually relied on the incidence indicator. This study reevaluates the coverage expansion plan for four serious diseases in Korea using the intensity of CHE. Unlike the previous studies that used the incidence indicator, the results of this study confirm that there is a policy effect. This result is due to the difference in characteristics between the incidence and intensity of CHE. Therefore, policymakers should apply CHE indicators differently based on the purpose of policy evaluation. The incidence of CHE should be used as the final achievement indicator, and the intensity of CHE should be used as the process indicator. Furthermore, because CHE has an inherent characteristic, which is measured by the ratio of household income to medical expenses, to reduce it, a differential OOP maximum policy for each income class might be more appropriate than a policy to strengthen the coverage of specific diseases.