Another Caution for Difference-in-Differences: Expected Gains

doi:10.21203/rs.3.rs-5000288/v1

Download PDF

Method Article

Another Caution for Difference-in-Differences: Expected Gains

https://doi.org/10.21203/rs.3.rs-5000288/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Many interventions are based on voluntary participation in the treatment group and difference-in-differences (DID) models frequently are used to estimate the effect of the treatment on treatment group versus the untreated control group. Expected gains in the form of resolve or capacity to adhere to the intervention are likely to be unobserved by the analyst and affect outcomes only after the subject learns the actual content of the intervention effect. When an omitted variable is both time-varying and subject-varying, it will not be undetectable by all the usual DID specification tests, including tests of the parallel trends assumption, and will not be corrected by the standard two-way fixed effect model. Both the internal and external validity of estimated treatment effect can be threatened, whether the estimates are biased from a policy standpoint depends on how the intervention will be expanded if it proves to be successful. When the analyst suspects that unobserved expected gains are a source of bias in a DID model, there are a number of appropriate econometric methods available that double as specification tests. We provide a simulation example to show how the problem arises, and how it can be addressed.

Difference-in-differences

Research Design

Expected Gains

Analytic Methods

Econometricians have had a long-standing interest in correcting for the effects of unobserved confounders (spurious correlation, omitted variable bias) in the analysis of causal effects. Ronald Fisher introduced randomized, controlled trials in 1926 (Fisher 1926); Philip Wright introduced the concept of instrumental variables in 1928 (Wright 1928; Stock and Trebbi 2003); and Heckman (1974) and Lee (1976) developed the sample selection model in the mid-1970s. Natural experiments were the approach du jour in the 1990s and remain popular today. Each of these approaches can be implemented with cross-sectional data in which the subject is observed only once. As the availability of panel data increased, methods that measure causal effects expanded from comparisons across subjects to include differences in the same subject’s responses over time. Foremost among those methods was difference-in-differences (DID). In evaluations of treatment effects, the DID approach compares outcomes ($\:{Y}_{it}$ ) for subjects in the treatment and comparison groups, before and after the application of the treatment.

The simplest version of the DID estimator is:

$$\:{Y}_{it}={\beta\:}_{0}+\:{Treat}_{i}{\beta\:}_{1}+{Post}_{t}{\beta\:}_{2}+({Treat\times\:Post)}_{it}{\beta\:}_{3}{+u}_{it}$$

Where:

Treat = 1 in both the pre-treatment and post-treatment time periods if the subject receives the treatment.

Post = 1 if the observation is from the post-treatment time period

$\:\beta\:s$ are estimated coefficients

$\:{u}_{it}$ = random error

The variable Treat removes the effect of average differences in time-invariant characteristics of subjects in the treatment versus comparison group, and the variable Post removes the effect of average differences in subject-invariant time-related characteristics between the pre- and post-treatment time periods.

In a more flexible form of Eq. 1, Treat is replaced by a vector of fixed effects for each subject (Subject in Eq. 2), and Post in Eq. 1 is replaced by a vector of fixed effects for each time period (Year in Eq. 2) (Bertrand, Duflo, and Mullainathan 2004). (For simplicity, we exclude the intercept and any explanatory variables other than TreatEffect_it that vary across both subjects and time.)

$$\:{Y}_{it}={Subject}_{i}{\beta\:}_{1}+{Year}_{t}{\beta\:}_{2}+{TreatEffect}_{it}{\beta\:}_{3}\:{+u}_{it}$$

The interaction of Treat and Post is replaced by the variable TreatEffect which is equal to one if the subject is a member of the treatment group and the observation is from a post-treatment time period. In this estimator, known as the two-way fixed effect (TWFE) estimator, the subject and time fixed effects are designed to isolate the effect of the treatment from the effect of time-invariant subject effects and subject-invariant time effects. The TWFE estimator is the most popular DID approach, although many variations have been developed in recent years to improve estimation of treatments in a broader range of settings, including those with a staggered rollout, growing or decaying effects, and treatments that turn on and off. Callaway and Sant’ Anna 2024, Borusyak and Jaravel 2017; Sun and Abraham 2020; de Chaisemartin and D’Haultfoeuille 2020). Our discussion of unobserved expected gains applies to the simplest DID model: an intervention that is applied continuously, starting at the same time for all members of the treatment group in the post-treatment period, without consideration of growing or decaying effects.

DID estimators are identified by consistency, that is, a treated unit’s observed outcome equals the counterfactual treated outcome (Hernan and Robbins 2020), irreversibility of the treatment (the treatment does not turn on and off), positivity (for all treated units with a given set of characteristics, there was a non-zero chance of being in the comparison group), and counterfactual assumptions, e.g., parallel trends in untreated potential outcomes across treatment and comparison groups, and no unique shocks (defined below).

The parallel trends assumption states that, absent treatment, the treatment group’s outcome would evolve in a manner similar to the control group’s outcome in the post-treatment period. Although the assumption cannot be directly tested, investigators commonly test the likelihood of the assumption being true by examining whether the outcome variable evolves in a similar manner in the two groups in the pre-treatment period. When the parallel trends test is satisfied, the comparison group’s time trend in the post-treatment period is assumed to be a valid counterfactual for the treatment group’s post-treatment trend in the absence of the treatment.

Daw and Hatfield 2018) and Zeldow and Hatfield (2021) discuss the ways in which confounding can arise in a DID analysis. They focus on observed confounders and define a confounder as “a variable with a time-varying effect on the outcome or a time-varying difference between groups. In the presence of confounding, the ATT estimates both the effect of the treatment as well as the magnitude of selection bias (Rambachan 2024).

However, there is an additional concern regarding unobserved variables that impact both treatment probability and outcomes. Equations 1 and 2 assume that u_it is uncorrelated with the treatment effect variable. This is the familiar common shocks assumption that rules out the occurrence of some unobserved outcome-shifting event that occurs at the same time the treatment is applied and affects the treatment and comparison groups differently. Expected gains are one source of unobserved confounding.

Heckman, Urzua, and Vytlacil (2006) and Heckman, Schmierer, and Urzua (2010) discussed the problem of subjects’ unobserved anticipated gains from an intervention in cross-sectional data. In the context of a job training program, the concern was that subjects with higher unobserved expected gains from the training would be more likely to apply for participation in the training program, and also more likely to have higher post-training earnings than non-applicants. The authors referred to the general problem as “essential heterogeneity.” The degree to which expected gains impacts the validity of inferences from a DID model depends on whether the effect of expected gains varies with time, as well as on the population to whom one wishes to generalize the results.

In the following sections, we define expected gains in the context of DID models, illustrate the difficulties in detecting expected gains, discuss when expected gains threaten the validity of inferences, and provide potential solutions. Because much of the DID literature uses potential outcomes notation, and much of the expected gains literature uses two-stage regression model notation, we use both to illustrate concepts of interest.

3a. Potential outcomes framework

Following Ghanem, Sant’Anna, and Wüthrich (2024), we define G as membership in the treatment group. The probability that G = 1 is a function of unobserved time-invariant expected gains (U). For simplicity, let G = 1 if U > 0 and G = 0 if U ≤ 0. Let ε be some positive value of U. Our DID estimate can be written as:

{E[Y(post) | G = 1] – E[Y(pre) | G = 1]} - {E[Y(post) | G = 0] – E[Y(pre) | G = 0]}

the ATT of G. Let the variable D represent receipt of the treatment. D = 1 for members of the treatment group in the post-treatment period, and D = 0 otherwise. The observed outcomes in the treated group are a function of both receipt of treatment and expected gains. That is,

E[Y(post) | G = 1] = E[Y(post) | D = 1] + E[Y(post) | U = ε]

and E[Y(pre) | G = 1] = E[Y(pre) | D = 1] + E[Y(pre) | U = ε]

When there are no expected gains, E[Y(post) | G = 1] = E[Y(post) | D = 1] and the ATT of G equals the ATT of D. If membership in the treatment group is effectively randomly determined, the ATT of D can be generalized to the ATE of D. Similarly, Gupta, Martinez, and Navathe (2023) conceptualize the ATT as being composed of the ATE and the effect of expected gains.

If the effect of expected gains on the outcome is time-invariant (E[Y(post) | U = ε]

= E[Y(pre) | U = ε], the ATT of G will also equal the ATT of D, and no further adjustment is needed:

{E[Y(post) | G = 1] – E[Y(pre) | G = 1]} - {E[Y(post) | G = 0] – E[Y(pre) | G = 0]} =

{( E[Y(post) | D = 1] + E[Y(post) | U = ε]) – (E[Y(pre) | D = 1] + E[Y(pre) | U = ε]) } - {E[Y(post) | D = 0] – E[Y(pre) | D = 0]} =

{E[Y(post) | D = 1] – E[Y(pre) | D = 1]} - {E[Y(post) | D = 0] – E[Y(pre) | D = 0]} =

the ATT of D.

However, if the effect of expected gains on the outcome varies with time (E[Y(post) | U = ε] > E[Y(pre) | U = ε]), Fig. 1), the ATT of G will no longer equal the ATT of D:

{E[Y(post) | G = 1] – E[Y(pre) | G = 1]} - {E[Y(post) | G = 0] – E[Y(pre) | G = 0]} = {( E[Y(post) | D = 1] + E[Y(post) | U = ε]) – (E[Y(pre) | D = 1] )} - {E[Y(post) | D = 0] – E[Y(pre) | D = 0]}.

What sort of unobserved expected gains might affect membership in the treatment group and have time-varying effects on outcomes? In a job training or smoking cessation program, or a difficult treatment regimen that entails active participation by the patient, a subject’s resolve to faithfully attend all the sessions, absorb the training material, and follow the instructor’s advice, might not manifest itself at all in the pre-treatment period. Only when the subject’s resolve is combined with the information and instructions that are part of the intervention does higher resolve affect the outcome. Similarly, some subjects may know that they have an inherent capacity such as personal or human capital that allow them to respond better to the prescribed treatment. Those resources also may be unobserved to the analyst and manifest themselves only in the post-treatment time period and only for subjects in the treatment group.

Essential heterogeneity is possible for units of analysis besides individuals. Alternative payment model interventions conducted by the Center for Medicare and Medicaid Innovation, often rely on volunteer enrollment and where unobserved organizational-level expected gains may be associated with both participation and changes in outcomes.

If unobserved expected gains affect outcomes only in the post-treatment period, the effect of expected gains will not be detected by any comparison of pre-treatment trends in the treatment and comparison groups (Fig. 2). The analyst will proceed, assuming the post-treatment experience of the comparison group is a valid counterfactual for the treatment group in the absence of the treatment.

3b. Two stage estimation methods

Any model of a subject-specific endogenous explanatory variable, e.g., an unobserved confounder, consists of at least two equations. The first equation, often referred to as the sample selection equation, describes how the subject is assigned to the treatment or comparison group. The second equation is the outcome equation or “equation of interest.” Either equation can be estimated using any appropriate functional form (Lee 1983).

Equations 2a and 2b show a different way of writing the DID model. Again, we assume that the treatment is administered at the same point in time for all subjects in the treatment group. The variable $\:{w}_{i}$ represents the subject’s expected gain and appears in both equations and $\:{v}_{i}$ is a “clean” error term, uncorrelated with $\:{w}_{i}$.

$\:{Treat}_{i}=f(\gamma\:{w}_{i}+{v}_{i}\:)$ (treatment selection equation) (2a)

$$\:{Y}_{it}={Subject}_{i}{\beta\:}_{1}+{Year}_{t}{\beta\:}_{2}+{TreatEffect}_{it}{\beta\:}_{3}{+(\delta\:}_{TreatEffect}{w}_{it}+\:{u}_{it})$$

(outcome “equation of interest”) (2b)

The coefficient on w in Eq. 2a, (γ), represents the causal effect of expected gains on selection into the treatment group. The coefficient on w in Eq. 2b, ($\:\:{\delta\:}_{TreatEffect})$ represents the causal effect of expected gains on the outcome variable. Notice that in Eq. 2b, w has acquired a t subscript, indicating that expected gains affects Y only in the post-treatment period.

If w retained only an i subscript in the outcome equation, it’s effect would be cancelled out by the pre-post comparison of the same individual. But because w has both an i and a t subscript, its effect on outcomes cannot be addressed by subject or time fixed effects, because the effect of expected gains exists only for subjects in the treatment group, and only in the post-treatment time period. Thus, the effect of unobserved expected gains must be addressed through econometric techniques including instrumental variables, sample selection models, two-stage residual inclusion (Terza 2018), and regression discontinuity (Cook 2008).

3c. Illustration with Simulated Data

A simulated dataset can illustrate the impact of time-varying effects of expected gains on estimates. Using Stata, we generate a dataset with 10,000 individual observations, each followed for 20 time periods (see appendix for details of the data generating process and results). Years 10–20 are considered the post-treatment period. Receipt of treatment is modeled as a function of expected gains, an instrumental variable Z that affects assignment to the treatment group but has no direct effect on the outcome variable, and a random error term. We then generate untreated and treated potential outcomes, where the treated potential outcomes among recipients of the treatment are a function of a time-varying effect of expected gains. We set the true treatment effect to zero so that the only effect on the outcome is due to expected gains.

3d. When Do Expected Gains Threaten the Validity of Inferences?

Does the time-varying effect of unobserved expected gains on outcomes in in DID models necessarily represent bias? That depends on the research question. The research question will determine the population to which one wants to generalize the results from the evaluation intervention (the target population), and the target population, in turn, depends on what policymakers intend to do if the initial evaluation suggests that the intervention is successful (Imai, King, and Stuart 2008; Lesko, et al., 2017).

The potential effect of expected gains on program effects can arise through non-random assignment to treatment, thus threatening internal validity (Abadie, et al., 2020), as well as through non-random sampling, thus threatening external validity). However, the threats to internal validity only have a practical impact if one wishes to generalize the treatment effect estimate to a population with a different treatment assignment mechanism. If the intent is to continue offering the program on a voluntary basis to a new, perhaps larger, but similar population, then the research question essentially concerns the effect of offering the treatment (intention to treat). In that case, the ATT for the voluntary participants in the evaluation sample need exhibit only internal validity. In the expanded application, subjects with relatively high expected gains will continue to enroll in the treatment group, thus replicating the results from the original evaluation. The estimated ATT will capture the effect of the program on a population of volunteers with higher than average expected gains relative to the comparison group with lower expected gains. No correction for the effect of expected gains is needed in this case.

However, if policymakers intend to expand the program to a population with different characteristics, perhaps by mandating participation in the treatment, the most policy relevant parameter would be the ATE for the entire population. The ATE will capture the effect of the treatment for the entire target population with average expected gains, not just volunteers. That ATE could be obtained from a randomized, controlled trial (RCT) conducted on the expanded target population, but an RCT may be politically difficult, expensive, or unethical (Black 1996). Another option would be to collect additional data, presumably survey data, on subjects’ expected gains, and control for that variable in the DID analyses. However, that approach also may be infeasible. In some cases, the analyst must turn to econometric corrections for unobserved confounders, remembering that methods such as instrument variables estimate only a local average treatment effect for the subpopulation whose membership in the treatment versus control group is influenced by the value of the instrument (Imbens and Angrist 1994; Imbens and Rubin 1007).

Although our focus is on the simplest DID model, expected gains also influence analyses that use more flexible DID estimators. Some estimators allow for parallel trends after conditioning on covariates – both pre and post, in the case of the two-way Mundlak estimator, and only in the post period, in the case of the Callaway & Sant’Anna estimator. However, these methods do not adjust for unobserved confounding. Although coefficients on treatment indicators in the two-way Mundlak estimator allow an investigator to “study the nature of selection bias into exposure”, they do not allow one to isolate the effect of a treatment from the influence of selection (Wooldridge 2021). Similarly, robustness tests meant to assess the degree to which results of DID models that assume parallel trends differ from results of models that allow time trends to differ for treatment and comparison groups will not allow the analyst to isolate the impact of unobserved, time varying, treatment group-specific confounders like expected gains (Bilinski and Hatfield 2020).

The literature on unobserved confounders in cross-sectional observational data studies is well-developed, but less so in panel data, including DID estimators. DID estimators are a useful way to deal with unobserved confounders in the form of time-invariant characteristics of subjects and subject-invariant time characteristics, but they are inadequate to correct for unobserved confounders that vary across the combination of time and subjects.

The subject’s unobserved expected gain from an intervention is a time-varying, and subject-varying source of essential heterogeneity that can occur exactly when the treatment is applied and has different effects for members of the treatment and comparison groups. As in all models of endogenous assignment to the treatment group, the effect of essential heterogeneity depends on what will be done next if the initial evaluation suggests that treatment is successful. If expansion of the intervention will involve offering the intervention with voluntary assignment to a similar population, then the results that incorporate expected gains will provide a policy-relevant estimate of the expected outcome. But if the expansion involves a population with different characteristics, e.g., mandated versus voluntary participants, then treatment effect estimates will need to correct for unobserved expected gains through randomization, collection of additional data, or econometric techniques.

Author Contribution

Both authors wrote sections of the manuscript and accompanying Stata code.

Data Availability

Our submission include Stata code to run a simulation. (I'm not sure if that's what you're looking for.)

Abadie, A., Athey, S., Imbens, G.W., Wooldridge, J.M.: Sampling-based versus design-based uncertainty in regression analysis. Econometrica. 88(1), 265–296 (2020)
Baiocchi, M., Cheng, J., Small, D.S.. Instrumental variable methods for causal inference. Stat Med. ;33(13):2297–2340.Bertrand, Marianne, Duflo, Ester, and, Mullainathan, S.: How Much Should We Trust Difference-in-Differences Estimates? Quarterly Journal of Economics 119:1 (February 2004) 249–275. (2014)
Bilinski, A., Hatfield, L.A.: Nothing to see here: Non-inferiority approaches to parallel trends and other model assumptions. Published online 2020. https://arxiv.org/abs/1805.03273
Black, N.: Why we need observational studies to evaluate the effectiveness of health care. BMJ. 312, 1215 (1996)
Borusyak, K., Jaravel, X.: Revisiting Event Study Designs. SSRN Work Pap. Published online May 8, (2017). https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2826228
Callaway, B., Sant’Anna, P.H.C.: Difference-in-Differences with multiple time periods. J. Econom. 225(2), 200–230 (2021). 10.1016/j.jeconom.2020.12.001
Cook, T.D.: Waiting for Life to Arrive: A history of the regression-discontinuity design in Psychology, Statistics and Economics. J. Econ. 142(2), 636–654 (2008)
de Chaisemartin, C., D’Haultfoeuille, X.: Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects. Am. Econ. Rev. 110(9), 2964–2996 (2020). 10.1257/aer.20181169
Daw, J.R., Hatfield, L.A.: Matching in Difference in Differences: between a Rock and a Hard. Place Health Serv. Res. 53(6), 4111–4117 (2018)
Fisher, R.: ‘The Arrangement of Field Experiments’. J. Ministry Agric. 33, 500–513 (1926)
Ghanem, D., Sant’Anna, P.H.C., Wüthrich, K.: Selection and parallel trends. arXiv:2203.09001v9. (2024)
Gupta, A., Martinez, J.R., Navathe, A.S.: Selection and causal effects in voluntary programs: Bundled payments in Medicare. NBER working paper series. Working paper 31256. ; (2023). http://www.nber.org/papers/w31256
Heckman, J.: Shadow Prices, Market Wages, and Labor Supply. Econometrica. 42(4), 679–694 (1974)
Heckman, J.J., Urzua, S., Vytlacil, E.: Understanding instrumental variables in models with essential heterogeneity. Rev. Econ. Stat. 88(3), 389–342 (2006)
Heckman, J.J., Schmierer, D., Urzua, S.: Testing the correlated random coefficient model. J. Econ. 158, 177–203 (2010)
Hernán, M.A., Robins, J.M.: Causal Inference: What If. Chapman & Hall/CRC, Boca Raton (2020)
Imai, K., King, G., Stuart, G.: Misunderstandings between experimentalists and observationalists about causal inference. J. Royal Stat. Soc. Ser. A. 171(2), 481–502 (2008)
Imbens, G.W., Angrist, J.D.: Identification and Estimation of Local Average Treatment Effects. Econometrica. 62(2), 467–475 (1994)
Imbens, G.W., Donald, B., Rubin: Estimating Outcome Distributions for Compliers in Instrumental Variables Models. Rev. Econ. Stud. 64, 555–574 (1997)
Lee, L.F.: Estimation of Limited Dependent Variables by Two Stage Method, unpublished Ph.D. thesis, Department of Economics, University of Rochester. (1976)
Lee, L.: Generalized Econometric Models Selectivity Econometrica. 51(2), 507–512 (1983)
Lesko, C.R., Buchanan, A.L., Westreich, D., Edwards, J.K., Hudgens, M.G., Cole, S.R.: Generalizing study results: A potential outcomes perspective. Epidemiology. 28(4), 553–561 (2017)
Rambachan, A., Roth, J.: A more credible approach to parallel trends. Rev. Econ. Stud. (2023)
Stock, J.H., Trebbi, F.: Retrospectives: Who Invented Instrumental Variable Regression? J. Economic Perspect. 17(3), 177–194 (2003)
Sun, L., Abraham, S.: Estimating Dynamic Treatment Effects in Event Studies with Heterogeneous Treatment Effects. ArXiv180405785 Econ. Published online September 22, 2020. http://arxiv.org/abs/1804.05785
Terza, J.V.: Two-stage residual inclusion estimation. Health Serv. Res. Health Econ. 53(3), 1890–1899 (2018)
Wooldridge, J.M.: Two-Way Fixed Effects, the Two-Way Mundlak Regression, and Difference-in-Differences Estimators. SSRN Electron. J. Published online. (2021). 10.2139/ssrn.3906345
Wright, P.: The Tariff on Animal and Vegetable Oils: Appendix B. 1928. Reproduced in Stock and Trebbi (2003)
Zeldow, B., Hatfield, L.A.: Confounding and egression adjustment in difference-in-differences studies. Health Serv. Res. 1–10 (2021). https://doi.org/10.1111/1475-6773.13666

No competing interests reported.

StatadofileinWord.docx

Download PDF

Reviewers agreed at journal
24 Sep, 2024
Reviewers agreed at journal
24 Sep, 2024
Reviewers agreed at journal
23 Sep, 2024
Reviewers invited by journal
23 Sep, 2024
Editor assigned by journal
23 Sep, 2024
Submission checks completed at journal
05 Sep, 2024
First submitted to journal
29 Aug, 2024

You are reading this latest preprint version

Another Caution for Difference-in-Differences: Expected Gains

Status:

Version 1

Abstract

Figures

1. Introduction

2. Methods

3. Results

3a. Potential outcomes framework

3b. Two stage estimation methods

3c. Illustration with Simulated Data

3d. When Do Expected Gains Threaten the Validity of Inferences?

4. Conclusion and Discussion

Declarations

Author Contribution

Data Availability

References

Additional Declarations

Supplementary Files

Status:

Version 1