Addressing nonignorable attrition with double sampling and bounds: the estimators of Coppock, Gerber, Green, and Kern (2017), Political Analysis 25(2):188-206, for randomized experiments in which some outcomes are missing.
Install the released version from CRAN:
install.packages("attrition")Or the development version from GitHub:
# install.packages("remotes")
remotes::install_github("acoppock/attrition")Attrition occurs when some subjects do not report outcomes or measurements are not recorded for some units. Attrition can happen in many settings, including:
- A two-wave survey experiment in which part of the sample never comes back for the second wave.
- A field experiment in which enumerators cannot recontact some participants after treatment has been allocated.
- More unusual situations, such as a study in which the outcome exists only for subjects who cleared an earlier hurdle, like wages for the subset who found work.
Under attrition, naive estimators are biased for the average treatment
effect. Chapter 7 of Gerber and Green (2012) gives a textbook
introduction to the problem and to the bounding approaches implemented
in the attrition package. Coppock, Gerber, Green, and Kern (2017)
describe a double sampling design for narrowing the bounds.
| Function | What it assumes |
|---|---|
estimator_ev() |
The outcome has known lower and upper limits (Manski 1990). |
estimator_ds() |
The same, plus a random follow-up sample of nonrespondents. |
estimator_ds_sens() |
The same, with ignorability allowed to fail for a fraction delta of the follow-up nonrespondents. |
sensitivity_ds() |
A search over delta for the point where the interval starts to include zero. |
estimator_trim() |
Treatment moves response in one direction only (Lee 2009), or nothing beyond random assignment (Imai 2008). The outcome need not be bounded. |
The first four take a strata argument for poststratification on a
discrete covariate, which targets the same identification region and
estimates it more precisely; estimator_trim() does not. All five have
tidy() methods and a formula interface for use with
DeclareDesign.
The package ships the replication study from the paper as
levendusky_replication: a two-wave survey experiment in which 1,980
subjects were asked about perceived polarization, and 536 of them did
not answer the second wave.
Refusing any assumption about the missing outcomes gives worst-case bounds. Because the outcome runs from 0 to 6, filling every missing value in one group with 0 and the other with 6 gives the lowest effect the data can support, and reversing the fills gives the highest.
library(attrition)
fit_ev <- estimator_ev(Y = Y_polarization_w2,
Z = Z,
R = R1,
minY = 0,
maxY = 6,
data = levendusky_replication)
tidy(fit_ev)
#> # A tibble: 3 × 10
#> term estimate std.error conf.low conf.high estimate_lower estimate_upper
#> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 bounds NA NA -1.67 1.84 -1.54 1.71
#> 2 lower_bou… -1.54 0.0790 NA NA NA NA
#> 3 upper_bou… 1.71 0.0767 NA NA NA NA
#> # ℹ 3 more variables: std.error_lower <dbl>, std.error_upper <dbl>,
#> # outcome <chr>The effect lies somewhere between -1.54 and 1.71, which is honest and nearly useless. Little of that width is sampling error: the confidence interval is barely wider than the bounds. A larger sample would not have helped, because the problem is 536 unknown outcomes rather than noise.
Double sampling addresses the unknown outcomes directly. After the first round of data collection, a random sample of the nonrespondents is drawn and pursued with more effort than the first attempt. Here, 50 nonrespondents per condition were offered $4.00 rather than the original $1.00, and 72 of those 100 answered. Because they are a random sample of the nonrespondents, their outcomes stand in for all 536, and only the 28 who refused twice still need worst-case treatment.
fit_ds <- estimator_ds(Y = Y_polarization_w2,
Z = Z,
R1 = R1,
Attempt = Attempt,
R2 = R2,
minY = 0,
maxY = 6,
data = levendusky_replication)
tidy(fit_ds)
#> # A tibble: 3 × 10
#> term estimate std.error conf.low conf.high estimate_lower estimate_upper
#> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 bounds NA NA -0.528 0.745 -0.342 0.572
#> 2 lower_bou… -0.342 0.113 NA NA NA NA
#> 3 upper_bou… 0.572 0.105 NA NA NA NA
#> # ℹ 3 more variables: std.error_lower <dbl>, std.error_upper <dbl>,
#> # outcome <chr>The identification region shrinks by a factor of 3.6, from 3.25 points wide to 0.91.
vignette("attrition") works through the design and all five estimators
on the shipped data, reproducing the paper’s Table 3 along the way. It
also draws the imputation that estimator_ev() averages over and checks
the picture against the estimates.
attrition 1.0.0 was prepared by Alexander Coppock working with Claude
(Anthropic), across the rewrite of the estimators, the test suite, and
the documentation. The method and the original implementation come from
Coppock, Gerber, Green, and Kern (2017). The 1.0.0 release rewrote the
internals, added a formula interface and tidy() methods, corrected
several defects in the earlier code, and wrote the vignette.
The code base has been reviewed but not written line by line, so the
guarantee offered is not that every line has been vouched for. It is
that the package reproduces the published analysis it implements.
vignette("attrition") reproduces Table 3 of the paper, and the test
suite holds each estimator to the published quantities, which have not
changed. The analytic variance added to estimator_trim() was checked
three ways: against the algebraically distinct form in Tauchmann’s Stata
leebounds, against the Monte Carlo standard deviation of the estimator
across sample sizes from 1,000 to 64,000, and by the coverage of a
conventional interval around each bound endpoint.
Coppock, Alexander, Alan S. Gerber, Donald P. Green, and Holger L. Kern (2017). Combining Double Sampling and Bounds to Address Nonignorable Missing Outcomes in Randomized Experiments. Political Analysis 25(2):188-206. https://doi.org/10.1017/pan.2016.6
Gerber, Alan S., and Donald P. Green (2012). Field Experiments: Design, Analysis, and Interpretation. New York: W. W. Norton.
Imbens, Guido W., and Charles F. Manski (2004). Confidence Intervals for Partially Identified Parameters. Econometrica 72(6):1845-1857. https://doi.org/10.1111/j.1468-0262.2004.00555.x
Lee, David S. (2009). Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects. Review of Economic Studies 76(3):1071-1102. https://doi.org/10.1111/j.1467-937X.2009.00536.x
Manski, Charles F. (1990). Nonparametric Bounds on Treatment Effects. American Economic Review Papers and Proceedings 80(2):319-323.