Balancing Precision and Retention
in Experimental Design

 

Gustavo Diaz
Department of Political Science
Northwestern University

gustavodiaz.org

Erin Rossiter
Department of Political Science
University of Notre Dame

erossiter.com

 

Paper and slides: gustavodiaz.org/talk

Bias-variance tradeoff as darts

But the game of darts is more complicated

Application: Social science experiments

Outcome measurement
Randomization Post-only Pre-post
Simple/Complete Standard Repeated measures
Blocking Block randomized Block randomized + repeated measures

Application: Social science experiments

Outcome measurement
Randomization Post-only Pre-post
Simple/Complete Standard Repeated measures
Blocking Block randomized Block randomized + repeated measures

Application: Social science experiments

Outcome measurement
Randomization Post-only Pre-post
Simple/Complete Standard Repeated measures
Blocking Block randomized Block randomized + repeated measures

Application: Social science experiments

Outcome measurement
Randomization Post-only Pre-post
Simple/Complete Standard Repeated measures
Blocking Block randomized Block randomized + repeated measures

Alternative designs improve statistical precision but need access to pre-treatment covariates

Cost may outweigh benefit

Concern: Sample loss

Explicit

  • More pre-treatment questions \(\rightarrow\) more attrition/inattention

  • Block-randomization \(\rightarrow\) discard units

Implicit

  • Adding a baseline survey \(\rightarrow\) half sample size

  • Four more survey questions (2 min.) \(\rightarrow\) 72% sample size

Goal

Show that precision gains offset sample loss

Paper:

  1. Replication of selected studies

  2. Simulation on randomly sampled studies

  3. Simulations/code/advice for pre-analysis stage

Goal

Show that precision gains offset sample loss

Paper:

  1. Replication of selected studies

  2. Simulation on randomly sampled studies

  3. Simulations/code/advice for pre-analysis stage

Procedure

  • Sample one study per journal at random (+3 alternates)

  • Select promising pre-treatment covariates for blocking

  • Simulate pre-treatment outcomes with varying correlation (0.25, 0.5, 0.75)

  • Simulate proportion of sample loss (0-0.5)

  • Estimate 6 different designs

  • Repeat 1,000 times for every design-parameter combination

  • Compare each alternative design with standard

Simulated designs

  1. Complete randomization + post-only outcome measurement (Standard design)

  2. Complete + pre-post

  3. Block on one covariate + post-only

  4. Block on one covariate + pre-post

  5. Block on all covariates + post-only

  6. Block on all covariates + pre-post

Sample

Original experiment
Simulation
Study Type Arms N Country n Blocking Covariates Predictiveness
1 Survey 6 2971 Italy 946 7 Low
2 Survey 12 3013 US 2784 3 High
3 Survey 3 1029 Uganda 561 4 Low
4 Field 2 2942 US 2712 5 Moderate
5 Field 2 275 Colombia 275 7 Low
6 Survey 2 1176 US 1175 2 Moderate

Results: Study 1

Results: Study 1

Results: Study 6

Summary

  • Puzzle: Alternative designs rare

  • Argument: Concerns about explicit/implicit sample loss offsetting precision gains

  • Findings: Alternative designs withstand sample loss

  • Wrinkle: Alternative designs require more attention!

  • Takeaway: Try alternative designs!