The Merits of Externally Invalid Survey Experiments

 

Gustavo Diaz
McMaster University
gustavodiaz.org

 

Slides: gustavodiaz.org/talk

Limitations

“Future research should confirm if our findings generalize…”

  • …with a representative sample
  • …in other countries
  • …beyond the survey setting
  • …when using behavioral outcomes

Usual workflow

  1. Research idea

  2. Realize resource/ethical/practical limitations

  3. Conduct experiment with limitations

  4. Wave hands about external validity

Goal

  • Should we ever implement an externally invalid survey experiment on purpose?

  • Identify what makes external invalidity desirable

  • Challenge: Different kinds of external (in)validity

External validity concerns

Type Concern
Samples Does this apply to a different population?
Contexts Does this apply in a different setting?
Treatments Do they resemble real-world phenomena?
Outcomes Do they reflect actual behaviors?

External validity concerns

Type Concern
Samples Does this apply to a different population?
Contexts Does this apply in a different setting?
Treatments Do they resemble real-world phenomena?
Outcomes Do they reflect actual behaviors?

External validity concerns

Type Concern
Samples Does this apply to a different population?
Contexts Does this apply in a different setting?
Treatments Do they resemble real-world phenomena?
Outcomes Do they reflect actual behaviors?

External validity concerns

Type Concern
Samples Does this apply to a different population?
Contexts Does this apply in a different setting?
Treatments Do they resemble real-world phenomena?
Outcomes Do they reflect actual behaviors?

External validity concerns

Type Concern
Samples Does this apply to a different population?
Contexts Does this apply in a different setting?
Treatments Do they resemble real-world phenomena?
Outcomes Do they reflect actual behaviors?

Examples

Samples

Samples

Contexts