Combining List Experiments and the Network Scale Up Method

Gustavo Díaz
McMaster University

Verónica Pérez Bentancur
Universidad de la República

Ines Fynn
Universidad Católica del Uruguay

Lucía Tiscornia
University College Dublin

 

Slides: gustavodiaz.org/talk

Background

  • Social scientists care about sensitive issues

  • Surveys are the only option

  • Asking about them directly leads to misreporting

  • Solution: Indirect questioning techniques

  • List experiments are a popular alternative

Example

Now I am going to read you things that make people angry or upset

Adapted from Kuklinski, Cobb, and Gilens (1997)

Example

After I read them all, tell me HOW MANY of them upset you

Adapted from Kuklinski, Cobb, and Gilens (1997)

Example

I don’t want to know which ones, just tell me HOW MANY

. . .

Control group

  1. The federal government increasing tax on gasoline
  2. Professional athletes getting million dollar contracts
  3. Large corporations polluting the environment

Adapted from Kuklinski, Cobb, and Gilens (1997)

Example

I don’t want to know which ones, just tell me HOW MANY

Treatment group

  1. The federal government increasing tax on gasoline
  2. Professional athletes getting million dollar contracts
  3. Large corporations polluting the environment

Adapted from Kuklinski, Cobb, and Gilens (1997)

Example

I don’t want to know which ones, just tell me HOW MANY

Treatment group

  1. The federal government increasing tax on gasoline
  2. Professional athletes getting million dollar contracts
  3. Large corporations polluting the environment
  4. A black family moving next door

Adapted from Kuklinski, Cobb, and Gilens (1997)

Problem: Low statistical precision

. . .

Adapted from Rosenfeld, Imai, and Shapiro (2016)

Problem: Low statistical precision

Adapted from Rosenfeld, Imai, and Shapiro (2016)

Problem: Low statistical precision

Adapted from Rosenfeld, Imai, and Shapiro (2016)

Problem: Low statistical precision

Adapted from Rosenfeld, Imai, and Shapiro (2016)

Ways to improve precision

Ways to improve precision

Combined estimator

  • Logic: You don’t need a list experiment for those who admit to the sensitive item in the direct question

. . .

\[ \hat{\mu} = \overline{X} + (1 - \overline{X}) (\overline{V}_{1,0} - \overline{V}_{0,0}) \]

Source: Aronow et al (2015)

Combined estimator

  • Logic: You don’t need a list experiment for those who admit to the sensitive item in the direct question

\[ \hat{\mu} = \underbrace{\overline{X}}_\text{Prop direct question yes} + \underbrace{(1 - \overline{X})}_\text{Weight} \times \underbrace{(\overline{V}_{1,0} - \overline{V}_{0,0})}_\text{LE diff-in-means if direct question no} \]

Source: Aronow et al (2015)

Combined estimator

  • Logic: You don’t need a list experiment for those who admit to the sensitive item in the direct question

\[ \hat{\mu} = \underbrace{\underbrace{\overline{X}}_\text{Prop direct question yes} + \underbrace{(1 - \overline{X})}_\text{Weight} \times \underbrace{(\overline{V}_{1,0} - \overline{V}_{0,0})}_\text{LE diff-in-means if direct question no}}_\text{Weighted average of prevalence rates} \]

Source: Aronow et al (2015)

Problem

  • Can’t always include direct questions

  • Need an indirect questioning technique that lets us infer individual responses to sensitive item

  • But most rely on anonymity

  • Can’t combine without extra assumptions, altered designs, or population-level data

Network Scale-Up Method (NSUM)

. . .

How many people do you know,

Adapted from McCarty et al (2001). Original has 29 anchors and 3 target groups

Network Scale-Up Method (NSUM)

How many people do you know, who also know you,

Adapted from McCarty et al (2001). Original has 29 anchors and 3 target groups

Network Scale-Up Method (NSUM)

How many people do you know, who also know you, with whom you have interacted in the last year

Adapted from McCarty et al (2001). Original has 29 anchors and 3 target groups

Network Scale-Up Method (NSUM)

How many people do you know, who also know you, with whom you have interacted in the last year in person, by phone, or any other channel.

  • Named Michael
  • Named Christina
  • Gave birth in the past 12 months
  • Commercial pilots
  • Have tested positive for HIV

Adapted from McCarty et al (2001). Original has 29 anchors and 3 target groups

Hierarchical model

  • Goal: Find individuals with unusually high exposure relative to personal network

  • Fit with hierarchical model via MLE in two steps

  • Focus on standardized residuals:

\[ r_{ik} = \sqrt{y_{ik}} - \sqrt{e \alpha_i + \beta_k} \]

Details: Zheng, Salganik, and Gelman (2006); Calvo and Murillo (2013); Ventura, Ley, and Cantú (2023)

Hierarchical model

  • Goal: Find individuals with unusually high exposure relative to personal network

  • Fit with hierarchical model via MLE in two steps

  • Focus on standardized residuals:

\[ r_{ik} = \underbrace{\sqrt{y_{ik}}}_\text{Observed sensitive network size} - \underbrace{\sqrt{e \alpha_i + \beta_k}}_\text{Expected sensitive network size} \]

Details: Zheng, Salganik, and Gelman (2006); Calvo and Murillo (2013); Ventura, Ley, and Cantú (2023)

Application

Criminal governance tools in Uruguay

  • Facebook sample of Montevideo residents (N = 2688)

  • Goal: Document extent of exposure to criminal governance strategies

. . .

Negative

  • Threaten neighbors
  • Evict neighbors

Positive

  • Make donations to neighbors
  • Offer jobs to neighbors

Treatments based on qualitative evidence from fieldwork (Pérez Bentancur and Tiscornia 2022)

Criminal governance tools in Uruguay

  • Facebook sample of Montevideo residents (N = 2688)

  • Goal: Document extent of exposure to criminal governance strategies

Negative

  • Threaten neighbors
  • Evict neighbors

Positive

  • Make donations to neighbors
  • Offer jobs to neighbors

Treatments based on qualitative evidence from fieldwork (Pérez Bentancur and Tiscornia 2022)

Direct question

During the last six months, in your neighborhood, have you seen gangs…

. . .

  • Threaten neighbors
  • Evict neighbors
  • Make donations to neighbors
  • Offering jobs neighbors
  • Blackmail neighbors
  • Blackmail businesses
  • Pay a neighbor’s phone or electricity bills

List experiments

Things people have experienced in the last six months:

. . .

List A List B
Saw people doing sports Saw people playing soccer
Visited friends Chatted with friends
Activities by feminist groups Activities by LGBTQ groups
Went to church Went to charity events

List experiments

Things people have experienced in the last six months:

List A List B
Saw people doing sports Saw people playing soccer
Visited friends Chatted with friends
Activities by feminist groups Activities by LGBTQ groups
Went to church Went to charity events
Did not drink mate Gangs threatening neighbors

List experiments

Things people have experienced in the last six months:

List A List B
Saw people doing sports Saw people playing soccer
Visited friends Chatted with friends
Activities by feminist groups Activities by LGBTQ groups
Went to church Went to charity events
Gangs threatening neighbors Did not drink mate

NSUM

How many people do you know, who also know you, with whom you have interacted in the last year in person, by phone, or any other channel

. . .

  • 15 reference groups + sensitive item

  • Choice range 0-10+

  • Recode as 1 if \(r_{ik} > \text{mean}(r_{ik}) + 1 \text{SD}\), 0 otherwise

See list of groups here

Single-question estimates

Single-question estimates

Combined estimates

Conclusion

  • Can use NSUM to improve precision of list experiment estimates

  • Can be combined with other precision-enhancing techniques

  • Logic applies to other indirect questioning techniques that rely on anonymity

Learn more: gustavodiaz.org

NSUM groups

From Las Piedras
Male 25-29
Police officers
University students
Had a kid last year
Passed away last year
Married last year
Female 45-49

Public employees
Welfare card holders
Registered with party
With kids in public school
Did not vote in last election
Currently in jail
Recently unemployed [Sensitive item]