Combining List Experiments and the Network Scale Up Method

Gustavo Díaz
McMaster University

Verónica Pérez Bentancur
Universidad de la República

Ines Fynn
Universidad Católica del Uruguay

Lucía Tiscornia
University College Dublin

 

Slides: gustavodiaz.org/talk

Background

  • Social scientists care about sensitive issues

  • Surveys are the only option

  • Asking about them directly leads to misreporting

  • Solution: Indirect questioning techniques

  • List experiments are a popular alternative

Example

Now I am going to read you things that make people angry or upset

Example

After I read them all, tell me HOW MANY of them upset you

Example

I don’t want to know which ones, just tell me HOW MANY

Control group

  1. The federal government increasing tax on gasoline
  2. Professional athletes getting million dollar contracts
  3. Large corporations polluting the environment

Example

I don’t want to know which ones, just tell me HOW MANY

Treatment group

  1. The federal government increasing tax on gasoline
  2. Professional athletes getting million dollar contracts
  3. Large corporations polluting the environment

Example

I don’t want to know which ones, just tell me HOW MANY

Treatment group

  1. The federal government increasing tax on gasoline
  2. Professional athletes getting million dollar contracts
  3. Large corporations polluting the environment
  4. A black family moving next door

Problem: Low statistical precision

Problem: Low statistical precision

Problem: Low statistical precision

Problem: Low statistical precision

Ways to improve precision

Ways to improve precision

Combined estimator

  • Logic: You don’t need a list experiment for those who admit to the sensitive item in the direct question

\[ \hat{\mu} = \overline{X} + (1 - \overline{X}) (\overline{V}_{1,0} - \overline{V}_{0,0}) \]

Combined estimator

  • Logic: You don’t need a list experiment for those who admit to the sensitive item in the direct question

\[ \hat{\mu} = \underbrace{\overline{X}}_\text{Prop direct question yes} + \underbrace{(1 - \overline{X})}_\text{Weight} \times \underbrace{(\overline{V}_{1,0} - \overline{V}_{0,0})}_\text{LE diff-in-means if direct question no} \]

Combined estimator

  • Logic: You don’t need a list experiment for those who admit to the sensitive item in the direct question

\[ \hat{\mu} = \underbrace{\underbrace{\overline{X}}_\text{Prop direct question yes} + \underbrace{(1 - \overline{X})}_\text{Weight} \times \underbrace{(\overline{V}_{1,0} - \overline{V}_{0,0})}_\text{LE diff-in-means if direct question no}}_\text{Weighted average of prevalence rates} \]

Problem

  • Can’t always include direct questions

  • Need an indirect questioning technique that lets us infer individual responses to sensitive item

  • But most rely on anonymity

  • Can’t combine without extra assumptions, altered designs, or population-level data

Network Scale-Up Method (NSUM)

How many people do you know,

Network Scale-Up Method (NSUM)

How many people do you know, who also know you,

Network Scale-Up Method (NSUM)

How many people do you know, who also know you, with whom you have interacted in the last year

Network Scale-Up Method (NSUM)

How many people do you know, who also know you, with whom you have interacted in the last year in person, by phone, or any other channel.

  • Named Michael
  • Named Christina
  • Gave birth in the past 12 months
  • Commercial pilots
  • Have tested positive for HIV

Hierarchical model

  • Goal: Find individuals with unusually high exposure relative to personal network

  • Fit with hierarchical model via MLE in two steps

  • Focus on standardized residuals:

\[ r_{ik} = \sqrt{y_{ik}} - \sqrt{e \alpha_i + \beta_k} \]

Hierarchical model

  • Goal: Find individuals with unusually high exposure relative to personal network

  • Fit with hierarchical model via MLE in two steps

  • Focus on standardized residuals:

\[ r_{ik} = \underbrace{\sqrt{y_{ik}}}_\text{Observed sensitive network size} - \underbrace{\sqrt{e \alpha_i + \beta_k}}_\text{Expected sensitive network size} \]

Application

Criminal governance tools in Uruguay

  • Facebook sample of Montevideo residents (N = 2688)

  • Goal: Document extent of exposure to criminal governance strategies

Negative

  • Threaten neighbors
  • Evict neighbors

Positive

  • Make donations to neighbors
  • Offer jobs to neighbors

Criminal governance tools in Uruguay

  • Facebook sample of Montevideo residents (N = 2688)

  • Goal: Document extent of exposure to criminal governance strategies

Negative

  • Threaten neighbors
  • Evict neighbors

Positive

  • Make donations to neighbors
  • Offer jobs to neighbors

Direct question

During the last six months, in your neighborhood, have you seen gangs…

  • Threaten neighbors
  • Evict neighbors
  • Make donations to neighbors
  • Offering jobs neighbors
  • Blackmail neighbors
  • Blackmail businesses
  • Pay a neighbor’s phone or electricity bills

List experiments

Things people have experienced in the last six months:

List A List B
Saw people doing sports Saw people playing soccer
Visited friends Chatted with friends
Activities by feminist groups Activities by LGBTQ groups
Went to church Went to charity events

List experiments

Things people have experienced in the last six months:

List A List B
Saw people doing sports Saw people playing soccer
Visited friends Chatted with friends
Activities by feminist groups Activities by LGBTQ groups
Went to church Went to charity events
Did not drink mate Gangs threatening neighbors

List experiments

Things people have experienced in the last six months:

List A List B
Saw people doing sports Saw people playing soccer
Visited friends Chatted with friends
Activities by feminist groups Activities by LGBTQ groups
Went to church Went to charity events
Gangs threatening neighbors Did not drink mate

NSUM

How many people do you know, who also know you, with whom you have interacted in the last year in person, by phone, or any other channel

  • 15 reference groups + sensitive item

  • Choice range 0-10+

  • Recode as 1 if \(r_{ik} > \text{mean}(r_{ik}) + 1 \text{SD}\), 0 otherwise

Single-question estimates

Single-question estimates

Combined estimates

Conclusion

  • Can use NSUM to improve precision of list experiment estimates

  • Can be combined with other precision-enhancing techniques

  • Logic applies to other indirect questioning techniques that rely on anonymity

NSUM groups

From Las Piedras
Male 25-29
Police officers
University students
Had a kid last year
Passed away last year
Married last year
Female 45-49

Public employees
Welfare card holders
Registered with party
With kids in public school
Did not vote in last election
Currently in jail
Recently unemployed [Sensitive item]