1 Process Tracing

Macartan

The big picture: Intuition

1.1 Process tracing with a DAG

• Simple insight: If you believe this model, then seeing \(M\) should tell you something about a query regarding the \(I\), \(D\) relationship in a case.

• For instance, we might have the intuition: If there was no mobilization in a high-inequality case that democratized, then inequality didn’t cause the transition.

But how do we formalize this strategy?

1.2 Process tracing with a Causal Model

• Great as this is, the DAG—by itself—tells us nothing about the direction of effects
  • Inequality might boost or depress mobilization
  • Mobilization might encourage or delay democratization
• For process-tracing, you need an informative causal model

1.3 Process tracing with a DAG

Key formal insight:

If you believe this model, then seeing \(M\) should tell you something about the \(\theta\)s—which are what define the effect of \(I\) on \(D\).

1.4 From the DAG alone…

We can tell when some evidence might potentially matter

1.5 Principle: Clues can lie in many places

Three clues for figuring out the effect of \(X\) on \(Y\)

2 Example: Democratization in Malawi (1994)

Alan

Let’s walk though the intuition with a case

2.1 Process tracing as new data for finer inferences

• We start with:

  1. A background theory about how things work in cases like this
  2. Background knowledge about the case: perhaps the outcome; perhaps the value of a causal condition or two

• What we don’t know is whether any of those conditions caused the outcome

• So:

  1. we collect more information “from other parts of the DAG” to help figure that out
  2. we want to have a plan for what to infer about effects depending on what we find

2.2 A case-level question: Malawi (1994)

• Suppose we observe democratization as the outcome in Malawi: \(D=1\)

• We also observe high inequality in Malawi: \(I=1\)

• We want to know: did \(I=1\) cause \(D=1\) in Malawi?

2.3 Prior data on Malawi

• Let’s process-trace: observe \(M\) in Malawi
  • Was there mass mobilization in Malawi?

2.4 New within-case data on Malawi

• Suppose we go to the field and we learn that mass mobilization DID occur in Malawi

  • So \(M=1\)

• What can we conclude?

• NOTHING YET!

2.5 Inference

• We knew \(I=1\), \(D=1\)

• We then saw \(M=1\)

• But which is more consistent with \(I=1\) causing \(D=1\)?

  • Mobilization occurring? or
  • Mobilization not occurring?

2.6 Why prior theory matters for process tracing

• Suppose I believe that high inequality usually prevents mass mobilization
  • Poverty makes it harder to mobilize
• Suppose I also believe that mobilization usually prevents democratization
  • Instills fear in autocrats and provokes repression
• Which value of \(M\) would be most consistent with \(I=1\) causing \(D=1\)?
  • \(M=0\)

2.7 Why prior theory matters for process tracing

• Suppose I believe that high inequality usually causes mass mobilization
  • Inequality creates grievances for protest
• Suppose I also believe that mobilization usually causes democratization
  • Instills fear in autocrats and provokes concessions
• Which value of \(M\) would be most consistent with \(I=1\) causing \(D=1\)?
  • \(M=1\)

2.8 Why prior theory matters for process tracing

• If I believe inequality usually causes democratization by preventing mobilization, which itself hinders democratization…
  • then observing \(M=0\) constitutes evidence that \(I=1\) caused \(D=1\)
• If I believe inequality usually causes democratization by causing mobilization, which itself causes democratization…
  • then observing \(M=1\) constitutes evidence that \(I=1\) caused \(D=1\)

2.9 Key point: Process tracing is always theory dependent

• The evidence in process tracing never speaks for itself

• Our inferences from PT evidence always depend on theory

  • On our beliefs about how the world works

• Most process tracing is either silent about these beliefs or expresses them informally

• We can formalize these beliefs

  • Maximize explicitness and analytic transparency

3 Procedure for process tracing

Macartan

3.1 General strategy

for process tracing with causal models

• We can figure out from the causal model which causal types are consistent with our query
• We can figure out from the causal model which causal types are consistent with the data we observe
• If we have probabilities for each of these we can figure out the probability of the query given the data

3.2 Key Bayesian insight

Figure 1: Logic of simple updating on arbitrary queries.

3.3 By hand: Procedure

1. Make a table with rows for all causal types (there may be many if you are doing by hand!!)
2. Add a column to indicate your priors over these causal type
3. Add a column to say if the query is satisfied by the causal types
4. Calculate the conditional distributions given the types

3.4 By hand: Example

Our DAG is:

\[X \rightarrow M \rightarrow Y\]

And we believe:

• \(X=1\) for half the cases, randomly
• \(X\) has a positive effect on \(M\) for half the cases (“causes”), in the other half \(M=0\) regardless of \(X\)
• \(M\) has a positive effect on \(Y\) for half the cases, in the other half \(Y=0\) regardless of \(M\)

3.6 With CausalQueries: Step 1

Define a model

model <- 
  make_model("X -> M -> Y") |>
  set_restrictions("M[X = 0] == 1") |>
  set_restrictions("Y[M = 0] == 1")

query <- "Y[X=1] > Y[X=0]"

3.7 With CausalQueries: Step 2

Get types consistent with query

get_query_types(model, query)

Causal types satisfying query's condition(s)  

 query =  Y[X=1]>Y[X=0] 

X0.M01.Y01  X1.M01.Y01


 Number of causal types that meet condition(s) =  2
 Total number of causal types in model =  8

3.8 With CausalQueries: Step 3

Get mapping from causal types to consistent data types

inspect(model, what = "ambiguities_matrix") 

ambiguities_matrix (Ambiguities matrix)
Mapping from causal types into data types:
         X0M0Y0 X1M0Y0 X1M1Y0 X1M1Y1
X0M00Y00      1      0      0      0
X1M00Y00      0      1      0      0
X0M01Y00      1      0      0      0
X1M01Y00      0      0      1      0
X0M00Y01      1      0      0      0
X1M00Y01      0      1      0      0
X0M01Y01      1      0      0      0
X1M01Y01      0      0      0      1

3.9 With CausalQueries: Step 4

Get prior probabilities of each causal type

CausalQueries:::get_type_prob(model)

[1] 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125

3.10 Put it all together

model  |>
  grab(what = "ambiguities_matrix") |>
  data.frame() |>
  mutate(
    in_query = get_query_types(model, query)$types,
    priors   = CausalQueries:::get_type_prob(model)) |>
  kable()

3.11 In one go with CausalQueries

query_model(model, query, given = c("X==0 & M ==0 & Y == 0"))

Causal queries generated by query_model (all at population level)

|label                                       |using      | mean|
|:-------------------------------------------|:----------|----:|
|Y[X=1] > Y[X=0] given X==0 & M ==0 & Y == 0 |parameters | 0.25|

3.12 Three clues example

Three clues for figuring out the effect of \(X\) on \(Y\)

3.13 Three clues example

Three clues for figuring out the effect of \(X\) on \(Y\)

observed = c("Y==1 & X==1",
           "Y==1 & X==1 & K1==1",
           "Y==1 & X==1 & K2==1",
           "Y==1 & X==1 & K3==1",
           "Y==1 & X==1 & K1==1 & K2==1 & K3==1")

query_model(model = model, 
            query = "Y[X=1] > Y[X=0]",
            given = observed)

Causal queries generated by query_model (all at population level)

|label                                                     |using      | mean|
|:---------------------------------------------------------|:----------|----:|
|Y[X=1] > Y[X=0] given Y==1 & X==1                         |parameters | 0.25|
|Y[X=1] > Y[X=0] given Y==1 & X==1 & K1==1                 |parameters | 0.25|
|Y[X=1] > Y[X=0] given Y==1 & X==1 & K2==1                 |parameters | 0.25|
|Y[X=1] > Y[X=0] given Y==1 & X==1 & K3==1                 |parameters | 0.25|
|Y[X=1] > Y[X=0] given Y==1 & X==1 & K1==1 & K2==1 & K3==1 |parameters | 0.25|

3.14 With information

model |>
  set_restrictions(decreasing("X", "K1")) |>
  set_restrictions(decreasing("K1", "Y")) |>
  set_restrictions(decreasing("K2", "Y")) |>
  set_restrictions(decreasing("Y", "K3")) |>
  set_parameters(given = "Y.0001", nodal_type = "11", .9)|>
  query_model(
    query = "Y[X=1] > Y[X=0]",
    given = observed)

Causal queries generated by query_model (all at population level)

|label                                                     |using      |  mean|
|:---------------------------------------------------------|:----------|-----:|
|Y[X=1] > Y[X=0] given Y==1 & X==1                         |parameters | 0.200|
|Y[X=1] > Y[X=0] given Y==1 & X==1 & K1==1                 |parameters | 0.250|
|Y[X=1] > Y[X=0] given Y==1 & X==1 & K2==1                 |parameters | 0.154|
|Y[X=1] > Y[X=0] given Y==1 & X==1 & K3==1                 |parameters | 0.212|
|Y[X=1] > Y[X=0] given Y==1 & X==1 & K1==1 & K2==1 & K3==1 |parameters | 0.224|

3.15 In one go with CausalQueries

Also try our shiny app

3.16 Process tracing with causal models

Key advantages of using a causal model:

• Makes clear how process tracing is always theory-dependent
• Provides a way of systematically integrating background knowledge (theory) into inference
• Makes highly explicit/transparent how we get from evidence to conclusions
• Enhances evaluability of findings
  • If I have different background beliefs from you, I can work out how my conclusions from same evidence would be different

4 Democratization example

Alan

4.1 Application

• We take a problem over which there is substantial theoretical debate and in which scholars have sought explicitly to apply process tracing strategies.

4.2 Setting restrictions

• Monotonicity of \(I\)’s effect on \(M\): no negative effect
• Monotonicity of \(I\)’s direct effect on \(D\): no positive effect
• Monotonicity of \(M\)’s effect on \(D\): no negative effect
• Monotonicity of \(P\)’s effect on \(D\): no negative effect

4.3 Where do these prior beliefs come from?

• Literature
  • We draw on leading works on inequality and democratization (Boix, Acemoglu and Robinson, Ansell and Samuels)

Could also come from:

• Deductive theorization
  • E.g., a formal model (see Chapter 6.3)
• Crowd-sourcing
  • Survey experts
• Important: we cannot read causal beliefs directly off population-level descriptives
  • E.g., the number of \(I=1, M=1\) cases tells us nothing about how often inequality causes mobilization

4.4 Process tracing results for several cases

• With different possible clues we could observe

4.5 Process tracing to answer a pathway question

• Similar idea for answering a pathway question
• Observing \(M\) can help us figure out how \(I\) caused \(D\)
• Once we provide population-level beliefs about causal effects

4.6 Other pieces of evidence

• \(P\) can also be informative about \(I\)’s effect on \(D\)
• Will depend on our population-level beliefs about \(P\)’s effects on \(D\) and interactions with \(M\)