No control group, no big deal part 1 (of 2): Propensity score matching designs

The world is a d@rn messy place. Only a few pinpoints on our planet are laboratories capable of strict experimental control. Yet, it’s our big, beautiful, messy world in which many big problems need to be studied and acted upon.

The problem is: how do we identify cause and effect if either of two major principles of experimentation (random selection of participants, and random assignment of participants to experimental vs. control conditions) are violated?

For example, how do we know if a prison-based terrorism risk reduction initiative works better to reduce post-detainment terror engagement in one jurisdiction vs. a different initiative in another jurisdiction. After all, we didn’t randomly assign the initiatives’ participants (i.e., the incarcerated offenders) to be incarcerated at one location vs. the other.

Therefore, who knows whether one initiative is performing better than the other, or whether there are differences between the offenders in the different jurisdictions (e.g., job skills, marital status, degree of family support, number and severity of prior offenses) that make offenders from one location (vs. the other) more likely to refrain from re-engaging in terrorism?

In this post…

Explanation of propensity score matching designs, for those who don’t care about propensity score matching designs.
Who cares?!
The “catch.”

Fear not; this discussion will not deteriorate into a stats tutorial. (Check out the resources below, if that’s what you wish.) Instead, this is written for those who know virtually nothing about the topic, but who would like to know a) why they should care, and/or b) how to learn more if they wish.

The bright idea

Going back to our example of two different prison-based terrorism risk reduction initiatives, among the few scientifically valid ways to make the above comparison would be to compare the post-detainment terrorism engagement rates of parolees in one jurisdiction one, with their similar counterparts in the other location.

The key here is finding very good counterparts, and that’s where propensity score matching comes in. For every person in the terrorism risk reduction initiative at one location, we identify their statistical “buddy,” at the other location. That buddy is someone who is statistically matched with the person in every way we can think of (and can measure reliably) on characteristics that we think might relate to their likelihood to engage in post-detainment terrorism engagement.

Remember some of those plausible characteristics: job skills, marital status, degree of family support, number and severity of prior offenses? If it’s a plausible factor, and can be measured reliably, it should be tossed into the analysis: an analysis called logistic regression that calculates the odds (i.e., the propensity, hence the name) of something happening. (More on this in “the catch” section.)

By equating the comparison groups, this is a powerful technique for supporting causal inference, insofar as the only difference that remains between the groups is the characteristic of interest: in this case, whether parolees participated the initiative at one location vs. the other.

Therefore, we could reasonably conclude that any difference in the post-detainment terror engagement rates between parolees in one jurisdiction (vs. the other) is caused by the risk reduction initiative.

Who cares?

The $hort answer i$: funding agencies care. Prestigious funding agencies (read “those with deep pockets”) have enough sense to consult scientists in the know, and those in the know know that well-done propensity score matching research designs are among the few designs available if one is to draw reasonably certain conclusions about cause and effect without the luxury of random assignment to comparison conditions.

Another answer is that policy makers should care: at least enough to ask if/how studies’ comparison groups were equated.

The “catch”

This type of analysis relies upon two basic foundations. First, that all plausible characteristics related to the outcome of interest (e.g., post-detainment terrorism engagement) have been used to match the statistical “buddies” in one location vs. the other location. Second, that those matching characteristics have been measured reliably (according to statistical conventions).

In short, to do this type of analysis, one typically needs quality data about the individuals in the study.

Here ye, researchers/evaluators: you might already have realized that, given the above “catch,” part of your job is to research, as thoroughly as you can, variables that might plausibly be associated with the outcome of interest (and, so, should be controlled via matching). If data aren’t available about those in your study, with respect to those variables, it will also be part of your job to figure out how to obtain reliable measurements of them.

Here ye, policy makers & funding agencies: part of your job is to ask researchers/evaluators, who plan to conduct propensity score analyses, about a) variables they used to match the statistical “buddies,” and b) what makes them think that they’ve measured them reliably. Answers to those questions will help you to judge the quality of the studies you’re commissioning and/or relying upon

Further reading

Austin, P. C. (2011). An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies. Multivariate Behavioral Research, 46, 399-424.
Rudner, L. M., & Peyton, J. (2006). Consider Propensity scores to compare treatments. GMAC – Graduate Management Admission Council.
Timberlake, D. S., Huh, J., & Lakon, C. M. (2009). Use of propensity score matching in evaluating smokeless tobacco as a gateway to smoking. Nicotine & Tobacco Research, 11, 455-462.

Somewhat slow-paced ‘how-to’ video tutorial playlist on propensity score matching:

http://www.youtube.com/watch?v=VYUjfFWPENM&feature=share&list=PLRW9kMvtNZOjvuA5LQnRKfWZFc5E5tdSg&index=2.

This article touches upon propensity score matching designs in the context of evaluating P/CVE initiatives. Williams, M. J., & Kleinman, S. M. (2013). A utilization-focused guide for conducting terrorism risk reduction program evaluations. Behavioral Sciences of Terrorism and Political Aggression, doi: 10.1080/19434472.2013.860183. (Full text available via the hyperlink in this reference.)

Have an idea for a future feature on The Science of P/CVE blog? Just let us know by contacting us through the form, or social media buttons, below.

We look forward to hearing from you!

All images used courtesy of creative commons licensing.

(Wow, if you’re still reading this, hit us up on Twitter or Facebook, research methods geek. Way to be.)

No control group, no big deal part 1 (of 2): Propensity score matching designs

You might also be interested in part two of this two-part series, “No control group, no big deal, part 2 (of 2): Regression discontinuity designs.”