No control group, no big deal, part 2 (of 2): Regression discontinuity designs

Ethnically-diverse friends ≈ Positive cross-ethnicity attitudes

Imagine this; you’re about to evaluate a P/CVE program, and a preliminary survey of the P/CVE program’s participants uncovers the unsurprising outcome that the more friends that participants have from other ethnicities, the more favorable their attitudes toward those from other ethnicities tend to be (and vice versa).

Here are some problems

Before you attempt to confirm this finding, and declare that increasing participants’ inter-ethnic friendships tends to improve their inter-ethnic attitudes, consider that…

Not everyone has the same need for the program. Based on our example, some already have relatively positive attitudes toward those from other ethnicities. Therefore, if you put everybody through the program, that might waste time and money.
To demonstrate the proposed causal link between one’s number of inter-ethnic friends and one’s positive inter-ethnic attitudes, you can’t do so experimentally, given that you can’t randomly assign people to have a greater number ethnically-diverse friends.

So what are you going to do?…

To target those in need of your program
To test the effectiveness of your program

A good answer: You’re going to use a regression discontinuity design!

In this post

What the heck is regression disconti-whatever-you-call-it?
Why bother?
The big idea
The design in action
Caveats
Why don’t we use this design more routinely?

What the heck is a regression discontinuity design?

Don’t worry, “regression discontinuity design,” is just a terrible term for a fancy way to compare two groups, when the group members are assigned to those groups based, not on random assignment, but on the basis of some other measured characteristic.

Why bother?

This technique should be considered anytime you need to demonstrate causality, but…

Experimental testing, using random assignment to the program (vs. a control condition), is impossible (read, “you’re probably doing research in the real world, not a lab”).
Random assignment would be unethical (e.g., withholding a potentially useful program from those who need it most, merely to create a control group).
Random assignment would be a waste of resources (e.g., why give a program to those who don’t need it?).

The Big Idea

In principle, regression disconti….let’s just call them RD designs, are equivalent to the “gold standard”–randomized control trials (RCTs)–to demonstrate causality of a given program/intervention.

In practice, there are some caveats for RD designs to live up to this expectation, but we’ll get to those below.

The design in action

From the opening example, the data look like this; the more cross-ethnicity friends someone has, the better their attitudes toward those from other ethnicities.

The cut-point

For this example, let’s offer the P/CVE program to those who need it most: those who are below the mid-point (our cut-point) on attitudes toward those from other ethnicities (shown in green).

If the program is successful…

If the P/CVE program improves attitudes toward those from other ethnicities, the data will look like this.

Discontinuity depiction

Those who participate in the program (i.e., those below our cut-point) show an improvement in their attitudes toward those from other ethnicities, regardless of how many friends they have from other ethnicities.

If the program is unsuccessful…

If the P/CVE program accidentally damages participants’ attitudes toward those from other ethnicities, the data will look like this.

Discontinuity depiction 2

Those who participate in the program (i.e., those below our cut-point) show a worsening in their attitudes toward those from other ethnicities, regardless of how many friends they have from other ethnicities

Caveats

1. The first caveat is that the cut-point characteristic must be genuinely related to the outcome of interest. Imagine if we had made the cut-point based upon participants’ height or weight. Although it’s possible (though doubtful) that we would find group differences based on that trait, it would be rather ridiculous to imagine that either of those traits affects someone’s cross-ethnicity attitudes.

2. Respect the cut-point. It’s important that those falling on one side of the cut-point indeed receive/don’t receive the program as intended. If you bend the rule by admitting some people to the program who weren’t intended to get it (or vice versa), you weaken the statistical power of this design and impair your ability to find statistically significant effects.

3. Speaking of power, one of the drawbacks of this design, compared to randomized control trials (RCTs), is that it requires at least 2.75 times more participants to show a comparatively large effect. However, you might be able to acquire such a larger sample size, because you might be able to identify/include a larger group of people than you would for an RCT.

4. There are other stats-geek caveats (e.g., measurement reliability), that researchers need to consider with this design, but such concerns exist for all designs. Interested parties should check out the resources below.

Why don’t we use this design more routinely?

Researchers, for your consideration:

This technique has been around since the 1960s (shout out to its creators, Campbell and Thistlewaite). According to research methods meister, Thomas Cook, it isn’t used more commonly because it hasn’t been widely taught (though, this is changing).

In his opinion (and mine), when randomized control trials are impossible, researchers should always consider using a regression discontinuity design. (However, there are other strong non-RCT designs; see this blog’s “No control group, no big deal, part 1 (of 2): Propensity score matching designs.”)

Policy makers, for your consideration:

Compared with RCTs (which designate people to programs merely on the basis of chance, like playing the lottery with others’ lives), an RD design is a way to test a program by giving it–in an accountable way–to those deemed most in need of it.

In other words, this design is not only ethically desirable, but can save time and money by offering a given program to those deemed most in need.

Additional resources

An in-depth, though highly accessible, overview of RD designs (including several graphics), by a venerable authority on research design, William M.K. Trochim: http://www.socialresearchmethods.net/kb/quasird.php.

A how-to guide for analyzing RD data using SPSS software: http://core.ecu.edu/psyc/wuenschk/SPSS/RegDiscont-SPSS.pdf

Have an idea for a future feature on The Science of P/CVE blog? Just let us know by contacting us through the form, or social media buttons, below.

We look forward to hearing from you!

All images used courtesy of creative commons licensing.

If you haven’t read it already, you might also be interested in part one of this two-part series, “No control group, no big deal part 1 (of 2): Propensity score matching designs.”