Prediction predicaments, part 2 (of 2): The base rate problem

In September, 2013, at a meeting of the Global Counterterrorism Forum, the U.S. Secretary of State, along with the Head of the Turkish Foreign Ministry, announced the Global Community Engagement and Resilience Fund (GCERF): a fund expected to raise more than $200 million to support local/grass-roots P/CVE initiatives.

According to a State Department fact sheet, “What distinguishes GCERF’s objectives from broader development efforts is its emphasis on youth engagement, education, vocational training, and women’s advocacy to promote resilience among at-risk populations.”

This is a wonderful objective! The trick, it would seem, is identifying those “at-risk.”

Here’s the problem; they won’t. More precisely, they can’t, and here’s why…

In this post

P/CVE diagnostic criteria
The “base rate problem”
The big question
We’re here to help

P/CVE diagnostic criteria

To assess risk–in accurate, reliable ways–akin to activities of the insurance industry, or (in some cases) medical science, one needs objective, data-driven diagnostic criteria. Currently, we’ve no such criteria in the realm of P/CVE. Though some in the field are championing data-driven approaches to P/CVE, currently the P/CVE field has yet to attain the accuracy of actuarial science.

Translation: no accurate diagnostic criteria = no accurate identification of “at risk” populations.

The “base rate problem”

Because terrorism, and other forms of violent extremism, are mercifully rare, there’s another challenge to predicting it, and it’s called the “base rate problem.”

Imagine this…

You have a diagnostic test that is 95% accurate both at identifying those at risk vs. not at risk of engaging in violent extremism. That’s excellent, right?

However, what is the per capita population of violent extremists in a given area? Who knows? For argument’s sake, let’s say such persons are the proverbial 1 in a million (in stats-speak, the “base rate” is 1/1,000,000).

Of course, the flip-side of our 95% accurate test is that it’s going to be wrong 5% of the time. Which means, in identifying–with 95% certainty–that 1 in a million violent extremist, it’s going to misidentify 5% of the others in that population of one million: approximately 50,000 regular citizens.

Translation: Even with seemingly high-accuracy tests, low base rate phenomena = low accuracy in identifying “at-risk” populations.

Where’s Waldo (and the violent extremist)?

The big question

In the P/CVE realm, the big question tends to be “Does a given P/CVE program work?” Given the base rate problem, P/CVE programs that work with general populations (in evaluation-speak, “primary prevention” programs) are highly unlikely to demonstrate, empirically, that their programs prevent violent extremism.

This does not necessarily imply that that primary-prevention programs are ineffective, but that the odds are stacked against them being able to demonstrate direct effects.

Policy makers can (and should) ask tougher “does this work?” questions of CVE programs that work with those who have already exhibited violent extremist tendencies (e.g., individuals who somehow glorify violent extremism), or programs that work with those who have attempted, or committed, ideologically-rationalized violence (e.g., so-called “deradicalization programs”).

Because such programs work with populations of pre-identified persons who are at-risk of committing (or recommitting) violence, they suffer to a lesser degree (if at all) from the base rate problem. Therefore, such CVE programs should, theoretically, have better chances of demonstrating direct effects.

Take-home message regarding primary-prevention P/CVE programs

Given both the lack of accurate diagnostic criteria, and the base rate problem, both policy and research discussions regarding primary-prevention P/CVE programs should ask “Will a given program plausibly work?” Answers to that question will be only as good as the theories that guide a given program, and the degree to which support for those theories is triangulated from data-driven studies.