Imagine this scenario…
You’re conducting an evaluation or quasi-experiment of two different P/CVE programs that are being rolled out in two or more locations (or the same program measured at two or more times), and you want to answer an important question: which iteration of the program is working better.
Statistically speaking, making this comparison is no big deal, right? In this scenario, you have two conditions, and you’ll use the most powerful stats technique available to you, given the level of measurement and distributions of your variables of interest. From there, you’ll be able to generalize your results to whatever population from which you sampled.
A (Solvable) Problem
If you remember back to those dusty stats or research methods textbooks, there’s an important assumption the above approach violates, known as…
Assumed Independence of Observation
The assumption behind many (arguably most) statistical techniques is that every measurement you take is independent of (i.e., not influenced by) other measurements. Such “other” measurements include measurement of other people in the study.
Speak English! Translation: The people measured in the examples above are not completely independent; rather, they likely intermingle with each other. For example, in the second scenario, those in classrooms share not only the same curriculum, but the same teacher, the same classroom environment, the same set of classmates, perhaps similar pre-program backgrounds (e.g., one group of people might be naturally more/less experienced or educated to begin with). In other words, we can’t assume their performance is simply due to the curriculum.
In This Post:
- So what?
- Nested designs
- Bottom line, if you’re not a stats geek
- Stats geek litmus test
So What?
It’s true that many statistical techniques are surprisingly “robust” against violations of their assumptions. In other words, you can make relatively accurate comparisons and estimates, even if the data don’t look textbook-pretty, (e.g., real-world data). However, most common statistical tests aren’t at all robust against violations of the assumption of independence.
English…do you speak it?! Translation: Wrong analyses = wrong conclusions. More specifically, you might:
- Think you’ve discovered significant effects when there are none (or vice versa).
- Misattribute the effects to the wrong source (e.g., at the individual vs. group level)
- Make a fool of yourself as a researcher
- Miss opportunities to study interesting cross-level processes
- Fail to understand the nuances of the outcomes.
So, do we just give up on these types of analyses? Heck no; we use…
Nested Designs
Also known as mixed or multi-level designs, (or worse sounding) multi-level models, (or much worse sounding) structural equation models, this statistical technique accounts for differences that can be attributed to de facto groups that are “nested” within the overall design.
The beauty is not only that we avoid making erroneous conclusions about our outcomes of interest, but we can estimate the effect(s) of the groupings themselves. For example, we could understand which research sites are over/under-performing, not because of the program itself, but due to some other grouping-related factor that we can begin to investigate.
There are other important advantages to nested designs, but those will primarily be of interest to stats geeks.
Bottom Line, If You’re Not a Stats Geek
If you’re a) planning, commissioning, evaluating, or otherwise overseeing a multi-site, or multi-time, study, and b) aren’t well-trained in multi-level stats, you should (dare we say, must) procure the counsel of a stats geek who is so trained.
Stats Geek Litmus Test
(The speed with which you grasp the above graphic is a negatively-correlated indicator of your risk of being a stats geek.)
Further Resources
Given that you’re still reading this, I regret to inform you that you might very well be a stats geek. Take heart; at least you’re potentially useful. If you’re already adept at multi-level modeling, more power to you; vaya con Dios. If, instead, you’d like some further resources/training in this domain, check out these outstanding workbooks.
Heck, R. H., Thomas, S. L., & Tabata, L. N. (2007). Multilevel and longitudinal modeling with IBM SPSS. New York: Routledge.
Mertler, C. A., & Vannatta, R. A. (2005/2010). Advanced and multivariate statistical methods: Practical application and interpretation (4th ed.). Los Angeles, CA: Pyrczak Publishing.
Have an idea for a future feature on The Science of P/CVE blog? Just let us know by contacting us through the form, or social media buttons, below.
We look forward to hearing from you!
All images used courtesy of creative commons licensing.