A little while ago a friend pointed out I had made a mistake in one of my previous posts.... it was the post What You Might Not Know About Psychology: A Student's Perspective. It was basically me, with my student hat on, attempting to explain some of the nuances and pitfalls of interpreting news and blogs about psychology. Having re-read it, it's a bit of a brass-tacks approach, something I would like to think I would soften if I wrote it again (if only intone, but not content)

In one section of this post I wrote the following:

All Psychology studies are confounded and unreliable because they use university students to test participants!

That's true. The exceeding majority of results are based on populations of students and standardized onto everyone, everywhere (particularly by the media). Additionally, students tend to be WEIRD (White, Educated, from an Industrialized nation, Rich and Democratic). That looks really bad on paper, but most psychologists tend to interpret these finding in a WEIRD context. Additionally, every person I've encountered that objects to a finding because it's WEIRD are WEIRD themselves....

That was a pretty short explanation of a much bigger problem in communicating psychology (particularly things like evolutionary psychology).

So my friend wrote me:

You are wrong. Sorry dude. You absolutely need to read this well known, and short, article:

www.uoguelph.ca/~psystats/readings_3380/mook%20article.pdf

And so I did. I accept that I misrepresented the invalidity point, either through lack of depth in communication, or ignorance (and probably both).

First off, good psychology is not about making generalizations based on small samples. It's about testing generalizations within small samples. It's a nuanced difference, but clear with a little explaining. Psychology (like all science) is based on hypotheses. That is, we make predictions which we hope to be found true, prior to running the experiment. Some kind of theoretical framework exists which psych's do not accept at face value. They seek to scrutinize and criticize it, or seek the limits of its explanatory power. The reasoning follows:

If xyz theory is true, then this, by extension, is also true. Thus, if we find it we have another bit of evidence to support the case of xyz theory. If we find a lack of evidence, or contradictory evidence, then we must consider alternatives.

xyz theory is the generalization that is being tested on a smaller sample. The findings need not necessarily be generalizable to the world at large, as I both explicitly and implicitly stated.

Mook (the author of the article I was referred too), has the following to say on

The distinction between generality of findings and generality of theoretical conclusions underscores what seems to me the most important source of confusion in all this, which is the assumption that the purpose of collecting data in the laboratory is to predict real-life behaviour in the real world. Of course, there are times when that is what we are 'trying to do', and there are times when it is not. When it is, then the problem of EV [external validity] confronts us, full force. When it is not, then the problem of EV is either meaningless or trivial, and a misplaced preoccupation with it can seriously distort our evaluation of the research.

Second, some things identified or described in the lab cannot be generalized into the real world. As above, that may not be the point of the exercise; identifying some effect that would be trivial in the real world (if applied) still may have value.

Take, for example, the Macbeth Effect - the act of washing ourselves tend to influence the manner in which we make moral decisions. I argue the effect is small, based on artificial conditions, and trivial in the real-world. I would be right to say such things, but I may have missed the point of the research. The Macbeth Effect, however small it may be, and however useless it may be in reality, still asks a very interesting question, describes a fascinating phenomenon, and raises further questions that would otherwise have been missed if we only focused 'things we see in the real world'. Washing our hands probably does nothing for us when we're thinking about tax evasion, not tipping the waiter, or using our neighbours garbage bin for our most foul refuse - let's pretend evidence exists that shows this is the case. ... So what? The effect does occur in a lab, and may yet reveal some very subtle cognitive characteristics that are present under certain situations. My inclination to devalue it because it isn't 'real world' is misled; that's not the point and the research still has value.

Third, sometimes it's not about representativeness at all. It's about logic.

For instance, if a theory predicts that x is true when y, then, if y then x. Let's take an example with a terrible sample.

An Hypothesis might follow that Apes, by virtue of being so like humans, are capable of learning language.

So we take a few chimps, a few bonobos and a smattering of gorilla's and orangs, and try to teach them to sign, or to use lexigrams, or to play with a ouija board.

Some can eke out a meaningful passage but have no syntax, some fail, some manage only nouns. Now these Apes are not representative of all Apes. They're orphaned; they're hand-raised; They eat hot-dogs and watch TV. No-one's claiming that these Apes are not representative and so what has been observed is moot. No, it demonstrates the limits (both practical and logical) of theory.

If x is true when y, then, if y then x.

Language is possible if one is near-human. Thus, if Apes are near-human, they should be able to learn aspects of language.

It's a bare-bones account. They hypothesis is not: Perfect English is possible if one is near-human, therefore all Apes should be able to recite poetry.

My initial claim was the WEIRD is always a bad thing, but we deal with it. This is true, but there's much more to it that I was both ignorant of and which I did not communicate fully. But I was called up on it, and now I know so much more about it. The article linked to - here - by Douglas Mook, is clear and concise, and accounts for other common criticisms that are levelled at psychology which I did not cover here (such as the artificiality of the lab setting).

It's a quick, fascinating read on how to understand experimental psychology, and should be requirement for all first-year students. Here, hopefully, I've corrected myself appropriately. I've certainly come out more informed for trying to do so.

---

Mook, D. (1983). In defense of external invalidity. American Psychologist, 38 (4), 379-387 DOI: 10.1037/0003-066X.38.4.379