I feel its time for a follow-up.

Some time ago I openly mocked Psi researchers for being charlatans. Doing so enraged a vocal minority, but more importantly it brought to my attention the (now infamous) Bem studies (here's a pretty good summary).

And so I composed a rebuttal. It was less science and more 'everyday skepticism' than I had hoped, but I didn't (nor do I currently) have the skill set to demolish it. Fortunately, smarter people than I have.

Here I intend to report on a key critique of the Bem paper. It was authored by Wagen-makers, Wetzels, Borsboom & van der Maas (2011). It is entitled Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi. Many psych people are aware of this, and much of the skeptical community, too. However, I suspect a great many people heard about these studies and, as the media and hype died away, never gave it a second (or critical) thought- presumably leaving the deceptively sweet taste of 'magic powers' lingering somewhere in their consciousness. I suspect a great many more did call bullshit on the paper, on first principles (as I did), but a more thorough dissection (on my part) is due.

1) Bem confuses (or ignores) the difference between Exploratory and Confirmatory research

The following is a direct rip from the Wagen-makers paper, quoting a passage Bem wrote for a book-chapter.

“To compensate for this remoteness from our participants, let us at least
become intimately familiar with the record of their behavior: the data. Examine
them from every angle. Analyze the sexes separately. Make up new composite
indexes. If a datum suggests a new hypothesis, try to find further evidence
for it elsewhere in the data. If you see dim traces of interesting patterns, try to
reorganize the data to bring them into bolder relief. If there are participants you
don’t like, or trials, observers, or interviewers who gave you anomalous results,
place them aside temporarily and see if any coherent patterns emerge. Go on a
fishing expedition for something–anything–interesting.” (Bem, 2000, pp. 4-5)

'Fishing Expedition'. Wow. Now my experience is generally limited, but my first year stats class taught me that fishing trips are a BAD THING. Wagen-makers argue that fishing trips are fine, so long as they are acknowledged as such before hand, are not presented as evidence for a thing (rather than the suggestion of its possibility). Whenever I've heard of academics stumbling across an unexpected phenomenon in a data set, it's usually followed by a series of more controlled and falsifiable experiments aimed at establishing if it was a freak-occurrence, or a genuine piece of serendipity.

It's basic statistics - the more unplanned analyses one runs, the greater the likelihood of 'discovering' a false hit. This appears to be what Bem engaged in - without good theoretical reasons Bem split the data down gender lines. Additionally, it appears he engaged in some questionable data transformations that appear unnecessary.

2) Bem ignores (or dismisses) prior probabilities

I think I covered this in my rebuttal post, but the key points are -

We have no good reason to suspect Psi exists. Here's a light take on the problem, but does outline the shape of the argument.

If such capacities existed (Psi not mentioned) then it would have been exploited. Wagen-makers uses the example of a casino. If a Psi exists, and individuals posses it, then they should have rolled the casinos by now. In all seriousnessprior probabilities can be quantified (and thus, are made falsifiable). Here's what it looks like:

Crazy, huh? You'll have to read the paper because my abilities to explain what's going on here are not sufficient to do so succinctly (or accurately). The point is, however, that in calculating a prior probability the value can be argued, counter-argued, and potentially agreed upon. This value can then be factored in to future (and even past) analyses to render a more consistent and meaningful conclusion.

3) p-values are inherently flawed

This is where Wagen-makers puts forth his pet project of rejecting p-values in favour of the more sophisticated Bayesian t-tests.

This is from the wiki article of Bayesian Inference:

In practical usage, "Bayesian inference" refers to the use of a prior probability over hypotheses to determine the likelihood of a particular hypothesis given some observed evidence; that is, the likelihood that a particular hypothesis is true given some observed evidence (the so-called posterior probability of the hypothesis) comes from a combination of the inherent likelihood (or prior probability) of the hypothesis and the compatibility of the observed evidence with the hypothesis (or likelihood of the evidence, in a technical sense). Bayesian inference is opposed to frequentist inference, which makes use only of the likelihood of the evidence (in the technical sense), discounting the prior probability of the hypothesis.

The trick is that p-values make judgments based on the likelihood of data being false/wrong, whereas Bayesian analysis work on the probability that the data is true/correct.

Wagen-makers gives an excellent and highly accessible interview on the Skeptics' Guide to the Universe Podcast (episode here) where he explains and defends his position well. I recommend it. And not only because I know nothing about Bayesian statistics.

4) The confusion between Exploratory and Confirmatory experimentation (part II)

This is what Wagen-makers claims makes good confirmatory experimentation.

a) No more fishing trips. Have a good and falsifiable hypothesis before you start.

b) Don't play silly-buggers with statistical techniques. Don't 'torture the data until they confess' as Bem endorses.

c) Know what analysis you intend to conduct before hand. Don't choose them post hoc.

d) Conduct complementary analyses of the same data. Conflicts should be reported.

e) Be transparent. Give us your code, your stimuli, and your data sets.

f) Know thy enemy as thy friend. That is, if you're messing around with Psi bring in Randy. Collaborate with those who disagree. A good scientist will work to create good science, even if it stands to topple the world they stand on. A skeptic would love to see mind-reading, and you can bet a skeptic is going to provide the appropriate rigor.

I have to say, points a, b and c were covered in my first year statistics course. d is new, but I can see the reasoning. e and f are aimed squarely at Psi. However, I would love to see more transparency (as per e) in so much research. When reading a paper sometime I just want to experience/view/evaluate the stimuli for myself, but it's often not accessible.

---

Ultimately the arguments put forth in the Wagen-makers paper are well explained and well presented. They probably require a little prior knowledge, but I've done what I can to make it a little more accessible (without oversimplyfing it; hopefully). I highly recommend reading the paper for yourself. [I obtained a copy of this paper from the following link. I'm hesitant to link directly myself, because it goes to a dropbox address. Here's the Neurological link, and the link to the Wagen-makers paper is about 1/7th the way down the page, as eye-balled on the scroll bar.]

I do believe this is enough food for thought for those in the Pro Psi camp who argued so passionately last time.

---

Wagenmakers EJ, Wetzels R, Borsboom D, & van der Maas HL (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of personality and social psychology, 100 (3), 426-32 PMID: 21280965

Bem, D. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100 (3), 407-425 DOI: 10.1037/a0021524