banner
You are not using a standards compliant browser. Because of this you may notice minor glitches in the rendering of this page. Please upgrade to a compliant browser for optimal viewing:
Firefox
Internet Explorer 7
Safari (Mac and PC)
Post Archive
2014 (0)2012 (3)2011 (73)
December (1)

Robot Insects
Tuesday, December 6, 2011
November (6)October (5)

BRB
Tuesday, October 25, 2011

Women's Intuition
Friday, October 21, 2011

Thinking with your Fingers
Monday, October 17, 2011

Choice
Tuesday, October 11, 2011

On Souls and Confections...?
Tuesday, October 4, 2011
September (7)

On Souls...?
Thursday, September 29, 2011

Beware the False Consensus Effect!
Saturday, September 24, 2011

Your Preferences - Preliminary Results
Thursday, September 22, 2011

Popularity Survey - DO IT FOR SCIENCE!
Tuesday, September 20, 2011

Zietsch's Response to PZ, Laden and Scicurious.
Tuesday, September 13, 2011

The Personality of Cities
Tuesday, September 6, 2011

Discussion #1 - Walking Speed and City Size
Sunday, September 4, 2011
August (6)

People who Doodle Learn Faster = Bullshit
Thursday, August 25, 2011

Good News Everyone!
Tuesday, August 23, 2011

Got the Time? Part II
Saturday, August 20, 2011

Got the time?
Sunday, August 14, 2011

Can Randomness Predict the Future?
Tuesday, August 9, 2011

The House of Psycasm
Sunday, August 7, 2011
July (7)June (6)May (8)

Part 1: Do We Have Freewill?
Tuesday, May 31, 2011

The Open Letter: A follow-up
Tuesday, May 31, 2011

This is a lie, she said.
Sunday, May 22, 2011

MSPaint is mightier than the Sword
Saturday, May 21, 2011

The Art of Indecision
Wednesday, May 18, 2011

Skeptical Checklist 1.1
Friday, May 13, 2011

The Skeptical Checklist 1.0
Sunday, May 8, 2011

Of Chimps, Children and Post-Grads...
Monday, May 2, 2011
April (5)March (5)February (7)January (10)

Magical Thinking: Voodoo, Prayer, Black Cats, and You
Thursday, January 27, 2011

The Art of Character Creation
Tuesday, January 25, 2011

Video Game Morality: Actions inside the box?
Friday, January 21, 2011

Psychobabble goes live!
Tuesday, January 18, 2011

The Trolley Problem: Who cares?
Sunday, January 16, 2011

Podcast delay and misc. Drugs!?
Saturday, January 15, 2011

My very own Natural Disaster
Wednesday, January 12, 2011

A window into your Genetics and Mate Preference?
Sunday, January 9, 2011

Oh sorry, I totally phased out there...
Thursday, January 6, 2011

Porn: A force of Mutual Benefits
Sunday, January 2, 2011
2010 (35)
December (7)

Statistical Pwnage
Tuesday, December 21, 2010

Why you should care...
Thursday, December 16, 2010

The paper of Influence
Wednesday, December 15, 2010

Sharing: Part I - Emotions
Sunday, December 12, 2010

No-one cuts deeper than a Science Blogger.
Thursday, December 9, 2010

Me Meme [Ohh, links now]
Tuesday, December 7, 2010

Half Full, or Half Empty? Well, That Depends on the Shape of the Glass.
Sunday, December 5, 2010
November (11)

What Your Voice Says About You
Tuesday, November 30, 2010

Babushka Blog: A Meta-Blog on ResearchBlogging.
Sunday, November 28, 2010

An Announcement:
Tuesday, November 23, 2010

Willful Self-Deception is Bliss
Sunday, November 21, 2010

Dance, Blogger, Dance!
Friday, November 19, 2010

The Science of Marriage
Sunday, November 14, 2010

The Moon and Antarctica
Thursday, November 11, 2010

Available: One Mentee. Good Condition.
Tuesday, November 9, 2010

Again we are limited by our puny human-ness
Sunday, November 7, 2010

Blogging Carnival - What is Psychopathology?: Origins
Friday, November 5, 2010

*sigh* Psi: A Rebuttal
Wednesday, November 3, 2010
October (12)

The Science of Mind-Reading
Thursday, October 28, 2010

How not to think yourself smart...
Tuesday, October 26, 2010

Welcome to Assassins' League
Sunday, October 24, 2010

TODO LATER. A story of procrastination and forgiveness.
Thursday, October 21, 2010

A man and his words.
Tuesday, October 19, 2010

Little kids, little minds...
Sunday, October 17, 2010

Smoking (maybemightcould) is Good.
Thursday, October 14, 2010

How to stop the Apocalpyse
Tuesday, October 12, 2010

How to trick yourself creative
Sunday, October 10, 2010

Siesta - It sounds like Fiesta, but isn't.
Thursday, October 7, 2010

The Dread Pirate Rift
Tuesday, October 5, 2010

Being Blonde. Natural or otherwise...
Sunday, October 3, 2010
September (5)
Rate This Post
Total votes: 0
Blogger Profile

Psycasm

Psycasm is the exploration of the world psychological. Every day phenomenon explained and manipulated to one's own advantage. Written by a slightly overambitious undergrad, Psycasm aims at exploring a whole range of social and cognitive processes in order to best understand how our minds, and those mechanisms that drive them, work.

My posts are presented as opinion and commentary and do not represent the views of LabSpaces Productions, LLC, my employer, or my educational institution.

Post Tags

m
Blog RSS Feed
RSS Add to My Yahoo Add to Google
Monday, March 21, 2011

ResearchBlogging.org

I feel its time for a follow-up.

Some time ago I openly mocked Psi researchers for being charlatans. Doing so enraged a vocal minority, but more importantly it brought to my attention the (now infamous) Bem studies (here's a pretty good summary).

And so I composed a rebuttal. It was less science and more 'everyday skepticism' than I had hoped, but I didn't (nor do I currently) have the skill set to demolish it. Fortunately, smarter people than I have.

Here I intend to report on a key critique of the Bem paper. It was authored by Wagen-makers, Wetzels, Borsboom & van der Maas (2011). It is entitled Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi. Many psych people are aware of this, and much of the skeptical community, too. However, I suspect a great many people heard about these studies and, as the media and hype died away, never gave it a second (or critical) thought- presumably leaving the deceptively sweet taste of 'magic powers' lingering somewhere in their consciousness. I suspect a great many more did call bullshit on the paper, on first principles (as I did), but a more thorough dissection (on my part) is due.

1) Bem confuses (or ignores) the difference between Exploratory and Confirmatory research

The following is a direct rip from the Wagen-makers paper, quoting a passage Bem wrote for a book-chapter.

“To compensate for this remoteness from our participants, let us at least
become intimately familiar with the record of their behavior: the data. Examine
them from every angle. Analyze the sexes separately. Make up new composite
indexes. If a datum suggests a new hypothesis, try to find further evidence
for it elsewhere in the data. If you see dim traces of interesting patterns, try to
reorganize the data to bring them into bolder relief. If there are participants you
don’t like, or trials, observers, or interviewers who gave you anomalous results,
place them aside temporarily and see if any coherent patterns emerge. Go on a
fishing expedition for something–anything–interesting.” (Bem, 2000, pp. 4-5)

'Fishing Expedition'. Wow. Now my experience is generally limited, but my first year stats class taught me that fishing trips are a BAD THING. Wagen-makers argue that fishing trips are fine, so long as they are acknowledged as such before hand, are not presented as evidence for a thing (rather than the suggestion of its possibility). Whenever I've heard of academics stumbling across an unexpected phenomenon in a data set, it's usually followed by a series of more controlled and falsifiable experiments aimed at establishing if it was a freak-occurrence, or a genuine piece of serendipity.

It's basic statistics - the more unplanned analyses one runs, the greater the likelihood of 'discovering' a false hit. This appears to be what Bem engaged in - without good theoretical reasons Bem split the data down gender lines. Additionally, it appears he engaged in some questionable data transformations that appear unnecessary.

2) Bem ignores (or dismisses) prior probabilities

I think I covered this in my rebuttal post, but the key points are -

We have no good reason to suspect Psi exists. Here's a light take on the problem, but does outline the shape of the argument.


If such capacities existed (Psi not mentioned) then it would have been exploited. Wagen-makers uses the example of a casino. If a Psi exists, and individuals posses it, then they should have rolled the casinos by now. In all seriousnessprior probabilities can be quantified (and thus, are made falsifiable). Here's what it looks like:

Crazy, huh? You'll have to read the paper because my abilities to explain what's going on here are not sufficient to do so succinctly (or accurately). The point is, however, that in calculating a prior probability the value can be argued, counter-argued, and potentially agreed upon. This value can then be factored in to future (and even past) analyses to render a more consistent and meaningful conclusion.

3) p-values are inherently flawed

This is where Wagen-makers puts forth his pet project of rejecting p-values in favour of the more sophisticated Bayesian t-tests.

This is from the wiki article of Bayesian Inference:

In practical usage, "Bayesian inference" refers to the use of a prior probability over hypotheses to determine the likelihood of a particular hypothesis given some observed evidence; that is, the likelihood that a particular hypothesis is true given some observed evidence (the so-called posterior probability of the hypothesis) comes from a combination of the inherent likelihood (or prior probability) of the hypothesis and the compatibility of the observed evidence with the hypothesis (or likelihood of the evidence, in a technical sense). Bayesian inference is opposed to frequentist inference, which makes use only of the likelihood of the evidence (in the technical sense), discounting the prior probability of the hypothesis.

The trick is that p-values make judgments based on the likelihood of data being false/wrong, whereas Bayesian analysis work on the probability that the data is true/correct.

Wagen-makers gives an excellent and highly accessible interview on the Skeptics' Guide to the Universe Podcast (episode here) where he explains and defends his position well. I recommend it. And not only because I know nothing about Bayesian statistics.

4) The confusion between Exploratory and Confirmatory experimentation (part II)

This is what Wagen-makers claims makes good confirmatory experimentation.

a) No more fishing trips. Have a good and falsifiable hypothesis before you start.

b) Don't play silly-buggers with statistical techniques. Don't 'torture the data until they confess' as Bem endorses.

c) Know what analysis you intend to conduct before hand. Don't choose them post hoc.

d) Conduct complementary analyses of the same data. Conflicts should be reported.

e) Be transparent. Give us your code, your stimuli, and your data sets.

f) Know thy enemy as thy friend. That is, if you're messing around with Psi bring in Randy. Collaborate with those who disagree. A good scientist will work to create good science, even if it stands to topple the world they stand on. A skeptic would love to see mind-reading, and you can bet a skeptic is going to provide the appropriate rigor.

I have to say, points a, b and c were covered in my first year statistics course. d is new, but I can see the reasoning. e and f are aimed squarely at Psi. However, I would love to see more transparency (as per e) in so much research. When reading a paper sometime I just want to experience/view/evaluate the stimuli for myself, but it's often not accessible.

---

Ultimately the arguments put forth in the Wagen-makers paper are well explained and well presented. They probably require a little prior knowledge, but I've done what I can to make it a little more accessible (without oversimplyfing it; hopefully). I highly recommend reading the paper for yourself. [I obtained a copy of this paper from the following link. I'm hesitant to link directly myself, because it goes to a dropbox address. Here's the Neurological link, and the link to the Wagen-makers paper is about 1/7th the way down the page, as eye-balled on the scroll bar.]

I do believe this is enough food for thought for those in the Pro Psi camp who argued so passionately last time.

---

Wagenmakers EJ, Wetzels R, Borsboom D, & van der Maas HL (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of personality and social psychology, 100 (3), 426-32 PMID: 21280965

Bem, D. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100 (3), 407-425 DOI: 10.1037/a0021524

This post has been viewed: 4676 time(s)

Blog Comments

yannisguerra
Rate Post:

Like 0 Dislike

Ah. Good thinking in an article. It's usually hard to refute brilliance. I am not sure that you will hear from the Psych supporters this time.

 

Add Comment?
Comments are closed 2 weeks after initial post.
Friends