The Seven Deadly Sins of Psychology
THE SEVEN DEADLY SINS OF PSYCHOLOGY
THE SEVEN DEADLY SINS OF PSYCHOLOGY
A Manifesto for Reforming the Culture of Scientific Practice
CHRIS CHAMBERS
PRINCETON UNIVERSITY PRESS
Princeton & Oxford
Copyright © 2017 by Chris Chambers
Requests for permission to reproduce material from this work should be
sent to Permissions, Princeton University Press
Published by Princeton University Press, 41 William Street,
Princeton, New Jersey 08540
In the United Kingdom: Princeton University Press, 6 Oxford Street,
Woodstock, Oxfordshire OX20 1TR
press.princeton.edu
Artwork for the lead chapter illustrations by Anastasiya Tarasenko
All Rights Reserved
ISBN 978-0-691-15890-7
Library of Congress Control Number: 2016945498
British Library Cataloging-in-Publication Data is available
This book has been composed in Whitman and Helvetica Neue
Printed on acid-free paper. ∞
Printed in the United States of America
1 3 5 7 9 10 8 6 4 2
For J, I and X,
for every junior scientist whose results weren’t good enough,
and for JD, who inspired so many but will never see what we might become
One should be able to see that things are hopeless
and yet be determined to make them otherwise.
—F. Scott Fitzgerald
CONTENTS
Preface
ix
CHAPTER 1. THE SIN OF BIAS 1
A Brief History of the “Yes Man”
4
Neophilia: When the Positive and New Trumps the Negative but True
8
Replicating Concepts Instead of Experiments
13
Reinventing History
16
The Battle against Bias
20
CHAPTER 2. THE SIN OF HIDDEN FLEXIBILITY 22
p-Hacking
24
Peculiar Patterns of p
29
Ghost Hunting
34
Unconscious Analytic “Tuning”
35
Biased Debugging
39
Are Research Psychologists Just Poorly Paid Lawyers?
40
Solutions to Hidden Flexibility
41
CHAPTER 3. THE SIN OF UNRELIABILITY 46
Sources of Unreliability in Psychology
48
Reason 1: Disregard for Direct Replication
48
Reason 2: Lack of Power
55
Reason 3: Failure to Disclose Methods
61
Reason 4: Statistical Fallacies
63
Reason 5: Failure to Retract
65
Solutions to Unreliability
67
CHAPTER 4. THE SIN OF DATA HOARDING 75
The Untold Benefits of Data Sharing
77
Failure to Share
78
Secret Sharing
80
How Failing to Share Hides Misconduct
81
Making Data Sharing the Norm
84
Grassroots, Carrots, and Sticks
88
Unlocking the Black Box
91
Preventing Bad Habits
94
CHAPTER 5. THE SIN OF CORRUPTIBILITY 96
The Anatomy of Fraud
99
The Thin Gray Line
105
When Junior Scientists Go Astray
112
Kate’s Story
117
The Dirty Dozen: How to Get Away with Fraud
122
CHAPTER 6. THE SIN OF INTERNMENT 126
The Basics of Open Access Publishing
128
Why Do Psychologists Support Barrier-Based Publishing?
129
Hybrid OA as Both a Solution and a Problem
132
Calling in the Guerrillas
136
Counterarguments
138
An Open Road
147
CHAPTER 7. THE SIN OF BEAN COUNTING 149
Roads to Nowhere
151
Impact Factors and Modern-Day Astrology
151
Wagging the Dog
160
The Murky Mess of Academic Authorship
163
Roads to Somewhere
168
CHAPTER 8. REDEMPTION 171
Solving the Sins of Bias and Hidden Flexibility
174
Registered Reports: A Vaccine against Bias
174
Preregistration without Peer Review
196
Solving the Sin of Unreliability
198
Solving the Sin of Data Hoarding
202
Solving the Sin of Corruptibility
205
Solving the Sin of Internment
208
Solving the Sin of Bean Counting
210
Concrete Steps for Reform
213
Coda
215
Notes
219
Index
263
PREFACE
This book is borne out of what I can only describe as a deep personal frustration with the working culture of psychological science. I have always thought of our professional culture as a castle—a sanctuary of endeavor built long ago by our forebears. Like any home it needs constant care and attention, but instead of repairing it as we go we have allowed it to fall into a state of disrepair. The windows are dirty and opaque. The roof is leaking and won’t keep out the rain for much longer. Monsters live in the dungeon.
Despite its many flaws, the castle has served me well. It sheltered me during my formative years as a junior researcher and advanced me to a position where I can now talk openly about the need for renovation. And I stress renovation because I am not suggesting we demolish our stronghold and start over. The foundations of psychology are solid, and the field has a proud legacy of discovery. Our founders—Helmholtz, Wundt, James—built it to last.
After spending fifteen years in psychology and its cousin, cognitive neuroscience, I have nevertheless reached an unsettling conclusion. If we continue as we are then psychology will diminish as a reputable science and could very well disappear. If we ignore the warning signs now, then in a hundred years or less, psychology may be regarded as one in a long line of quaint scholarly indulgences, much as we now regard alchemy or phrenology. Our descendants will smile tolerantly at this pocket of academic antiquity, nod sagely to one another about the protoscience that was psychology, and conclude that we were subject to the “limitations of the time.” Of course, few sciences are likely to withstand the judgment of history, but it is by our research practices rather than our discoveries that psychology will be judged most harshly. And that judgment will be this: like so many other “soft” sciences, we found ourselves trapped within a culture where the appearance of science was seen as an appropriate replacement for the practice of science.
In this book I’m going to show how this distortion penetrates many aspects of our professional lives as scientists. The journey will be grim in places. Using the seven deadly sins as a metaphor, I will explain how unchecked bias fools us into seeing what we want to see; how we have turned our backs on fundamental principles of the scien
tific method; how we treat the data we acquire as personal property rather than a public resource; how we permit academic fraud to cause untold damage to the most vulnerable members of our community; how we waste public resources on outdated forms of publishing; and how, in assessing the value of science and scientists, we have surrendered expert judgment to superficial bean counting. I will hope to convince you that in the quest for genuine understanding, we must be unflinching in recognizing these failings and relentless in fixing them.
Within each chapter, and in a separate final chapter, I will recommend various reforms that highlight two core aspects of science: transparency and reproducibility. To survive in the twenty-first century and beyond we must transform our secretive and fragile culture into a truly open and rigorous science—one that celebrates openness as much as it appreciates innovation, that prizes robustness as much as novelty. We must recognize that the old way of doing things is no longer fit for purpose and find a new path.
At its broadest level this book is intended for anyone who is interested in the practice and culture of science. Even those with no specific interest in psychology have reasons to care about the problems we face. Malpractice in any field wastes precious public funding by pursuing lines of enquiry that may turn out to be misleading or bogus. For example, by suppressing certain types of results from the published record, we risk introducing ineffective clinical treatments for mental health conditions such as depression and schizophrenia. In the UK, where the socioeconomic impact of research is measured as part of a regular national exercise called the Research Excellence Framework (REF), psychology has also been shown to influence a wide range of real-world applications. The 2014 REF reported over 450 “impact case studies” where psychological research has shaped public policy or practice, including (to name just a few) the design and uptake of electric cars, strategies for minimizing exam anxiety, the development of improved police interviewing techniques that account for the limits of human memory, setting of urban speed limits based on discoveries in vision science, human factors that are important for effective space exploration, government strategies for dealing with climate change that take into account public perception of risk, and plain packaging of tobacco products.1 From its most basic roots to its most applied branches, psychology is a rich part of public life and a key to understanding many global problems; therefore the deadly sins discussed here are a problem for society as a whole.
Some of the content, particularly sections on statistical methods, will be most relevant to the recently embarked researcher—the undergraduate student, PhD student, or early-career scientist—but there are also important messages throughout the book for more senior academics who manage their own laboratories or institutions, and many issues are also relevant to journalists and science writers. To aid the accessibility of source material for different audiences I have referred as much as possible to open access literature. For articles that are not open access, a Google Scholar search of the article title will often reveal a freely available electronic copy. I have also drawn on more contemporary forms of communication, including freely available blog entries and social media.
I owe a great debt to many friends, academic colleagues, journal editors, science writers, journalists, press officers, and policy experts, for years of inspiration, critical discussions, arguments, and in some cases interviews that fed into this work, including: Rachel Adams, Chris Allen, Micah Allen, Adam Aron, Vaughan Bell, Sven Bestmann, Ananyo Bhattacharya, Dorothy Bishop, Fred Boy, Todd Braver, Björn Brembs, Jon Brock, Jon Butterworth, Kate Button, Iain Chalmers, David Colquhoun, Molly Crockett, Stephen Curry, Helen Czerski, Zoltan Dienes, the late Jon Driver, Malte Elson, Alex Etz, John Evans, Eva Feredoes, Matt Field, Agneta Fischer, Birte Forstmann, Fiona Fox, Andrew Gelman, Tom Hardwicke, Chris Hartgerink, Tom Hartley, Mark Haselgrove, Steven Hill, Alex Holcombe, Aidan Horner, Macartan Humphreys, Hans Ijzerman, Helen Jamieson, Alok Jha, Gabi Jiga-Boy, Ben Johnson, Rogier Kievit, James Kilner, Daniël Lakens, Natalia Lawrence, Keith Laws, Katie Mack, Leah Maizey, Jason Mattingley, Rob McIntosh, Susan Michie, Candice Morey, Richard Morey, Simon Moss, Ross Mounce, Nils Mulhert, Kevin Murphy, Suresh Muthukumaraswamy, Bas Neggers, Neuroskeptic, Kia Nobre, Dave Nussbaum, Hans Op de Beeck, Ivan Oransky, Damian Pattinson, Andrew Przybylski, James Randerson, Geraint Rees, Ged Ridgway, Robert Rosenthal, Pia Rotshtein, Jeff Rouder, Elena Rusconi, Adam Rutherford, Chris Said, Ayse Saygin, Anne Scheel, Sam Schwarzkopf, Sophie Scott, Dan Simons, Jon Simons, Uri Simonsohn, Sanjay Srivastava, Mark Stokes, Petroc Sumner, Mike Taylor, Jon Tennant, Eric Turner, Carien van Reekum, Simine Vazire, Essi Viding, Solveiga Vivian-Griffiths, Matt Wall, Tony Weidberg, Robert West, Jelte Wicherts, Ed Wilding, Andrew Wilson, Tal Yarkoni, Ed Yong, and Rolf Zwaan. Sincere thanks go to Sergio Della Sala and Toby Charkin for their collaboration and fortitude in championing Registered Reports at Cortex, Brian Nosek, David Mellor, and Sara Bowman for providing Registered Reports with such a welcoming home at the Center for Open Science, and to the Royal Society, particularly publisher Phil Hurst and publishing director Stuart Taylor, for embracing Registered Reports long before any other multidisciplinary journal. I also owe a debt of gratitude to Marcus Munafò for joining me in promoting Registered Reports at every turn, and to the 83 scientists who signed our Guardian open letter calling for installation of the format within all life science journals. Finally, I extend a special thanks to Dorothy Bishop, the late (and much missed) Alex Danchev, Dee Danchev, Zoltan Dienes, Pete Etchells, Hal Pashler, Frederick Verbruggen, and E. J. Wagenmakers for extensive draft reading and discussion, to Anastasiya Tarasenko for creating the chapter illustrations, and to my editors Sarah Caro and Eric Schwartz for their patience and sage advice throughout this journey.
THE SEVEN DEADLY SINS OF PSYCHOLOGY
CHAPTER 1
The Sin of Bias
The human understanding when it has once adopted
an opinion … draws all things else to support
and agree with it.
—Francis Bacon, 1620
History may look back on 2011 as the year that changed psychology forever. It all began when the Journal of Personality and Social Psychology published an article called “Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect.”1 The paper, written by Daryl Bem of Cornell University, reported a series of experiments on psi or “precognition,” a supernatural phenomenon that supposedly enables people to see events in the future. Bem, himself a reputable psychologist, took an innovative approach to studying psi. Instead of using discredited parapsychological methods such as card tasks or dice tests, he selected a series of gold-standard psychological techniques and modified them in clever ways.
One such method was a reversed priming task. In a typical priming task, people decide whether a picture shown on a computer screen is linked to a positive or negative emotion. So, for example, the participant might decide whether a picture of kittens is pleasant or unpleasant. If a word that “primes” the same emotion is presented immediately before the picture (such as the word “joy” followed by the picture of kittens), then people find it easier to judge the emotion of the picture, and they respond faster. But if the prime and target trigger opposite emotions then the task becomes more difficult because the emotions conflict (e.g., the word “murder” followed by kittens). To test for the existence of precognition, Bem reversed the order of this experiment and found that primes delivered after people had responded seemed to influence their reaction times. He also reported similar “retroactive” effects on memory. In one of his experiments, people were overall better at recalling specific words from a list that were also included in a practice task, with the catch that the so-called practice was undertaken after the recall task rather than before. On this basis, Bem argued that the participants were able to benefit in the past from practice they had completed in the future.
As you might expect, Bem’s results generated a flood of confusion and controversy. How could an event in
the future possibly influence someone’s reaction time or memory in the past? If precognition truly did exist, in even a tiny minority of the population, how is it that casinos or stock markets turn profits? And how could such a bizarre conclusion find a home in a reputable scientific journal?
Scrutiny at first turned to Bem’s experimental procedures. Perhaps there was some flaw in the methods that could explain his results, such as failing to randomize the order of events, or some other subtle experimental error. But these aspects of the experiment seemed to pass muster, leaving the research community facing a dilemma. If true, precognition would be the most sensational discovery in modern science. We would have to accept the existence of time travel and reshape our entire understanding of cause and effect. But if false, Bem’s results would instead point to deep flaws in standard research practices—after all, if accepted practices could generate such nonsensical findings, how can any published findings in psychology be trusted? And so psychologists faced an unenviable choice between, on the one hand, accepting an impossible scientific conclusion and, on the other hand, swallowing an unpalatable professional reality.
The scientific community was instinctively skeptical of Bem’s conclusions. Responding to a preprint of the article that appeared in late 2010, the psychologist Joachim Krueger said: “My personal view is that this is ridiculous and can’t be true.”2 After all, extraordinary claims require extraordinary evidence, and despite being published in a prestigious journal, the statistical strength of Bem’s evidence was considered far from extraordinary.