The Seven Deadly Sins of Psychology Read online

  THE SEVEN DEADLY SINS OF PSYCHOLOGY

  THE SEVEN DEADLY SINS OF PSYCHOLOGY

  A Manifesto for Reforming the Culture of Scientific Practice

  CHRIS CHAMBERS

  PRINCETON UNIVERSITY PRESS

  Princeton & Oxford

  Copyright © 2017 by Chris Chambers

  Requests for permission to reproduce material from this work should be

  sent to Permissions, Princeton University Press

  Published by Princeton University Press, 41 William Street,

  Princeton, New Jersey 08540

  In the United Kingdom: Princeton University Press, 6 Oxford Street,

  Woodstock, Oxfordshire OX20 1TR

  press.princeton.edu

  Artwork for the lead chapter illustrations by Anastasiya Tarasenko

  All Rights Reserved

  ISBN 978-0-691-15890-7

  Library of Congress Control Number: 2016945498

  British Library Cataloging-in-Publication Data is available

  This book has been composed in Whitman and Helvetica Neue

  Printed on acid-free paper. ∞

  Printed in the United States of America

  1 3 5 7 9 10 8 6 4 2

  For J, I and X,

  for every junior scientist whose results weren’t good enough,

  and for JD, who inspired so many but will never see what we might become

  One should be able to see that things are hopeless

  and yet be determined to make them otherwise.

  —F. Scott Fitzgerald

  CONTENTS

  Preface

  ix

  CHAPTER 1. THE SIN OF BIAS 1

  A Brief History of the “Yes Man”

  4

  Neophilia: When the Positive and New Trumps the Negative but True

  8

  Replicating Concepts Instead of Experiments

  13

  Reinventing History

  16

  The Battle against Bias

  20

  CHAPTER 2. THE SIN OF HIDDEN FLEXIBILITY 22

  p-Hacking

  24

  Peculiar Patterns of p

  29

  Ghost Hunting

  34

  Unconscious Analytic “Tuning”

  35

  Biased Debugging

  39

  Are Research Psychologists Just Poorly Paid Lawyers?

  40

  Solutions to Hidden Flexibility

  41

  CHAPTER 3. THE SIN OF UNRELIABILITY 46

  Sources of Unreliability in Psychology

  48

  Reason 1: Disregard for Direct Replication

  48

  Reason 2: Lack of Power

  55

  Reason 3: Failure to Disclose Methods

  61

  Reason 4: Statistical Fallacies

  63

  Reason 5: Failure to Retract

  65

  Solutions to Unreliability

  67

  CHAPTER 4. THE SIN OF DATA HOARDING 75

  The Untold Benefits of Data Sharing

  77

  Failure to Share

  78

  Secret Sharing

  80

  How Failing to Share Hides Misconduct

  81

  Making Data Sharing the Norm

  84

  Grassroots, Carrots, and Sticks

  88

  Unlocking the Black Box

  91

  Preventing Bad Habits

  94

  CHAPTER 5. THE SIN OF CORRUPTIBILITY 96

  The Anatomy of Fraud

  99

  The Thin Gray Line

  105

  When Junior Scientists Go Astray

  112

  Kate’s Story

  117

  The Dirty Dozen: How to Get Away with Fraud

  122

  CHAPTER 6. THE SIN OF INTERNMENT 126

  The Basics of Open Access Publishing

  128

  Why Do Psychologists Support Barrier-Based Publishing?

  129

  Hybrid OA as Both a Solution and a Problem

  132

  Calling in the Guerrillas

  136

  Counterarguments

  138

  An Open Road

  147

  CHAPTER 7. THE SIN OF BEAN COUNTING 149

  Roads to Nowhere

  151

  Impact Factors and Modern-Day Astrology

  151

  Wagging the Dog

  160

  The Murky Mess of Academic Authorship

  163

  Roads to Somewhere

  168

  CHAPTER 8. REDEMPTION 171

  Solving the Sins of Bias and Hidden Flexibility

  174

  Registered Reports: A Vaccine against Bias

  174

  Preregistration without Peer Review

  196

  Solving the Sin of Unreliability

  198

  Solving the Sin of Data Hoarding

  202

  Solving the Sin of Corruptibility

  205

  Solving the Sin of Internment

  208

  Solving the Sin of Bean Counting

  210

  Concrete Steps for Reform

  213

  Coda

  215

  Notes

  219

  Index

  263

  PREFACE

  This book is borne out of what I can only describe as a deep personal frustration with the working culture of psychological science. I have always thought of our professional culture as a castle—a sanctuary of endeavor built long ago by our forebears. Like any home it needs constant care and attention, but instead of repairing it as we go we have allowed it to fall into a state of disrepair. The windows are dirty and opaque. The roof is leaking and won’t keep out the rain for much longer. Monsters live in the dungeon.

  Despite its many flaws, the castle has served me well. It sheltered me during my formative years as a junior researcher and advanced me to a position where I can now talk openly about the need for renovation. And I stress renovation because I am not suggesting we demolish our stronghold and start over. The foundations of psychology are solid, and the field has a proud legacy of discovery. Our founders—Helmholtz, Wundt, James—built it to last.

  After spending fifteen years in psychology and its cousin, cognitive neuroscience, I have nevertheless reached an unsettling conclusion. If we continue as we are then psychology will diminish as a reputable science and could very well disappear. If we ignore the warning signs now, then in a hundred years or less, psychology may be regarded as one in a long line of quaint scholarly indulgences, much as we now regard alchemy or phrenology. Our descendants will smile tolerantly at this pocket of academic antiquity, nod sagely to one another about the protoscience that was psychology, and conclude that we were subject to the “limitations of the time.” Of course, few sciences are likely to withstand the judgment of history, but it is by our research practices rather than our discoveries that psychology will be judged most harshly. And that judgment will be this: like so many other “soft” sciences, we found ourselves trapped within a culture where the appearance of science was seen as an appropriate replacement for the practice of science.

  In this book I’m going to show how this distortion penetrates many aspects of our professional lives as scientists. The journey will be grim in places. Using the seven deadly sins as a metaphor, I will explain how unchecked bias fools us into seeing what we want to see; how we have turned our backs on fundamental principles of the scien
tific method; how we treat the data we acquire as personal property rather than a public resource; how we permit academic fraud to cause untold damage to the most vulnerable members of our community; how we waste public resources on outdated forms of publishing; and how, in assessing the value of science and scientists, we have surrendered expert judgment to superficial bean counting. I will hope to convince you that in the quest for genuine understanding, we must be unflinching in recognizing these failings and relentless in fixing them.

  Within each chapter, and in a separate final chapter, I will recommend various reforms that highlight two core aspects of science: transparency and reproducibility. To survive in the twenty-first century and beyond we must transform our secretive and fragile culture into a truly open and rigorous science—one that celebrates openness as much as it appreciates innovation, that prizes robustness as much as novelty. We must recognize that the old way of doing things is no longer fit for purpose and find a new path.

  At its broadest level this book is intended for anyone who is interested in the practice and culture of science. Even those with no specific interest in psychology have reasons to care about the problems we face. Malpractice in any field wastes precious public funding by pursuing lines of enquiry that may turn out to be misleading or bogus. For example, by suppressing certain types of results from the published record, we risk introducing ineffective clinical treatments for mental health conditions such as depression and schizophrenia. In the UK, where the socioeconomic impact of research is measured as part of a regular national exercise called the Research Excellence Framework (REF), psychology has also been shown to influence a wide range of real-world applications. The 2014 REF reported over 450 “impact case studies” where psychological research has shaped public policy or practice, including (to name just a few) the design and uptake of electric cars, strategies for minimizing exam anxiety, the development of improved police interviewing techniques that account for the limits of human memory, setting of urban speed limits based on discoveries in vision science, human factors that are important for effective space exploration, government strategies for dealing with climate change that take into account public perception of risk, and plain packaging of tobacco products.1 From its most basic roots to its most applied branches, psychology is a rich part of public life and a key to understanding many global problems; therefore the deadly sins discussed here are a problem for society as a whole.

  Some of the content, particularly sections on statistical methods, will be most relevant to the recently embarked researcher—the undergraduate student, PhD student, or early-career scientist—but there are also important messages throughout the book for more senior academics who manage their own laboratories or institutions, and many issues are also relevant to journalists and science writers. To aid the accessibility of source material for different audiences I have referred as much as possible to open access literature. For articles that are not open access, a Google Scholar search of the article title will often reveal a freely available electronic copy. I have also drawn on more contemporary forms of communication, including freely available blog entries and social media.

  I owe a great debt to many friends, academic colleagues, journal editors, science writers, journalists, press officers, and policy experts, for years of inspiration, critical discussions, arguments, and in some cases interviews that fed into this work, including: Rachel Adams, Chris Allen, Micah Allen, Adam Aron, Vaughan Bell, Sven Bestmann, Ananyo Bhattacharya, Dorothy Bishop, Fred Boy, Todd Braver, Björn Brembs, Jon Brock, Jon Butterworth, Kate Button, Iain Chalmers, David Colquhoun, Molly Crockett, Stephen Curry, Helen Czerski, Zoltan Dienes, the late Jon Driver, Malte Elson, Alex Etz, John Evans, Eva Feredoes, Matt Field, Agneta Fischer, Birte Forstmann, Fiona Fox, Andrew Gelman, Tom Hardwicke, Chris Hartgerink, Tom Hartley, Mark Haselgrove, Steven Hill, Alex Holcombe, Aidan Horner, Macartan Humphreys, Hans Ijzerman, Helen Jamieson, Alok Jha, Gabi Jiga-Boy, Ben Johnson, Rogier Kievit, James Kilner, Daniël Lakens, Natalia Lawrence, Keith Laws, Katie Mack, Leah Maizey, Jason Mattingley, Rob McIntosh, Susan Michie, Candice Morey, Richard Morey, Simon Moss, Ross Mounce, Nils Mulhert, Kevin Murphy, Suresh Muthukumaraswamy, Bas Neggers, Neuroskeptic, Kia Nobre, Dave Nussbaum, Hans Op de Beeck, Ivan Oransky, Damian Pattinson, Andrew Przybylski, James Randerson, Geraint Rees, Ged Ridgway, Robert Rosenthal, Pia Rotshtein, Jeff Rouder, Elena Rusconi, Adam Rutherford, Chris Said, Ayse Saygin, Anne Scheel, Sam Schwarzkopf, Sophie Scott, Dan Simons, Jon Simons, Uri Simonsohn, Sanjay Srivastava, Mark Stokes, Petroc Sumner, Mike Taylor, Jon Tennant, Eric Turner, Carien van Reekum, Simine Vazire, Essi Viding, Solveiga Vivian-Griffiths, Matt Wall, Tony Weidberg, Robert West, Jelte Wicherts, Ed Wilding, Andrew Wilson, Tal Yarkoni, Ed Yong, and Rolf Zwaan. Sincere thanks go to Sergio Della Sala and Toby Charkin for their collaboration and fortitude in championing Registered Reports at Cortex, Brian Nosek, David Mellor, and Sara Bowman for providing Registered Reports with such a welcoming home at the Center for Open Science, and to the Royal Society, particularly publisher Phil Hurst and publishing director Stuart Taylor, for embracing Registered Reports long before any other multidisciplinary journal. I also owe a debt of gratitude to Marcus Munafò for joining me in promoting Registered Reports at every turn, and to the 83 scientists who signed our Guardian open letter calling for installation of the format within all life science journals. Finally, I extend a special thanks to Dorothy Bishop, the late (and much missed) Alex Danchev, Dee Danchev, Zoltan Dienes, Pete Etchells, Hal Pashler, Frederick Verbruggen, and E. J. Wagenmakers for extensive draft reading and discussion, to Anastasiya Tarasenko for creating the chapter illustrations, and to my editors Sarah Caro and Eric Schwartz for their patience and sage advice throughout this journey.

  THE SEVEN DEADLY SINS OF PSYCHOLOGY

  CHAPTER 1

  The Sin of Bias

  The human understanding when it has once adopted

  an opinion … draws all things else to support

  and agree with it.

  —Francis Bacon, 1620

  History may look back on 2011 as the year that changed psychology forever. It all began when the Journal of Personality and Social Psychology published an article called “Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect.”1 The paper, written by Daryl Bem of Cornell University, reported a series of experiments on psi or “precognition,” a supernatural phenomenon that supposedly enables people to see events in the future. Bem, himself a reputable psychologist, took an innovative approach to studying psi. Instead of using discredited parapsychological methods such as card tasks or dice tests, he selected a series of gold-standard psychological techniques and modified them in clever ways.

  One such method was a reversed priming task. In a typical priming task, people decide whether a picture shown on a computer screen is linked to a positive or negative emotion. So, for example, the participant might decide whether a picture of kittens is pleasant or unpleasant. If a word that “primes” the same emotion is presented immediately before the picture (such as the word “joy” followed by the picture of kittens), then people find it easier to judge the emotion of the picture, and they respond faster. But if the prime and target trigger opposite emotions then the task becomes more difficult because the emotions conflict (e.g., the word “murder” followed by kittens). To test for the existence of precognition, Bem reversed the order of this experiment and found that primes delivered after people had responded seemed to influence their reaction times. He also reported similar “retroactive” effects on memory. In one of his experiments, people were overall better at recalling specific words from a list that were also included in a practice task, with the catch that the so-called practice was undertaken after the recall task rather than before. On this basis, Bem argued that the participants were able to benefit in the past from practice they had completed in the future.

  As you might expect, Bem’s results generated a flood of confusion and controversy. How could an event in
the future possibly influence someone’s reaction time or memory in the past? If precognition truly did exist, in even a tiny minority of the population, how is it that casinos or stock markets turn profits? And how could such a bizarre conclusion find a home in a reputable scientific journal?

  Scrutiny at first turned to Bem’s experimental procedures. Perhaps there was some flaw in the methods that could explain his results, such as failing to randomize the order of events, or some other subtle experimental error. But these aspects of the experiment seemed to pass muster, leaving the research community facing a dilemma. If true, precognition would be the most sensational discovery in modern science. We would have to accept the existence of time travel and reshape our entire understanding of cause and effect. But if false, Bem’s results would instead point to deep flaws in standard research practices—after all, if accepted practices could generate such nonsensical findings, how can any published findings in psychology be trusted? And so psychologists faced an unenviable choice between, on the one hand, accepting an impossible scientific conclusion and, on the other hand, swallowing an unpalatable professional reality.

  The scientific community was instinctively skeptical of Bem’s conclusions. Responding to a preprint of the article that appeared in late 2010, the psychologist Joachim Krueger said: “My personal view is that this is ridiculous and can’t be true.”2 After all, extraordinary claims require extraordinary evidence, and despite being published in a prestigious journal, the statistical strength of Bem’s evidence was considered far from extraordinary.