PSYC 422 Lecture 3

A/Prof Deborah Apthorp

Power, bias and research ethics - how to be a better researcher (Part II)

Slides are open source

Link for the HTML version of these slides (always up to date)

So again, please note that the slides for this are available online if you click on the link or use the QR code. There’s also a PDF on mylearn if you prefer that.

Outline

• Cautionary Tales from Psychology research
• Questionable Research Practices
• Possible solutions
• Replicability and replication projects
• How open should we be?
• Preregistration and Registered Reports
• What you can do!

So here is an outline of what we’re going to talk about today. First, some cautionary tales from the last few years that have become part of the folklore about what triggered the so-called “Reproducbility Crisis” or “Replication Crisis”. Then we’ll talk about some possible solutions to the crisis. We’ll talk about what replicability is, and discuss a few replication projects. Then we’ll examine the question of how open we should or can be in our research. I want to talk briefly about preregistration and Registered Reports - you may have heard a little about this already if you came to my talk at the Honours intensive. And lastly I want to suggest things you can do to help!

Advice of a famous social psychologist

“There are two possible articles you can write: (1) the article you planned to write when you designed your study or (2) the article that makes the most sense now that you have seen the results. They are rarely the same, and the correct answer is (2).”

-Bem, 2003, pp. 171-172

A cautionary tale I: Daryl Bem

• Daryl Bem, 2011, Journal of Personality & Social Psychology (not open access - cited 822 times)
• Respected researcher, top-tier journal
• Published 9 experiments which claimed to show effects of precognition
• Took established psychological experiments & time-reversed them
• Showed evidence of precognition in undergraduate students!

Bem used a variety of techniques but the general approach was to “time reverse” established psychological effects. For example, the experiment that produced the largest effect size (Experiment 9) took as its starting point the trivial observation that memory for words is better if one is allowed to rehearse the words as opposed to being exposed to them just once. Of course, this usually involves rehearsing the words before one’s memory for them is tested.

The astonishing claim made by Bem – apparently supported by his experimental data – was that memory for words is improved even if the rehearsal does not take place until after recall has been tested. The effect was dubbed the “retroactive facilitation of recall”.

A cautionary tale I: Daryl Bem

• Encouraged others to replicate - even provided software
• Researchers from 3 institutions (Stuart Ritchie, Chris French, and Richard Wiseman) decided to do this
• Pre-registered
• Failed to replicate any of the effects
• See “A Student’s Guide to Open Science” Chapter 2

So what happened next?

A cautionary tale I: Daryl Bem

• Submitted to original journal (JPSP)
• Rejected without review - “we do not publish replications”
• Also rejected without review by Science Brevia & Psychological Science
• Reviewed at British Journal of Psychology - rejected (Bem was a reviewer)
• Finally published in PLoS One (“Failing the Future” - open access - cited 152 times)

So note that the original has been cited over 800 times - from the previous slide. But are these all uncritical citations?

Also note that the article was criticised heavily on publication, and other researchers also failed to replicate it - see textbook for details.

Side note - citations?

…do they support, mention or contradict?

• New platform scite.ai uses machine learning to determine what KIND of citations a paper gets
• Supporting, mentioning, contradicting
• Most studies “mention” Bem rather than supporting
• Of the 5 “supporting” citations, 3 are by Bem himself and the other 2 don’t actually support it

Just as a side note, there’s a newish platform called Scite that, unlike ChatGPT, is actually useful for looking at scientific papers! On this platform you can have a look at whether statements around citations of a paper support it, mention it, or contradict the original paper. If you have a look at the citations on scite, most of them are actually just mentioning or contradicting Bem (for instance, using it as an example of QRPs)

NOTE TO SELF: scite is no longer free - should I still recommend it?

A cautionary tale II: Diderick Stapel

• Diderick Stapel - famous Dutch social psychologist
• Received the “Career Trajectory Award” from the Society of Experimental Social Psychology in 2009
• 130 articles, 24 book chapters
• Key findings: Messy environments promote stereotyping; meat eaters “more selfish” than vegetarians
• Students never saw raw data; results “too good to be true”?
• (Also covered in Chapter 2 of “A Student’s Guide to Open Science”)

So here’s another cautionary tale for you about a famous social psychologist called Diderick Stapel. He published lots of eye-catching papers with seemingly “common-sense” results. But the students never saw the raw data - he kept that very close to his chest and was very particular about it.

A cautionary tale II: Diderick Stapel

• Eventually, students blew the whistle
• All the data had been fabricated
• At least 58 articles retracted
• How did he go unchallenged for so long?
• Why did he do it?
• Free translation of his book here!

Stapel’s book was called “Ontsporing” in Dutch, which means “Derailment”. It was originally published in Dutch, but Nick Brown has translated it and offers it for free at this link. In it, Stapel goes into some detail about what led him to fake data, which is pretty unusual - you don’t get to hear this from most people who do it. Some of it seems to have been a desire to show things as they “ought to be” - neat rather than messy, showing things that should be obviously true.

Quote

“Outright fraud is somewhat impossible to estimate, because if you’re really good at it you wouldn’t be detectable,” said Simonsohn, a social psychologist. “It’s like asking how much of our money is fake money – we only catch the really bad fakers, the good fakers we never catch.”

So this is a slightly chilling quote, I think, from an an article that’s published here. We don’t really know how much fraud is out there, but it’s probably more than we’d like.

A cautionary tale III - John Bargh

• Classic finding: “social priming” - participants walked out of lab more slowly after being primed with “elderly”-related words (Bargh, Chen & Burrows, 1996)
• Cited over 2000 times - but not replicated often
• Stephane Doyen & colleagues attempted replication
• A few changes - more Ss, double blinding, and infrared sensors
• Effect did not replicate

This time, the priming words had no impact on the volunteers’ walking speed. They left the test room neither more slowly nor more quickly than when they arrived. Doyen suspected that Bargh’s research team could have unwittingly told their volunteers how they were meant to behave… Perhaps they themselves moved more slowly if they expected the volunteer to do so. Maybe they spoke more languidly, or shook hands more leisurely… Maybe they were responsible for creating the very behaviour they expected to see.

To test that idea, Doyen repeated his experiment with 50 fresh volunteers and 10 fresh experimenters. The experimenters always stuck to the same script, but they knew whether each volunteer had been primed or not. Doyen told half of them that people would walk more slowly thanks to the power of priming, but he told the other half to expect faster walks.

…He found that the volunteers moved more slowly only when they were tested by experimenters who expected them to move slowly… Let that sink in: the only way Doyen could repeat Bargh’s results was to deliberately tell the experimenters to expect those results.

Barghed!

• Ed Yong’s (2012) piece about this is here (OSF version here).

So Bargh retaliated by publishing this piece in Psychology today, entitled “Nothing in Their Heads”. In it, he directs personal attacks at the authors of the paper, calling them “incompetent or ill-informed”, and at the journal PLoS, saying that it “does not receive the usual high scientific journal standards of peer-review scrutiny”, and also at the journalist Ed Yong who wrote the piece about it, calling it “superficial online science journalism”.

Questionable research practices (QRPs)

• Fig. 1. The nine circles of scientific hell (with apologies to Dante and xkcd). © Association for Psychological Science.

This is from a brief and quite humorous article by the anonymous blogger Neuroskeptic - I encourage you to read it. The article gives a suggested taxonomy of Questionable Research Practices, or QRPs, which is a useful starting point for talking about what they are. I’ll go over each in a bit more detail.

Neuroskeptic, 2012

• First Circle: Limbo
• Second Circle: Overselling
• Third Circle: Post-Hoc Storytelling
• Fourth Circle: p Value Fishing
• Fifth Circle: Creative Use of Outliers

• Sixth Circle: Plagiarism
• Seventh Circle: Non-publication of Data
• Eighth Circle: Partial Publication of Data
• Ninth Circle: Inventing Data

Limbo? “Those who have committed no scientific sins per se, but who have turned a blind eye to them” Overselling - clickbait/hype/abstracts that don’t represent the results

Post-hoc storytelling = HARKing

P-value fishing otherwise known as p-hacking

Creative use of outliers - also p-hacking (related to optional stopping??)

Plagiarism - obviously everyone knows from undergrad, but what about self-plagiarism (some v high profile cases recently)

Non-publication = file drawer

Partial publication - way too common. Have been asked to do it by reviewers myself.

Inventing data - see Stapel. Also might be more common than we think (if people are good at it we won’t catch them). Also now more common to use AI to fabricate things.

How common are QRPs?

So, how common actually are QRPs? These authors decided to try to find out by asking researchers directly. They used some incentives for truth-telling, which we’re going to find out a bit more about in a minute.

There’s a great podcast about this paper here which you can have a listen to if you’re interested.

Prevalence of QRPs in Psychology

Bayesian truth serum??

• Half subjects told that the truthfulness of their responses would be determined by an algorithm (BTS)
• This would determine the value of a donation to the charity of their choice
• This was true!
• The use of BTS increased % who admitted to QRPs, particularly those they judged less defensible
• Overall percentage very high!!

So, for what they called “Bayesian Truth Serum”, they told half the subjects they’d determine how truthful their responses were using an algorithm, which would determine the value of a donation to their charity of choice. This was actually true! Also, it increased the percentage of those who admitted to QRPs. Specually those they thought were worse. What’s interesting here, though, is that the overall percentage is very high!

And here you can see Figure 1 from the paper. In black is the self-admitted amount, in white is how prevalent the respondents THOUGHT the practice was, and in grey is the prevalence estimate that was determined from the admission estimates (the percentages are the geometric mean of each of the three bars). So you can see that they were still pretty prevalent back on 2012. Have things changed for the better? You’d certainly hope so.

Update: 2024 QRP study

• That study was published in 2012. Surely things have improved by now?

• A more recent paper (Schneider et al., 2024) examined QRPs across “3,402 researchers in Denmark and 1,307 inthe UK, USA, Croatia and Austria”

• Estimates for individual self-reported use of 25 QRPs ranged from 10 to 64% (Danish sample - 11 to 65 in international sample)

• Most common: Selective over-citing of own publications (64-65%)

• Least common: Plagiarising other researchers’ unpublished ideas (10 -11%)

So the QRPs listed in here are not all the same as those in the John et al. paper, so perhaps it’s not really comparable. Still, it’s disappointing that things haven’t improved more.

Schneider et al. 2024 - Figure 2

This is Figure 2 from the paper. It’s a bit hard to read here, but you should be able to refer to the paper to have a look. The Danish sample is in blue, the international one in grey.

Why are we here?

• As scientists, we want to know what is true.
• Systematic replications are a tool of scientific progress:
  • How reliable is the data published in journals?
  • What practices would lead to a higher level of replicability?

The Reproducibility Project (RP)

• Aim: How reproducible are results in Psychology?
• 255 volunteer researchers
• 64 universities
• 11 countries
• Attempt to replicate 97 key findings in Psychology

Estimating the reproducibility of psychological science

• Now published in “Science”
• Only about half of studies replicated
• Generated lots of comment!
• Check it out at Science
• Podcast about it here.

Basic findings

• Approximately \(2/3^{rds}\) of studies did not replicate using a p-value cutoff
• (Cancer Biology: 54%)

Overall, 36% of the replications yielded significant findings (p value below .05) compared to 97% of the original studies that had significant effects The mean effect size in in the replications was approximately half the magnitude of the effects reported in the original studies. Studies in the field of cognitive psychology had a higher replication rate (50%) than studies in in the field of social psychology (25%).

Basic findings

• No original study with a \(p > . 025\) replicated
• When combined in meta-analysis with original studies, \(1/3^{rd}\) no longer had sufficient evidence for existing

Effect sizes and p-values

• Almost all replications had a smaller effect size than originals
• Diagonal line = equal effect size

So the diagonal line represents where the replication effect size would be equal to original effect size. The dotted line represents a replication effect size of 0. Points below the dotted line show effects in the opposite direction of the original. And the density plots are separated by significant (green) and nonsignificant (red) effects.

Overall, the original study effect sizes (M = 0.403, SD = 0.188) were reliably larger than replication effect sizes (M = 0.197, SD = 0.257), Wilcoxon’s W = 7137, P < 0.001.

Comparing effect sizes across indicators, surprisingness of the original effect, and the challenge of conducting the replication were related to replication success for some indicators. Surprising effects were less reproducible, as were effects for which it was more challenging to conduct the replication.

Explanations for replication failures

• Original studies:
• Emphasis on innovation
• Emphasis on positive results
• Low power
• Flexibility in data analysis/collection
• Focus on clean and tidy results

• Replication studies:
• False negatives
• Sampling error
• Low fidelity

Investigating variability in replicability: Many Labs project

• One possible way of making psychology studies more robust is to distribute them across multiple labs
• This was the genesis of the first Many Labs project - now known as Many Labs 1 (OSF site here)
• Also offered the opportunity to look at variations across sample and setting
• Now followed up by Many Labs 2, 3 and 4 (5, Many Babies 1–6, #EEGMany Labs, and Many Dogs (those are the ones I know of!)

Many Labs 1 - findings

• 13 classic findings; 36 independent samples; 6,344 participants.
• Preregistered protocol
• 10 effects replicated, 1 borderline, 2 didn’t
• Not much difference between US and international studies
• Criticism - papers were chosen as more likely to replicate

Many Labs 2

Investigating replicability across sample and setting

• Many Labs 2 - now published in Advances in Methods & Practices in Psychological Science (open access)
• 186 researchers, 28 classic & contemporary findings, 60 laboratories
• MEDIAN sample size 7,157!
• 14/28 failed to replicate (by strict criterion of p <.0001) in spite of very large sample size
• Effect size less than half of original, on average
• Little evidence that replication depended on sample

This is a little tricky to reac but you can see the original effect sizes are the little grey diamonds, and each little bar is an individual lab’s result.

Prediction markets

• A second related paper had people bet on which papers would replicate
• Now published in Journal of Economic Psychology
• Markets correctly predicted 75% of the replication outcomes
• Survey & markets also predicted effect sizes
• Can you predict which studies will replicate?
• Play the online game and find out!

A second paper “Predicting Replication Outcomes in the Many Labs 2 Study” has been published in the Journal of Economic Psychology. This paper reported evidence that researchers participating in surveys and prediction markets about the Many Labs 2 findings could predict which of the studies were likely to replicate and which were not. In prediction markets, each share for a finding that successfully replicates is worth $1 and each share for a finding that fails to replicate is worth nothing. Researchers then buy and sell shares in each finding to predict which ones will succeed and fail to replicate. The final market price is interpretable as the predicted probability that the original finding will replicate. Anna Dreber, senior author of the prediction market paper, and Professor at the Stockholm School of Economics and University of Innsbruck said “We now have four studies successfully demonstrating that researchers can predict whether findings will replicate or not in surveys and prediction markets with pretty good accuracy. This suggests potential reforms in peer review of grants and papers to help identify findings that are exciting but highly uncertain to invest resources to see if they are replicable.”

Other Many Labs projects

• Many Labs 3 - Evaluating participant pool quality across the academic semester via replication
• Many Labs 4 - Failure to replicate Mortality Salience effect
• Many Labs 5 - Testing pre-data-collection peer review as an intervention to increase replicability
• Many Babies - Many Babies 1 now published - projects up to no. 4!
• A thoughtful critique of the Many Labs projects by Charlie Ebersole (one of the project leaders).
• Useful summary here

In Many Labs 5, this was testing the claim in the Reproducibility project that the failure to replicate was due to researchers not adhering to the original protocols. Spoiler alert - it wasn’t.

A similar approach - the Psych Science Accelerator

• A recent initiative (formed 2017), aimed at providing psychological science with much greater power
• Website here - introductory paper here
• Currently 2468 researchers representing 73 countries on all 6 populated continents
• New studies and replications; democratically selected
• Podcast here (note - I get a mention at the start!)

Other initiatives to solve the “reproducibility crisis”

• Open access publishing
• Open data
• Open code
• Preprints
• Preregistration
• Registered Reports

• Double blind peer review
• Open peer review
  • Publish peer reviews
• Pay peer reviewers?
• Post-publication peer review

Open access publishing

Issues

• Early open access journals (e.g. PLoS One) were established to break the selectivity of traditional journals for “novelty” and publishing only significant results
• Currently, most (but not all!) open access journals require the author to pay for publication
• Costs vary (PeerJ - $399 - 499 for life; PLoS One - US$1350 per article; Nature - US$11,390!)
• Costs can be claimed on grants - but not everyone has a grant
• Proliferation of “predatory” open access journals

Open data

• In some fields, this has been the norm for decades (astronomy, climate science)
• In psychology, still very uncommon (even less so in neuroscience)
• Recently, some journals have introduced policies that authors must share their data
• PLoS - introduced policy in 2014, but not always adhered to
• Psych Science - Badges (but are they any use? Aha! Funny that you ask…)

The Data Badge Project

Origins

• It all began with a tweet
• Nick Brown called for volunteers to help with a project
• All articles in the April 2019 issue of Psychological Science earned an “Open Data” badge (the first issue where this was the case)
• Could a team of researchers reproduce the published results?
• Of course I signed up!

The Data Badge Project

Organisation

• Final team consisted of 12 researchers
• Varying levels of experience - Ph.D students & upwards
• Each paper allocated to at least 3 reproducers
• Communication via Google Groups
• First stage - independent reproduction attempts
• Uploaded to OSF
• Phase 1 reports

The Data Badge Project

Team

• Sohpia Crüwell
• Me
• Bradley J. Baker
• Lincoln Colling
• Malte Elson
• Sandra J. Geiger

• Sebastian Lobentanzer
• Jean Monéger
• Alex Patterson
• D. Samuel Schwarzkopf
• Mirela Zaneva
• Nicholas J. L. Brown

The Data Badge Project

My experience

• I was initially assigned 4 papers to work on
• All had data, 2 had code (MATLAB)
• Time taken per paper: between 2-3 hours and 2 days
• One without code was easy to reproduce (simple analyses)
• The other without code had VERY raw data
• Took days, could not exactly reproduce results
• The two with code were not any easier

The Data Badge Project

Phase 2 Reports

• We did not commence this stage until each paper had at least 3 Phase 1 reports
• I ended up working on a 5th paper due to drop-outs
• We communicated via threads on Google Groups
• Sophia Crüwell took over as first author
• Collaborated on Google Docs for summary reports
• Agreed on ratings for each paper: How reproducible? Exactly/Essentially/Partially/Mostly Not/Not at all

The Data Badge Project - Results!

Individual ratings

The Data Badge Project - Results!

Group ratings

The Data Badge Project

Conclusions

• Do badges work? Not on their own
• Journals offering badges could provide more support for authors
• Criteria for awarding badges should be clear and explicit (e.g. code, readme, raw data)

• Badge checks
  • Authors could provide evidence of independent code checks
  • Journals could provide this service (via editorial staff or peer review)

The Data Badge Project

Publication

• We submitted to Psychological Science in April 2022
• To our surprise, not rejected!
• Generally positive, constructive peer reviews
• After revision, reviewers happy
• HOWEVER, editor insisted on several more rounds of changes
• Now published in Psych Science!! (Cited 48 times already!)

Preprints

• It’s becoming more common to post papers to “preprint servers” before peer review
• Again, this has been traditional in Physics, Astrophysics and Mathematics for decades (arXiv was founded in 1991).
• Now bioRxiv, PsyArXiv, etc.
• Advantages: Open access, free, immediate access to research findings
• Disadvantages: Not peer reviewed, can post anything (including rubbish science)
• Much of this came to light during COVID.

Enter PCI!

• Peer Community In (PCI) is a new initiative offering free preprint review & recommendation
• Currently 19 different “communities” and 2000 recommenders
• Once reviewed, can publish in Peer Community Journal (diamond open access) or submit to a PCI-friendly journal as pre-reviewed
• Could this overtake commercial journals?

Preregistration

• You can now publicly declare your research plan before collecting any data
• This would ideally reduce researcher degrees of freedom, p-hacking and publication bias
• Should increase accountability and credibility
• Should refocus peer review on the study phase
• Some researchers argue it’s not necessary (see “Is Preregistration Worthwhile?”)

Registered reports

• Becoming MUCH more common recently (see here for details & list of journals which accept them)
• Step 1: Submit Intro, Method only; peer reviewed thoroughly.
• Step 2: Collect data and analyse as planned
• Step 3: Peer review 2 (check everything done as planned)
• Step 4: Publish! (regardless of results)

Registered reports

Infographic from the CoS

Registered reports

• Currently >300 journals accept Registered Report formats
• Registered reports cut publication bias (Allen & Mehler, 2019, PLoS Biology)

Registered reports cut publication bias

Reforming peer review

Double-blind peer review

• Already in place at some journals
• Like a double-blind clinical trial: authors don’t know who they’re reviewing; reviewers don’t know who the authors are
• Objections: In practice, it will be fairly obvious who the authors are (self-citations, small fields, etc)
• It may also become obvious who reviewers are (we currently try to guess anyway)

Reforming peer review

Open peer review

• Several models
• Authors see reviews and reviewers see each others’ reviews; names revealed after publication (Frontiers)
• Entire process open & reviews published alongside paper (PeerJ)
• Argument: Junior reviewers will be too intimidated to be honest about senior colleagues’ work?
• Plus side: Get credit for your time spent reviewing
• Increased transparency of process

Reforming peer review

Post-publication peer review

• Idea: publish everything, let the reviews sort it out afterwards (e.g. PubPeer)
• Problem: Need chance for revision?
• Problem: People won’t participate in review (majority of publications will be unreviewed)
• Solution: Incremental review process?
• Solution: Credit for good reviewers
• Solution: Borrow from social media e.g. Reddit?
• Note: eLife recently transitioned to this model amid some controversy!

What you can do

• If you’re doing Honours next year, consider preregistering your project at OSF (even for this year it’s not too late)
• Could we do replications in 3rd year? Or even in Honours? Lobby!
• APAC Accreditation Public Consultation is currently open here
• Some ideas for participation from OSF
• FORRT (Framework for Open and Reproducible Research Training) - get involved here!

Collaborative replication for undergraduates (CREP)

• The Collaborative Replications and Education Project
• A crowdsourced replication project for undergraduate researchers
• Can become an author on the final paper if involved enough
• Project website here

Questions? Ask on MyLearn!