Reporter and podcaster Jesse Singal has given us a wonderful look into how and why social sciences promote junk science. The Quick Fix: Why Fad Psychology Can’t Cure Our Social Ills (April 6, 2021) should be required reading in rhetoric courses that examine either science or public policy. Research methods courses should also include this book in their reading lists.
Singal has a history of factual reporting that upsets progressives and conservatives, while he describes himself as center-left. Singal is one of the few true examples of “cancel culture” at work.
I know progressive colleagues who automatically reject anything by Singal. He’s committed a great many “sins” against the progressive agenda, asking questions that might have annoying answers.
He correctly observes that many of the assumptions made by the loud Twitter-verse on the left are for their comfort. Incomplete or misleading data? That’s okay if it promotes the narratives of oppression, racism, classism, sexism, xenophobia, and other social justice causes. Dare to point out the errors of omission and you’re castigated.
Dare to question scholars aligned with the social justice movements, and you’ll find yourself alongside Singal, Sam Harris, and John McWhorter.
Signal points to TED Talks as a popular platform for progressively minded researchers. Find an audience for your claims, get a book deal, and go on tour. TED Talks allow scholars to be the public intellectuals they desire to be.
Teaching, I often turn to TED Talks and parodies of TED Talks for examples of what’s wrong with technical rhetoric in the United States. These well-delivered presentations offer some of the best public speaking examples available for analysis. They also reflect our desire for easy answers to complex questions.
TED Talk speakers take complex research, working papers, and academic theories and package these for general audiences. Okay, maybe not general audiences. No, the TED Talk target audiences are upper-middle-class, college-educated, expert-trusting, knowledge economy workers and their children. The best schools now teach high school and college students how to deliver these YouTube-ready works of performance teaching.
There’s certainly irony in the many TED Talks that warn us that traditional lectures, standardized tests, intelligence aren’t predictors of success. Yet, data suggest “grit” alone isn’t enough. TED Talks are meant to comfort the knowledge workers, to assuage any guilt they have for succeeding and being at the top of our culture.
I enjoy Malcolm Gladwell’s books, essays, and reporting. However, he is the equivalent of literary TED Talks. He packages nice little bits of social psychology, neuropsychology, behavioral economics, and other research to deliver punchy “truths” that overstate the actual research findings. Gladwell gives us hope that 10,000 hours will make us experts. That we can determine the perfect words for love. That we can overcome our subconscious decision-making.
The truth is, most “soft science” research has serious replication, reliability, and internal validity issues. Sociology, psychology, and similar fields have weaknesses that the dominant political biases in these fields mask.
Researcher Ulrich Schimmack has devised a model to expose the replication issues within these soft sciences.
The R-Index blog was created by Ulrich Schimmack on December 1, 2014.
The purpose of the R-Index blog is to increase the replicability of published results in psychological science.
What is the R-Index? The R-Index is a statistical tool to examine the replicability of empirical studies.
My vision is that providing information about replicability of published results creates an incentive for researchers and editors of scientific journal to publish results that other researchers can replicate (with the same amount of resources) and that provide a solid foundation for psychological theories of human behavior.
There is no arguing that the soft sciences have a genuine replication problem, a true crisis of trust in the published data and conclusions. Even left-leaning publications such as Vox have reported on the extent of the replication crisis in psychology.
Singal writes about what the researchers admit: their studies fail to meet scientific standards. Psychology, sociology, and behavioral economics conduct research that falls far, far short of what is possible in chemistry or physics. Testing humans isn’t easy.
Too many researchers only seek out the answers they want. That’s particularly true in sociology and political science. The faculty at major universities overwhelmingly self-identify as progressives, socialists, and even Marxists (Gross & Simmons, 2007). The research these prestigious faculty pursue reflects biases, just as what stories a newspaper decides to cover reflects biases.
In our current environment, questioning research on racial bias approaches heresy. Singal, however, points to the implicit-association test (IAT) as an example of social psychology research that fails to meet replication standards. It’s junk science, but saying so is assumed to reveal the racism of the person questioning the sloppy scholarship. It’s a trap: don’t dare question studies on race, or we will call you a racist.
Schimmack explains the IAT problem in detail, which Signal cannot since The Quick Fix is intended for general readers and not quantitative researchers. First, Schimmack establishes the broader problem in psychology and sociology.
Junk Science: Bias in the Implicit Bias Literature
Ulrich Schimmack
December 4, 2020For decades, psychologists have misused the scientific method and statistical significance testing. Instead of using significance tests to confirm or falsify theoretical predictions, they only published statistically significant results that confirmed predictions. This selection for significance undermines the ability of statistical tests to distinguish between true and false hypotheses (Sterling, 1959).
Another problem is that psychologists ignore effect size. Significant results with the nil-hypothesis (no effect) only reject the hypothesis that the effect size is not zero. It is still possible that the population effect size is so small that it has no practical significance. In the 1990s, psychologists addressed this problem by publishing standardized effect sizes. The problem is that selection for significance also inflates these effect size estimates. Thus, journals may publish effect size estimates that seem important, when the actual effect sizes are trivial.
The impressive reproducibility project (OSC, 2015) found that original effect sizes were cut in half in replication studies that did not select for significance. In other words, population effect sizes are, on average, inflated by 100%. Importantly, this average inflation applied equally to cognitive and social psychology. However, social psychology has more replication failures which also implies larger inflation of effect sizes. Thus, most published effect sizes in social psychology are likely to provide misleading information about the actual effect sizes.
The problems with social psychology research rest in the methodologies adopted. Again, this explanation is technical, yet important to offer since it reinforces that the disagreements are not mere opinions. Signal’s critiques reflect problems with data gathering, data analyses, and research conclusions. Schimmack explains the effect size issue in particular:
As I pointed out in my criticism of research practices in social psychology (Schimmack, 2012), other paradigms in social psychology have produced equally shocking inflation of effect sizes.
One possible explanation is that researchers do not care about effect sizes. Researchers may not consider it unethical to use questionable research methods that inflate effect sizes as long as they are convinced that the sign of the reported effect is consistent with the sign of the true effect. For example, the theory that implicit attitudes are malleable is supported by a positive effect of experimental manipulations on the implicit association test, no matter whether the effect size is d = .8 (Dasgupta & Greenwald, 2001) or d = .08 (Joy-Gaba & Nosek, 2010), and the influence of blood glucose levels on self-control is supported by a strong correlation of r = .6 (Gailliot et al., 2007) and a weak correlation of r = .1 (Dvorak & Simons, 2009).
How have IAT researchers responded to the realization that original effect sizes may have been dramatically inflated? Not much. Citations show that the original article with the 10 times inflated effect size is still cited much more frequently than the replication study with a trivial effect size.
Closer inspection of these citations shows that implicit bias researchers continue to cite the old study as if it provided credible evidence.
Not all IAT citations are sloppy. Some scholars are careful to point out that even if IAT results are accurate, they might not be generalizable. Also, assuming short-term exposure to positive or negative depictions of a group affect IAT participants, that does not suggest that sensitivity training, “white awareness” training, or other programs result in any long-term behavioral changes.
Using IAT to make people aware of their biases, and then use these results during training, has no support in research.
A positive example that cites Nosek and Joy-Gaba (2010) correctly comes from an outsider.
Natalie Salmanowitz’s article writes in Journal of Law and the Biosciences that “a short, impersonal exposure to counterstereotypical exemplars cannot be expected to counteract a lifetime of ingrained mental associations” (p. 180).
As a university instructor, I have to endure hours and hours of sensitivity training annually. Does this training change opinions? Does it make the workplace better? Considering my experiences as a disabled person, no. Not in the least. Sadly, people learn to give the “right” answers on tests and to trainers, and then go on being the jerks they were before the training.
Teaching the penalties for breaking non-discrimination laws seems to have a greater effect than trying to open minds and hearts. We’ve seen this in broader society, too. Sometimes, the threat of consequences is the best way to halt bigots. Mandated training makes us feel like we’re agents of change. (Of course, many employers adopted these training courses to avoid legal liability later when an employee does discriminate against a coworker or client.)
Schimmack writes:
In conclusion, science is self-correctiong, IAT researchers are not self-correcting, therefore IAT research is not science until IAT researchers are honest about the research practices that produced dramatically inflated effect sizes and irreproducible results. Open practices alone are not enough. Honesty and a commitment to pursing the truth (rather than fame or happiness) is essential for scientific progress.
What’s troubling is that IAT is but one example among many such “testing instruments” and research projects that lack replicability. We buy into these fads because we want simple, easy fixes. Other examples include these “researched” claims, many with accompanying TED Talks and media tours.
- Teach children “pride” and they will do better on tests.
- Adopt the right physical pose and you’ll be more confident.
- Avoid cold air or you’ll seem distant and uninviting.
- Promoting a positive self-image can prevent PTSD.
Sadly, we’ll keep buying into fads, if Signal’s hypothesis is correct. We should probably test it.