Information Overload (or maybe I’m just a slow reader)

The Decade of Navel Gazing (2010–???) took another step toward cementing its legacy this week with the publication of Nosek and Bar-Anan’s proposal for Scientific Utopia. The authors present a six-step program which would help psychology increase the efficiency, speed, and thoroughness of its communication and dissemination of knowledge, taking us to the promised land of quick, available, and ubiquitous new research. Let there be no mistake: this system would be fast! Tired of waiting two years for your completed studies to matriculate their way into your colleagues’ mailboxes in the form of a hard-bound, print-format journal? The complete digitization of research reports will allow all new papers to be uploaded instantly to an open-access forum where you could see it within hours of the final t-tests being conducted! Nervous that your dissertation will flop in the peer-review process—due to the seemingly cavalier musings of an anonymous reviewer—after years of grunt and toil? The open-access forum will house all peer-reviews, as well as commentaries on peer-reviews, providing a way for you to receive instant feedback on your published paper from any researcher in the field and to immediately update your paper to address this feedback! Utopia, indeed! No more publication lags, no more publication drags, and no more blind-sided reviewer red flags!

Scientific Utopia to some, but to me—and many of the commentators—it sounds like information overload in the form of increasingly shallow, hasty, and scattered research. Nelson, Simmons, & Simonsohn note that an open-access forum will relax any checks on the undisclosed and flexible research practices they have previously highlighted, causing our field to become even more bloated with short-reports, sloppy studies, and false positive findings. Barbara Spellman notes that, with manuscripts appearing in ever-increasing abundance, psychologists will experience mind-spinning confusion when trying to separate sound from shoddy research. Such confusion will make it nearly impossible for researchers to consolidate their knowledge about any particular research domain; theoretical progress through meta-analysis will become extremely difficult. Finally, Jens Asendorpf notes that the authors make no provision for quality checks on articles before they are published—only after they are published—such as the gate-keeping roles currently played by journal editors. An open-access forum would, in the words of Laura King’s commentary, cause journal editors to become akin to “whoever it is that runs Rotten Tomatos.” Journal editors are scientists who have been selected by their peers as experts, and the open-access forum would strip them of the ability to use theat expertise to encourage the publication of quality research. In sum, scientific utopia would result in a cacophony of unchecked research proliferation, thereby slowing the cumulative, consolidative process through which psychology makes broad theoretical advances.

But Nosek and Bar-Anan are admittedly pitching efficiency and speed, so to critique the effect of their proposal on research quality somewhat misses the point. Yet, in two crucial ways, Scientific Utopia would degrade the efficiency with which the individual scientist conducts business. First, consider the researcher who attempts to read the literature under this new system. This poor soul will spend hours of his day a) reading poorly written articles describing shoddy research that have nevertheless entered the open-access forum due to the lack of checks on publication; and b) reading articles that, perhaps even by the time he finishes them, no longer exist as they did when he started to read them, a result of researchers’ ability to continuously update their papers in response to criticism. In Scientific Utopia, the individual scientist could spend an entire day sifting through the open-access forum before reading and digesting a single article that retains some semblance of both quality and permanence.

Second, consider the researcher who writes an empirical paper for publication. This tireless worker will spend weeks honing an introduction, crafting a methods and results section, and generally discussing the reported studies. Yet, in Scientific Utopia, the majority of those written pages will go unread. Even the shortest brief reports are at least a few thousand words in length, and longer articles can span 20-30 PDF pages. That’s a lot of words, and unless I’m a really slow reader, an open system in which the number of papers available increases dramatically would make reading all those words impossible. Nosek and Bar-Anan themselves admit to receiving new article alerts from no less than six different sources (e.g., journals, RSS, bloggers), some of them arriving on a daily basis. Can such productive and prolific scientists—who surely have other tasks to complete beyond reading—possibly read all those articles with depth, completeness, and thoughtfulness? Seems unlikely. Now, an easy retort would state that a scientist does not need to read an article front-to-back to extract the relevant knowledge. That’s reasonable, but it begs the question of why we spend so much time and effort crafting long articles in the first place? If we do make strides toward quickening the pace with which we disseminate and consume research, we need to consider the inefficiency of devoting time and effort to crafting long, thoughtful manuscripts, if many of those thoughts will go entirely unnoticed.

But shouldn’t science embrace depth and thoughtfulness of ideas, even if those ideas take a little longer to develop and reach the public domain? Even in this late-summer season, a soft, ripe peach tastes way better than a hard, crunchy peach. The issue of research quality and depth vs. research quantity and efficiency seems to form the crux of the debate begun by Nosek and Bar-Anan. Please chime in with more thoughts!

Categories: Lab Musings