Blog Post »

Information Overload (or maybe I’m just a slow reader)

The Decade of Navel Gazing (2010–???) took another step toward cementing its legacy this week with the publication of Nosek and Bar-Anan’s proposal for Scientific Utopia. The authors present a six-step program which would help psychology increase the efficiency, speed, and thoroughness of its communication and dissemination of knowledge, taking us to the promised land of quick, available, and ubiquitous new research. Let there be no mistake: this system would be fast! Tired of waiting two years for your completed studies to matriculate their way into your colleagues’ mailboxes in the form of a hard-bound, print-format journal? The complete digitization of research reports will allow all new papers to be uploaded instantly to an open-access forum where you could see it within hours of the final t-tests being conducted! Nervous that your dissertation will flop in the peer-review process—due to the seemingly cavalier musings of an anonymous reviewer—after years of grunt and toil? The open-access forum will house all peer-reviews, as well as commentaries on peer-reviews, providing a way for you to receive instant feedback on your published paper from any researcher in the field and to immediately update your paper to address this feedback! Utopia, indeed! No more publication lags, no more publication drags, and no more blind-sided reviewer red flags!

Scientific Utopia to some, but to me—and many of the commentators—it sounds like information overload in the form of increasingly shallow, hasty, and scattered research. Nelson, Simmons, & Simonsohn note that an open-access forum will relax any checks on the undisclosed and flexible research practices they have previously highlighted, causing our field to become even more bloated with short-reports, sloppy studies, and false positive findings. Barbara Spellman notes that, with manuscripts appearing in ever-increasing abundance, psychologists will experience mind-spinning confusion when trying to separate sound from shoddy research. Such confusion will make it nearly impossible for researchers to consolidate their knowledge about any particular research domain; theoretical progress through meta-analysis will become extremely difficult. Finally, Jens Asendorpf notes that the authors make no provision for quality checks on articles before they are published—only after they are published—such as the gate-keeping roles currently played by journal editors. An open-access forum would, in the words of Laura King’s commentary, cause journal editors to become akin to “whoever it is that runs Rotten Tomatos.” Journal editors are scientists who have been selected by their peers as experts, and the open-access forum would strip them of the ability to use theat expertise to encourage the publication of quality research. In sum, scientific utopia would result in a cacophony of unchecked research proliferation, thereby slowing the cumulative, consolidative process through which psychology makes broad theoretical advances.

But Nosek and Bar-Anan are admittedly pitching efficiency and speed, so to critique the effect of their proposal on research quality somewhat misses the point. Yet, in two crucial ways, Scientific Utopia would degrade the efficiency with which the individual scientist conducts business. First, consider the researcher who attempts to read the literature under this new system. This poor soul will spend hours of his day a) reading poorly written articles describing shoddy research that have nevertheless entered the open-access forum due to the lack of checks on publication; and b) reading articles that, perhaps even by the time he finishes them, no longer exist as they did when he started to read them, a result of researchers’ ability to continuously update their papers in response to criticism. In Scientific Utopia, the individual scientist could spend an entire day sifting through the open-access forum before reading and digesting a single article that retains some semblance of both quality and permanence.

Second, consider the researcher who writes an empirical paper for publication. This tireless worker will spend weeks honing an introduction, crafting a methods and results section, and generally discussing the reported studies. Yet, in Scientific Utopia, the majority of those written pages will go unread. Even the shortest brief reports are at least a few thousand words in length, and longer articles can span 20-30 PDF pages. That’s a lot of words, and unless I’m a really slow reader, an open system in which the number of papers available increases dramatically would make reading all those words impossible. Nosek and Bar-Anan themselves admit to receiving new article alerts from no less than six different sources (e.g., journals, RSS, bloggers), some of them arriving on a daily basis. Can such productive and prolific scientists—who surely have other tasks to complete beyond reading—possibly read all those articles with depth, completeness, and thoughtfulness? Seems unlikely. Now, an easy retort would state that a scientist does not need to read an article front-to-back to extract the relevant knowledge. That’s reasonable, but it begs the question of why we spend so much time and effort crafting long articles in the first place? If we do make strides toward quickening the pace with which we disseminate and consume research, we need to consider the inefficiency of devoting time and effort to crafting long, thoughtful manuscripts, if many of those thoughts will go entirely unnoticed.

But shouldn’t science embrace depth and thoughtfulness of ideas, even if those ideas take a little longer to develop and reach the public domain? Even in this late-summer season, a soft, ripe peach tastes way better than a hard, crunchy peach. The issue of research quality and depth vs. research quantity and efficiency seems to form the crux of the debate begun by Nosek and Bar-Anan. Please chime in with more thoughts!

Discussion (2 Comments)

Jess

Thanks for summarizing some of the key issues here, Aaron! Having read the Nosek paper and, so far, just one of the commentaries (King), I find myself pretty confused about what problems this system is supposed to solve. There are certainly problems with the current system–most notably, issues like false positives and the (very very rare) fraudulent data, which have generated the whole navel gazing we’ve been doing in the first place– but it’s not clear how this new very open system would solve either, any more so than the current system. And, I completely agree with King that Nosek really underestimates the importance of editors and gate keeping in keeping our science good. Yes, not all editors are great, and some are terrible, but the solution to that problem isn’t to get rid of editors altogether! There are plenty of ways of making editors more accountable, and, truth be told, in most cases I think editors are accountable.

Beyond this, I completely agree with Aaron’s sense that what Nosek proposes would create a system that makes it incredibly difficult for consumers of science to know how to best spend our time. It also would require a tremendous amount MORE time on the part of authors. Reading the final section of the Nosek paper, in which they describe what the utopia would look like for the typical researcher/academic, I must say it read much more like a dystopia than a utopia, in my mind. Where, in the midst of all the responding to reviews, commenting on others’ articles, debating with other commentators, and reading comments on one’s own papers (some of which may have been ‘published’ many years prior), is there any time for doing research? Nosek suggests that there will be some distribution of labor where some academics become largely reviewers, and others become largely researchers; in his mind, this is a good thing, but such a division seems almost certain to come with status differentiation as well, and it’s hard to imagine that this system wouldn’t eventually result in very few people doing reviews because of the sense that reviewers are somehow ‘less than’ real researchers. Though I agree that it’s problematic that the current system does not reward reviewers in a formalized manner, there’s a big part of me that thinks this is actually pretty awesome; we academics do reviewing because we know it’s the right thing to do for our science. And, there are tangible rewards–the editors know which reviewers are good and which aren’t, and it’s hard to imagine that that knowledge doesn’t contribute to editorial decision-making down the road.

I look forward to reading the rest of the commentaries, but, so far, I’m mostly pretty perplexed about why Nosek and Bar-Anon think that such a major overhaul of the system will be useful. Quicker availability of findings is just one aspect of efficiency– on the whole, it seems to me this system will largely slow things down, and, even if not, any increased efficiency will sure come at the cost of quality.

One last thing– I do agree with one of Nosek’s suggestions–to increase the number of online journals, and reduce the publishing houses’s stranglehold on researchers. I also like the idea of having data sets published online– I’m not sure if Nosek suggested this here, but I know it’s something he and others have argued for in the past.

Nick Holtzman

Another great post Aaron. One good thing about an open online system, however, is that it would potentially increase reliability of the reviews because there are more potential reviewers. It is really hard for just two or three people to arrive at a reliable rating of a manuscript, but usually there are only two or three reviewers in the traditional method.

Make a Comment »