Peer Review in the Internet Age: Five (5) Easy Pieces
Gerry McKiernan
"... [In] the digital world, the evaluation process stands
ready to be reinvented in a clear, rational way by the relevant research
communities themselves."1
Jean-Claude Guédon
In general,
peer review can be defined as "the assessment by an expert of material
submitted for publication."2 Specifically, its purpose is to
ensure that published research is important, internally-consistent, original,
technically-reliable, timely, well-presented, and most importantly, benefited
from guidance by experts.3 Overall, "the underlying strength of
editorial peer review is the concerted effort by large numbers of researchers
and scholars who work to assure that valid and valuable works are published,
and conversely to assure that invalid or non-valuable works are not
published."4
While established peer review has its supporters, it has long been criticized as "... slow, expensive, profligate of academic time, highly subjective, prone to bias, easily abused, poor at detecting gross defects, and almost useless in detecting fraud."5 In a recent review article on the peer review process, Rowland analyzes and briefly characterizes many of the deficiencies of classic peer review as follows:
Summary rejections by editor without sending the paper to referees; choice of referees by the editor (choosing for example, a known harsh referee for a paper the editor wishes to see rejected);
Discrimination against authors because of their nationality, native language, gender or host institution; situations where author and referee are competitors in some sense, or belong to competing schools of thought;
Too many articles out of one piece of research, or duplicate publication; intellectual theft: omission or downgrading of junior staff by senior authors; plagiarism (stealing others yet unpublished work that has been sent for review), delaying publication of potentially competing research;
Fabrication of results; falsification of data false claim of
authorship for results.6
Among the most
common complaints about peer review is the long delay associated with the
review process. As observed by Stevan Harnad, an active proponent of
author-self-archiving and institutional repositories, "[t]here is much
muttering about publication delay, a real enough problem, especially in paper
publication, but peer review itself is often responsible for as much of the
delay as the paper publication and distribution process itself."7
Scientific
Publishing as Rhetoric
For some, the fundamental problems of peer review are inherent in
the peer review process itself as it is currently implemented. As noted by
Sosteric,
"the traditional mode
of peer review obscures the problems of reference and the rhetorical dimension
of science. The rhetorical process ... [that] is at the heart of science and
peer review conveniently disappears with the final publication of the
manuscript. In its place is an ideal typical representation (the scientific
paper) of the realist assumptions about empirical reference. All the academic
world sees is a polished manuscript where the personal involvement of the
researcher and reviewers has been systematically eliminated.8
As an alternative to conventional peer review, Sosteric, Gross,
and others, promote the framework of the 'ideal speech situation', a
"theoretical construct that describes the ideal type of interpersonal
interaction that should exist in a rhetorical situation” proposed and developed
by Jürgen Habermas, the noted German philosopher and sociologist.9 Drawing
upon Habermas, Gross describes the ideal speech situation in the following
terms: 1) the ideal speech situation permits each interlocutor an equal
opportunity to initiate speech. 2) there is mutual understanding between
interlocutors; 3) there is space for clarification; 4) all interlocutors are
equally free to use of any speech act; and, 5) there is equal power over the
exchange.10
As applied in the context of peer review, Gross notes that ideally
"scientific peer review would permit unimpeded authorial initiative,
endless rounds of give and take, [and] unchecked openness among authors,
editors, and referees.11
"
... The Times They Are A-Changin'"
In 2001, CERN, the European Organisation for Nuclear Research, in
Two years after the workshop, a policy briefing of the European
Science Foundation was published. The publication not only profiled the variety
of issues relating to Open Access and the OAI, but also summarized the themes
of the first OAI workshop, and the second held at CERN in mid-October 2002, as
well. In addition, the briefing included consensus recommendations from each
workshop. While the "participants [of the first workshop] were unanimous
in their belief that the certification of scholarly work remains a fundamental
part of a system for scholarly communication," they also "believed that
the electronic environment allows for novel approaches to accord a stamp of
quality to scholarly works." The suggested 'new metrics' that could be
extracted from a fully electronic communication system include the discussion
level generated by a paper submitted to a publication system with open peer
review and peer commentary features; automated citation indexing beyond the
standard Institute for Scientific Information (ISI) print-focused service; and
access statistics.13
LAMPSS:
Lots of Alternative Models Provide Sensible Solutions
[1]
Open Peer Review
During much of its recent history, conventional peer review has
been wholly or partially anonymous. In the former arrangement, neither reviewer
nor the author is known to each other; in the latter, the author is identified.
Until five years ago, the British Medical Journal (BMJ), the
high-impact, general medical journal of the British Medical Association, had
"used a closed system of peer review, where the authors do not know who
has reviewed their papers ... but the reviewers do, however, know the names of
the authors."14 In announcing a change in its editorial policy,
Richard Smith, the BMJ editor further observes that “[m]ost medical journals use the same system,
... based on custom not evidence. Now we plan to let authors know the identity
of reviewers. Soon we are likely to open up the whole system so that anybody
interested can see the whole process on the World Wide Web. The change is based
on evidence and an ethical argument.15,16 He further
notes that "the primary argument against closed peer review is that it
seems wrong for somebody making an important judgment on the work of others to
do so in secret." In a supportive argument, Smith quotes Drummond Rennie,
a deputy editor of JAMA: The Journal of the American Medical Association,
stating that identifying the reviewer links "privilege and duty, by
reminding the reviewer that with power comes responsibility: that the scientist
invested with the mantle of the judge cannot be arbitrary in his or her
judgment and must be a constructive critic."17
Journals that have implemented open peer review include not only
the British Medical Journal (www.bmj.com), but also select journals
published by BioMed Central (www.biomedcentral.com), as well as Internet
Health: Journal of Research, Application, Communication & Ethics,
(www.virtualmed.netfirms.com/internethealth/), among others.
[2]
Commentary-Based
A journal that offers a novel form of 'open review' as well as
conventional peer review is the Electronic Transactions on Artificial
Intelligence (ETAI) (www.etaij.org). "The ETAI is an electronic
journal that uses the Internet medium to the fullest extent for scientific
communication: not merely for distributing the articles, but also for the
review procedure, for bibliographic infrastructure, [and other
applications]."18 With respect to reviewing, “it ... use[s]
a novel, two-stage procedure where the first review phase is open and allows
the peer community to ask questions to the author, and to create a discussion
about the contribution. The second phase - called refereeing in the ETAI - is
like conventional journal refereeing except that the major part of the required
feedback is supposed to have occurred already in the first, review phase.”19
After refereeing,
accepted articles are published in the ETAI journal in both a paper edition and
an electronic edition where the latter is available free of charge, and
contains a variety of other information, including past discussions.20 21
Launched in September 1996, the Journal of Interactive Media in
Education (JIME) (www-jime.open.ac.uk), a borne-digital e-journal has
incorporated one of the most varied forms of peer interaction and review. Its
review environment provides an "opportunity to redesign the conventional
journal review model to be more open, responsive and dynamic" In the JIME
review model "authors have the right of reply", "reviewers are
named and accountable for their comments, and their contribution
acknowledged," and the wider research community has the chance to shape a
submission before [forma] publication."22 Specifically, “[a]rticles submitted to JIME are first reviewed by three
reviewers who are named, and acknowledged for their contribution to a review.
They post their reviews as threaded comments to a private [Web] site. Reviewers
have the option of posting anonymously, but usually ... [they] are happy [to
be] ... named... . [In turn], authors are encouraged to respond to [reviewer]
comments. This [particular part of the process] takes place during an agreed period
when authors and reviewers are able to respond in a timely manner.”23
On the basis of this interaction, if the assigned editor
ascertains that the manuscript is of acceptable quality based on this
discussion, the submission "will be published will then be published as a
preprint for public open peer review, and announced to relevant communities to
invite their participation." This second phase of the JIME open review is
open for a month after formal posting. After public review and comment, the
assigned JIME editor will "post to the discussion an editorial report
summarising the most significant issues, and specifying change requirements to
the authors." After formal acceptance and publication, JIME continues to
support discussion about the revised, published article. In addition, the most
interesting review comments/exchanges are published with the final version of
the revised submission, "providing readers with insight into the issues
that arose during review, and enabling them to build on those discussions."24
Within JIME, authors can “post links to publications to point to
subsequent work. Readers can post comments and links to point to work which has
not been referenced, or did not exist when the article was written. [In
addition], [a]uthors, reviewers and anyone else who has subscribed to the
article will receive email alerts to new postings to its discussion forum.”25
[3]
Community Based
Paul Ginsparg, the creator and developer of arXiv.org, 26
the physics e-print service originally based at the
"At some later point (which could vary from article to article, perhaps with no time limit), a much smaller set of articles would be selected for the full peer review process [and assigned to an upper tier]. The initial selection criteria for this smaller set could be any of a variety of impact measures, to be determined, and based explicitly on their prior widespread and systematic availability and citability: e.g., reader nomination or rating, citation impact, usage statistics, editorial selection, ... ."28
It is expected
that this approach would lead to an overall efficiency in the review process by
expending 'community intellectual resources' on submissions that are "most
likely to be archivally useful, and hence merit the enhanced editorial
treatment for upgrade into the upper tier, including, for example, text
clarifications and other improvements."29 Upper tier
enhancements could also include “anything from a thorough blind refereeing to
open professional annotation and comment. The upper tier could also combine
commentary on many related papers at once. The point is that it's possible to
provide more signal of various sorts to users on a smaller subset of articles,
without worry about fairness issues of limited dissemination for the rest, and
this can be done at lower overall cost than the current system, both in time
spent by editors and in elective time spent by referees.”30
As noted by Ginsparg, "the standard tier would provide a rapid distribution system only marginally less elite than much of the current publication system, and enormously useful to readers and authors," particularly in fields "in which the time to publication time is perceived to be too long." In addition, “the standard tier availability could also be used to collect confidential commentary from interested readers so that eventual referees would have access to a wealth of currently inaccessible information held by the community ... ."31
More than five years ago, David Stern, Director of Science
Libraries and Information Services at
In the Stern model, there are no direct submissions to the Editorial
Board, instead manuscripts would be directed to the Editorial Board in one of
three ways: “1. nominated by the eprint Moderator upon receipt for the archival
server; 2. notification sent to the Editorial Board when a threshold
number of hits are generated by any one manuscript on the archive server; or, 3. nominated by readers of material from the archive; this process
requires a letter of support outlining the importance of the work to the
Editorial Board.”34
[4]
Usage-based
As noted several alternative peer review models recommend a metric
that uses access statistics as an indicator of significance. While the European
Science Foundation policy briefing observes that 'new metrics' could include
the discussion level generated by a paper, as well as access statistics as
well, the Ginsparg model, recommends that second tier review candidates might
be selected on "a variety of impact measures, that would include reader
nomination or rating, or usage statistics." In the Stern model, candidates
for review for an editorial board would be identified "when a threshold
number of hits are generated by any one manuscript on the archive server."
More recently, Harnad, notes that a "new potential measure of on-line
impact, not available in the on-paper era, is usage, in the form of 'hits.'
This measure is noisy ([in that] it can be inflated by automated web-crawlers,
short-changed by intermediate caches, abused by deliberate self-hits from
authors, and undiscriminating between nonspecific site-browsing and item-specific
reading) ..., [but] seems to have some signal-value too, partly correlated with
and partly independent of citation impact ... ."35
[5]
Citation-based
More than fifty years ago Eugene Garfield developed and applied a
method of citation indexing to enhance access to the scholarly scientific
journal literature.36 In October 1999, the Open Citation Project:
Reference Linking and Citation Analysis for Open Archives
(http://opcit.eprints.org/) was officially inaugurated. One of its most notable
outcomes was the creation of Citebase
(http://citebase.eprints.org/cgi-bin/search), a Web-based citation index for
Web scholarship and "Google for the refereed research literature."37
Among its various functionalities, Citebase can provide several of the established scientometric measures of research impact, notably citation counts for a Web-based article, citation counts for the researcher, as well as co-citation analyses. In addition, it has enabled the creation, development, and implementation of new measures of 'impact', notably citation counts for the preprint phase of a publication, usage measures ('hits') for preprints and postprints, time-course analyses, and usage/citation correlators and predictors. Using Citebase, pre- and post-publication citations for individual papers can be measured against usage for the first time.38
More recently in February 2004, the Institute for Scientific
Information (ISI) - the producer of the Web of Science and numerous specialized
print and electronic citation indexes covering a variety of disciplines and
subjects, announced a new initiative to create, test, and provide a 'Web
Citation Index' for Web scholarship. In addition to tracking online
publications, the new index will also document citations to print works by
online sources.39
The new service will utilize the NEC CiteSeer technology
(http://citeseer.ist.psu.edu/cis), which includes "extraction of
bibliographic citations, autonomous citation indexing, calculating citation
statistics and related documents, reference linking to cited articles, citation
context display, automatic notification based on user profiles, correlation of
related documents, full-text indexing, query-sensitive summaries of the context
of search terms in an article, citation graph analysis, and targeted Web
crawling."40
New
Opportunities
"Let us be imaginative in exploring the remarkable
possibilities of this brave new medium."41
As observed by Harnad, "the Net ...[not only] offers the
possibility of implementing peer review more efficiently and equitably
...," but more significantly, provides a "real revolutionary
dimension" with such features as "open peer commentary on published
and ongoing work." In addition, the Net provides “room … for unrefereed discussion
too, [notably] in high-level discussion forums ...."42 Such
enhancements to conventional peer review need not, however, be limited to
features that some may view as simple extensions of the traditional model. In
addition to 'ideal' conversations, metrics such as access statistics, as well
as citing and linking, can also offer impartial indicators of valid and
significant scholarship in all its forms, at any and all stages.
1. Jean-Claude Guédon, In
2. Carin M. Olson, "Peer Review of the Biomedical
Literature," American Journal of Emergency Medicine 8 no. 4 (July
1990): 356.
3.
Ibid.
4. Anne C. Weller, Editorial Peer Review: Its Strengths and
Weaknesses (
5. Richard Smith, "Opening Up BMJ Peer Review," BMJ
318 no. 7175 (January 2 1999): 4-5. Also available at:
http://bmj.bmjjournals.com/cgi/content/full/318/7175/4 (accessed April 4,
2004).
6.
Fytton Rowland, "The Peer-Review Process," Learned Publishing
15 no. 4 (October 2002): 247-258. Report version available at
http://www.jisc.ac.uk/uploaded_documents/rowland.pdf (accessed April 4, 2004).
7. Stevan Harnad, "Implementing Peer Review on the Net:
Scientific Quality Control in Scholarly Electronic Journals,” in Scholarly
Publication: The Electronic Frontier, ed. by Robin P. Peek and Gregory B.
Newby, 112 (Cambridge MA: MIT Press, 1996). Also available at:
http://www.ecs.soton.ac.uk/~harnad/Papers/Harnad/harnad96.peer.review.html
(accessed April 4, 2004).
8. Mike Sosteric, "Interactive Peer Review: A Research
Note," Electronic Journal of Sociology 2 no. 1 (1996),
http://socserv.socsci.mcmaster.ca/EJS/vol002.001/SostericNote.vol002.001.html
(accessed April 4, 2004).
9. Antti Kauppinen, "Habermas Links," http://www.helsinki.fi/~amkauppi/hablinks.html (accessed April 4, 2004).
10.
"Interactive Peer Review."
11.Alan
G. Gross, The Rhetoric of Science (Cambridge, MA: Harvard University
Press, 1990), 137.
12. Raf Dekeyser, Herbert Van de Sompel, and Corrado Pettenati,
"Workshop on the Open Archives Initiative (OAI) and Peer Review
Journals in Europe," http://preprints.cern.ch/OAi/ (accessed April 4,
2004).
13. Alison Buckholtz, Raf Dekeyser, Melissa Hagemann, Thomas Krichel, and Herbert Van de Sompel, "Open Access: Restoring Scientific Communication to Its Rightful Owners," European Science Foundation Policy Briefing 21 (April 2003): 1-8. Also available at: http://www.arl.org/sparc/SPB21_OAI.pdf (accessed April 4, 2004).
14. "Opening Up BMJ Peer Review.”
15.
Ibid.
16.
See also: bmj.com, "Our Peer Review Process,"
http://bmj.bmjjournals.com/advice/peer_review.shtml (accessed April 4, 2004).
17.
"Opening Up BMJ Peer Review."
18. Electronic Transactions on Artificial Intelligence, "General Information about the ETAI," http://www.ida.liu.se/ext/etai/tempvis/welcome/sframe.html (accessed April 4, 2004).
19.
Ibid.
20.
Ibid.
21. See also Electronic Transactions on Artificial Intelligence, "How to Submit," http://www.ida.liu.se/ext/etai/orj-howsubmit.html (accessed April 4, 2004).
22. Journal of Interactive Media in Education, "About
JIME," http://www-jime.open.ac.uk/about.html (accessed April 4, 2004).
23.
Ibid.
24. Ibid.
25.
Ibid.
26. Gerry McKiernan,
"arXiv.org: The
27. Paul Ginsparg, "Can Peer Review be Better Focused?,"
Science & Technology Libraries 22 no. 3/4 (2004): 14. Preprint
available at http://arxiv.org/blurb/pg02pr.html (accessed April 4, 2004).
28.
Ibid.
29.
Ibid.
30.
Ibid.
31.
Ibid., 14-15.
32. David Stern, "The eprint Moderator Model," in Guide
to Information Sources in the Physical Sciences (Englewood: CO: Libraries
Unlimited, Inc., 2000), 168. Also available at
http://www.lib.unc.edu/prices/1999/PRIC214.HTML#214.5.
33.
Ibid.
34.
Ibid., 169.
35. Stevan Harnad, "Scientometric OAI Search Engines, "Posting to LIS-ELib [LIS-ELIB@JISCMAIL.AC.UK], Sun, 25 Aug 2002 15:28:55 +0100, http://www.jiscmail.ac.uk/cgi-bin/ wa.exe?A2=ind0208&L=lis-elib&T=0&F=&S&P =1362 (accessed April 4, 2004).
36.
Eugene Garfield, Citation Indexing: Its Theory and Application in Science,
Technology, and Humanities (New York: John Wiley & Sons, 1979).
37. Steve Hitchcock, Donna Bergmark, Tim Brody, Christopher
Gutteridge, Les Carr, Wendy Hall, Carl Lagoze, Stevan Harnad, "Open
Citation Linking: The Way Forward," D-Lib Magazine 8 no 10 (October
2002) http://www.dlib.org/dlib/october02/hitchcock/10hitchcock.html (accessed
April 4, 2004).
38. Steve Hitchcock, Tim Brody, Christopher Gutteridge, Les Carr
and Stevan Harnad, "The Impact of OAI-based Search on Access to Research
Journal Papers," Serials 16 no. 3 (November 2003): 255-260. Also
available at: http://opcit.eprints.org/serials-short/serials11.html (accessed
April 4, 2004).
39.
Vincent Kiernan, "Company to Track Citations of Online Scholarship," The
Chronicle of Higher Education 50, no. 28, (March 19, 2004): A31.
40. Barbara Quint, "Thomson ISI to Track Web-based
Scholarship with NEC's CiteSeer," Information Today, News Break &
The Weekly News Digest, (March 1, 2004),
http://www.infotoday.com/newsbreaks/nb040301-1.shtml (accessed April 4, 2004).
41.
"Implementing Peer Review on the Net," 115.
42. Ibid., 116.
“Peer Review in the Internet Age: Five (5)
Easy Pieces,” Against the Grain 16, no. 3
(June 2004): 50, 52-55.
| FINAL | 04-23-04 | 11:45| ~3,720 |