Reprint: A Proposal for the Establishment of Review Boards
Skip other details (including permanent urls, DOI, citation information)
This work is protected by copyright and may be linked to without seeking permission. Permission must be received for subsequent distribution in print or electronically. Please contact firstname.lastname@example.org for more information. :
For more information, read Michigan Publishing's access and usage policy.
A version of this article first appeared at http://www.cpm.mmu.ac.uk/cpmrep50.html.
Academic researchers today have many of the same needs as researchers 50 years ago. These include the ability to find and read relevant work done by other academics, being able to disseminate their work to other academics, and to gain recognition for it. However the difficulties facing a researcher in satisfying these needs has changed. Whereas problems of access and cost used to dominate, now there is a problem of being swamped with too much information. It is now very easy to publish and disseminate work, but correspondingly difficult to find the relevant quality information in amongst the rest.
The traditional remedy to this search problem is the peer-reviewed journal. A relatively small number of trusted academics select what they judge to be worthwhile for the rest to read. The system has its limitations. Readers may find that the available journals are edited and reviewed by people whose selection criteria are different to their own, and the articles they may have wanted have been turned down. In the past it was necessary to severely restrict what was published because of the time and cost involved. Now, with the advent of cheap distribution via the Internet, a new trade-off between the various costs and the flexibility of the selection process is possible.
This paper explores one possible selection process, one where mark-up and archiving is separated from review. The idea is that authors publish their papers on their own Web pages or in a public paper archive. In a quite separate process, boards of scholars review selections of these papers and publish collations of their reviews. The collected reviews are stored in such a way as to enable readers to use a Web search engine to select the papers they want to read using these quality judgments and paper content.
Paper journals have significant costs associated with them: the costs of writing the papers; reviewing them; organizing the review process; editing, printing, distributing, archiving, and indexing them; searching for them; and reading them. Some of those tasks are essential to the process and should not be eliminated. However, the advent of desktop computers connected to the Internet means that the cost structure is different for electronic publications.
Web journals eliminate the need for printing (at least by the publisher); significantly reduce the distribution and archiving costs; and facilitate the organization of the review process, the indexing, and the reader's search. What costs there are can be almost totally subsumed into the academic's time and resources (Internet access, printing facilities, word-processing). That means that the only major unsubsumed costs left are the organization of review, and the mark-up of papers. Since a lot of the mark-up can be demanded of the author, given the availability of word-processing facilities, the remaining unsubsumed costs are sufficiently low that Web journals can be set up with no explicit commercial structure at all. Also, the flexible nature of electronic representation of knowledge can be utilized to enable new forms of knowledge presentation and interaction between academics, such as peer commentary.
"It is increasingly impractical to merely "surf" if you are in need of a particular type of information"
However, most Web journals still retain some of the other roles of paper journals, in that these journals still own the papers they publish. They mark them up to ensure they are well presented and to make the style consistent, they archive the papers on their Web site, and they (typically) own the copyright on their papers. This ownership has subtle costs in that it makes the structure of knowledge less flexible — for example, it means that a single paper can be presented to only one audience, the readers of the journal in which it is published. This inflexibility is a disadvantage for readers because they may not be presented with a paper of interest to them in the journals they read.
Another disadvantage of most Web journals is in the closed nature of the review process. There is a private dialogue between the reviewers and editors of the journal and the author. Frequently that dialogue concerns not only questions of fact and presentation, but also the content of the paper, even when the issues concerned are controversial and far from settled. That not only deprives the journal's readers of a role in this discussion but also denies them the information that can be gained from it (e.g. the detail of the reviewers' judgments). The only detail the readers have access to is whether or not the paper was published. As a result, the readers can search journals only on the content of what is published, for they can not search using any of the detailed judgement information created by the review process.
Limitations of Current Systems
The phenomenal growth of academic information available on the Internet has been accompanied by the development of some systems to help readers find and access what they want. However these are, so far, all inadequate in that they do not allow the reader to select on a mixture of quality and content criteria.
The explosion in the amount of information available on the Web means that it is increasingly impractical to merely "surf" if you are in need of a particular type of information. Internet search engines considerably increase your chances of finding information if the information is out there and your search target is quite specific. However, search engines do not allow you to filter results by quality, standard of presentation, or other judgment-based criteria. Thus they are not necessarily helpful if you want information on a popular topic. This is a problem that is unlikely to be solved by purely technical solutions, since the competition for readers in popular areas means that Web authors will constantly adapt to exploit the particular features of Web search engines, whatever they are. (Some search engines such as Google infer something useful about quality using the fact that links are an implicit endorsement of what they link to, but this is necessarily of a generic nature, and might say little about the academic quality.)
Web Review Services:
A number of services have sprung up to rate Web sites. Examples include Encyclopedia Britannica, which reviews and selects what it considers to be quality Web sites. Users of these services can be assured of a certain level of quality when they browse among the selected sites. For this to work, the user must trust the service, a trust that can either be based upon reputation or on previous experience with the service. However, the generic nature of such services means that they will not be able to cover individual papers in sufficient depth to be useful to academics.
Following Paul Ginsparg's ftp archive for physics papers, a number of public academic-paper archives have been set up. Academics are free to upload any papers they have written to those sites, which then archive them. The advantage of such archives is that they are known to search engines (unlike individual home pages) and their consistent storage format makes searching them effective. In addition, such archives tend to be more permanent, and at the same URL, than papers scattered on individuals' home pages. Finally, many of the archives have e-mail alert services that notify readers when new papers in selected categories arrive. While these archives do not guarantee quality (anyone can post, after all) the fact that papers tend to be posted by academics allows users to guess quality based on the author's name, title, and institution. Nevertheless, it is not possible on any of these archives to search by measurements of quality, only by content.
On-line Indexing Services:
Academics have compiled indices to aid other academics for hundreds of years. Such indices are most convenient when they are on-line. A few of these services (especially specialist services like Hippias) include links to on-line papers so that the results of searches can be immediately accessed. Inclusion in such an index does give a substantial, if indirect, guarantee of quality (because they include papers from journals they think are reliable). However the inclusion of papers still gives little indication about the kind and level of their quality (although some are approaching this problem through citation information and statistics).
Review Processes That Use Public Archives:
There are now several peer-review services that are being set up or run in conjunction with public paper archives. Astrophysical Journal Letters encourages authors to place the final accepted version of their papers on the physics pre-print archive (as described in their information for authors). Several of the American Physical Society publications encourage the submission of papers to the archive (see their page on Manuscript Submission for APS Research Journals). Christopher Bauma and Thomas Krichel have made a proposal that would strengthen and extend the RePEC service as part of their application  for funding of their Economics Distributed Electronic Library (EDEL) [formerly http://netec.wustl.edu/AcMeS/] project. 
Such groups seek to add a quality rating to the papers on the public archive. That gives readers more ways to determine whether the papers are worth reading. However, the process saves no money (it depends on the standard journal-review process), nor does it provide the information as an integral part of the archive because papers cannot be searched by their journal-acceptance status. Finally, the added information about the quality of the papers is only whether it was accepted or not. Such information is limited in utility, because it does not let readers choose their own criteria for quality.
Electronic Review Boards
My idea is to set up review boards that will review on-line papers (whether on home pages or in archives). The reviewers would focus on a particular area of knowledge. The reviews would consist of judgments of the papers in a number of different ways. A paper could be given a grade from one to five on each of several aspects, including presentation, relevance, soundness of argument, originality, importance of questions considered, and importance of results. Reviews would also include short comments about the paper. These grades and comments would be submitted electronically and automatically collated into a set of publicly accessible records. These records would include information about the content (title, author, abstract, keywords, location, format, etc.) as well as the compiled qualitative judgments (number or reviewers, average grades on each aspect, collated comments, etc.).
These records would be generally accessible. A specially configured search engine would allow readers to search for information using a mixture of topic-based judgment and content-based search strategies. Two examples of this are:
Readers could choose to look at papers judged to have important conclusions related to particular subjects, and thereby keep up with the important developments in a field.
Readers could look for papers with high originality that mention a particular keyword. Even if these papers are bad in other respects, they might be useful for finding new ideas.
In this way the reader can avoid being swamped by information with irrelevant characteristics and be assured that what is accessed is of relevent quality. The flow of quality information is not restricted in a generic way earlier in the publishing process; instead all the information is made available to readers so that they can place their own constraints on it.
Today, readers' choices are two: to read journals whose content may not be selected according to the readers' precise needs, or to select from the Web or public archive on content and have to wade through papers that do not meet their quality requirements. The traditional system is illustrated below.
My system delays the selection process as long as possible and thus does not discard any information that may turn out to be of use later. It provides the user with the means to exclude the mass of papers that do not meet his or her needs at that time. The proposed system is illustrated below.
Those illustrations are, of course, oversimplified. The great variety of established journals cover different standards and different purposes. Readers can use implicit and explicit knowledge of the quality and coverage of a journal to aid their search process. I presume that if review boards were set up, there would be at least as great a variety of boards as there are of journals. The important difference is that the journal system imposes premature, inflexible, and unnecessary restrictions on the flow of knowledge; the review-board system would not do that.
At the moment the only practical and general way to avoid scanning papers whose qualities are insufficient for a reader's purposes is to accept the selections of a journal whose criteria may not be identical to to the reader's criteria. Of course, if a reader is in the happy position of having access to a manageable set of journals which (collectively) cover all of his or her needs efficiently, then the journal system is sufficient. Many readers are not in such a position.
Scope of the Review Board
There is no reason why a review board should restrict itself to those papers that are submitted to it by authors. It could choose to review any papers that would be relevant to its readers (although if the paper were published in a paper journal it would not be possible to supply an on-line link to it). Thus a review board could not only review new papers for its readers, but also provide a new view of existing papers by re-judging an existing on-line paper for their audience.
A possible review process is illustrated here.
In order to establish the feasibility of this proposal, I outline a possible implementation below. This is for illustration only; it is likely that there are better ways of realizing the above ideas. As is frequently the case, the people issues are more difficult than the technical ones, so I will leave those for later and deal with the technical requirement first.
Each review board would have its own Web site, with the summaries of its reviews listed by the title of the paper reviewed. These summaries would include the paper's title, author, publication date, URL, abstract, keywords, etc., as well as the average score given by the reviewers on each of the different scales, and a collection of the public remarks made by the reviewers. The principle interface to this information for readers would be via a customized search engine at site that allows them to search for papers using a combination of quality and content criteria.
The format of the judgment information that is to be made available to the public will limit the uses to which it can be put. For this reason I suggest that the review information be stored in a publicly available Web database, either on its own site or as part of a public archive, as viewable Web pages in XML format. That way it would be easy to adapt an XML-based search engine such as Harvest [formerly http://www.tardis.ed.ac.uk/harvest/] to index and search the information. In addition, the review Web sites can be browsed by the public and indexed by public search engines just like any other Web site.
The fields in the review Web pages should conform to an established document-description standard (such as ReDIF) that was designed for academic papers. The fields might include author or authors, title, authors' institutions, publication date, keywords, and URL; information on the review board such as title, URL, classification codes, keywords, and e-mail address of the maintainer; and the judgment information, which could be anything the Review Board desires, including average originality rating, number of reviewers, review date, average importance rating, standard of presentation, soundness, and reviewer comments. Some of these, such as the title, can only be searched, but other data, such as level of presentation, are ordinal in nature and should be represented numerically so that the searcher can specify a minimum level (e.g. presentation < 2).
"The biggest barrier to the establishment of a review board is the initial gathering of people to make it happen"
The Santa Fe Convention of the Open Archives Initiative provides a specification for a standardized interface that can be used for accessing on-line papers in a variety of formats (including ReDIF) as well as for formatting the answers to queries to the review database. A widespread adoption of this standard could greatly ease the implementation and management of review boards and enable review boards to integrate into a network of similar academic-information services. This convention is still in development, however, and it is not clear how it will deal with these kinds of reviews, which are essentially documents that refer to other documents.
Input scripts: To ease the creation of the database of judgment information, scripts should be written that allow reviewers to enter their review and translate it into the required format (merging any numerical judgments with any existing ones). Such scripts could also perform some basic checking on the information (like the presence of the paper at the specified URL). The information update could then be released into the review board's database with minimal extra checking required. A security system should allow only accredited reviewers to input reviews and an internal audit trail is needed to determine which reviewers submitted reviews. The facilities for implementing these scripts and associated security measures are widely available.
Search Engine: A search engine will need to be installed (either by the review board or by a third party) to deliver the information in an accessible way to the reader. Ideally this should be customized to allow the easiest access to the judgment data. The reader would then merely follow the suggested links to access the papers they want. Given the public accessibility of the database information, a variety of different interfaces could be provided for the reader. One suitable indexing and search system is Harvest.
Introductory Pages: In addition to the judgment database and the search engine with its interface, there would need to be a few Web pages to provide the context for the service. These would include: a title page for the review board, a page identifying the reviewers and their supporting institutions, a page describing the meaning of the judgment information provided (which would typically involve some description of the process whereby it was derived), and some help pages. The actual pages containing the information should be linked to the title page.
Alert Service: An additional service that would be useful to readers would be a flexible alert service that would send e-mail to subscribers whenever new review information was released that met their selection criteria. Some of the on-line indexing services already provide such services, but I have not seen free software that would do this.
Dienst Server: If the Review Board uses the XML standard, a special Web server such as the Dienst Server, which implements the Open Archives Dienst Subset of XML, will be needed to make the review pages available. That subset of XML is a standard interface for queries about documents archived at a site. In this case each review would be treated as an archived document that comments on a paper. By putting the reviews on such a server, the Review Board would allow developers of other services to exploit and integrate the information generated by the review board for their users.
Given that most of the software is already in existence and merely needs to be put together (with the exception of the additional alert service), the biggest barrier to the establishment of a review board is the initial gathering of people to make it happen.
First and foremost, a review board would need a body of academics to do the reviewing. They would need to be sufficient in number, dedication, and coherence to produce a sufficient volume of quality review information to be useful to readers. It is likely that these reviewers would have to spend some time writing reviews before the service was announced. It is likely (but dependent on the board's policy) that two or three reviews would have to be collected on each paper before the judgment was released.
The job of managing the review process is greatly lessened and far more amenable to partial automation due to its one-shot nature. That is, there is no process of paper rewriting: the paper is judged as it publicly appears. However (dependent on the board's exact policy), there is still a residual job in checking the information before it is released, leading conversations on policy and direction with the reviewers, and occasionally requesting reviews on papers.
Once a board is established, a reader's choice to use the service will be largely based upon his or her experience with it. A review board has the same problem as new journals, namely credibility. I would anticipate that many of the traditional solutions would be used: known and trusted reviewers, prestigious institutional associations, and basic advertising.
Of course, ultimately there could be a Review board of Review boards to help readers choose!
Advantages and Disadvantages of the System
The major advantages of this proposal are as follows:
It gives the readers access to richer review information than merely whether the paper is deemed publishable in a particular journal. This information is already produced by reviewers; it is just kept private.
Since this extra information can be provided in a form that can be utilized by search engines, the reader can set his or her own selection criteria using a mix of judgment and content based information depending on his or her particular needs.
Because ownership of the papers does not pass to the review boards, several review boards could review the same paper. That would encourage boards to seek out papers in other fields that are useful in their discipline.
The fact that the presentation and ownership of papers is not the responsibility of the review board, and the fact that the review process is simplified, means that review boards take less of an academic's time than a journal. Thus a greater variety of review boards will spring up in response to different needs in the academic community (and maybe beyond).
The system adds value to the totality of academic knowledge in a way which is far more flexible than the existing system. For example, once the judgment information is published, software writers can write new search engines to allow new ways of searching and relating this knowledge.
The system is open, in that there is no private journal-author dialogue. Thus there would be less possibility of editors or reviewers coercing an author to change the content of a paper merely to suit their prejudices.
Possible disadvantages of the system could include:
The time saved from the private review dialogue, the automation of the management of the review process, and the mark-up of text to fit a journal's style will be less than the time spent in duplicate review of the same paper by different boards or at different times by the same board. However, I do not think that review boards will be swamped with unnecessary work, for they will quickly develop rules to adjust their workloads. Having said this I do think that in many cases the time spent re-reviewing a paper for a different audience, and hence making it accessible to them, is time valuably spent.
The general standard of papers would drop as a result of the lack of a review dialogue. I do not think that a drop in standards would occur, merely that the mechanism of paper improvement would be different. The fact that a paper would be publicly judged with no chance of redrafting before the judgment is made public will mean that authors will take greater care in their first drafts and get more feedback on the paper from colleagues, from mailing lists, and at workshops. I can envisage review services that offer the author private feedback, but I think these would be provided by academic institutions internally or they would charge for their services. Such services would relieve the burden on the journals and the review boards.
Reliable sources of quality judgment may be lost. Review boards would gain status and permanence in much the same way as journals have. Initially they will be judged on the eminence of the reviewers and their institutions. Later they will also be judged by their output. I think it will take a long time before people switch allegiances from trusted sources of quality information. If anything, the danger is that a transference to new review boards will be slower than is justified by the quality of their output. Also, I would expect the major journals to establish their own review boards under their known 'brand' to keep their readers. Those journals might also choose to sell their reviews.
Academics in some fields will not have the skills or equipment to use the new system. This is true to different extents in different fields. I doubt that academics in computer science would have much trouble with the new system, so review boards could be set up in computer science and other technological disciplines first. The system would then percolate down to other areas over time.
Overall, the ease with which review boards can be established and the flexibility of their nature will mean that the whole system will become more responsive and flexible. Some older, more entrenched institutions will be bypassed. However, in my experience most establishments have the ability to adapt if the need is suitably pressing. One assurance is that the whole system is adaptive and evolutionary: the new will only take over from the old if people vote for it by using the new review services.
The shift in the cost structure of publishing caused by cheap computing power and by the Internet means that a more flexible system of information filtering can now be implemented, allowing readers to set selection criteria that meet their individual needs. This can be done by publishing some of the judgment information that reviewers already produce, in a form that readers can utilize using a search engine. This delayed selection of papers ensures that the system is as flexible as possible in helping readers find the information they want.
The separation of the review process from that of paper ownership and the simplification of the review process from a closed dialogue to a simple one-shot evaluation, enables the semi-automation of the management and hence further reduces costs. This efficiency should enable a large variety of review boards to be set up, segmented not only by subject matter but also by approach and type of information offered to the reader.
The final balance of time saved and wasted using a system of review boards compared to the journal system will only be discovered in practice. However it is indicative that the main work left is that essential to the academic process: writing, selecting and reading.
Bruce Edmonds is a Senior Research Fellow at the Centre for Policy Modelling at Manchester Metropolitan University in the United Kingdom. He took his first degree in mathematics at Oxford in 1983 and his doctorate in the philosophy of science at Manchester University in 1999. His research interests include the philosophy of complexity, modelling methodology for the social sciences, socially situated intelligence, memetics, and context. He is the publisher and an editor of the free-access peer-reviewed Web journal The Journal of Memetics - Evolutionary Models of Information Transmission. You may contact him by e-mail at email@example.com.
Akman, V. and Surav, M. (1996). Steps Toward Formalizing Context, AI Magazine, 17:55-72. http://cogprints.org/464/)
Baum, C. F. and Krichel, T. EDEL: Economics Distributed Electronic Library. An application to the NSF. [formerly http://netec.wustl.edu/AcMeS/edel.letter.pdf and http://netec.mcc.ac.uk/AcMeS/edel.A4.pdf]
Campbell, D. T. (1974). Evolutionary Epistemology. In (P.A. Schlipp, ed.) The Library of Living Philosophers, Vol. XIV: The Philosophy of Karl Popper. Open Court.
Collins, G. and Pryor, L. (1995). Planning under uncertainty: Some key issues. Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI), 1567-1573. [formerly http://home.clara.net/pryor/papers/keys-and-boxes.ps]
David H. Wolpert and William G. Macready (1995). No free lunch theorems for search, Technical Report, Santa Fe Institute, Number SFI-TR-95-02-010. [formerly http://www.santafe.edu/sfi/publications/Working-Papers/95-02-010.ps]
Davies, G. and Thomson, D. (1998). Memory in Context: Context in Memory. Chichester: John Wiley.
Encyclopedia Britannica's Web's best sites. http://www.britannica.com/
Fisher, R. A. (1958). The Genetical Theory of Natural Selection. Dover. Revision of first edition published 1930 by the Oxford University Press.
Ginsparg, P. (1994) First Steps Towards Electronic Research Communication. Computers in Physics, 8:390-396. http://xxx.lanl.gov/blurb/
Harvest. [formerly http://www.tardis.ed.ac.uk/harvest/]
Hempel, C. (1965). Aspects of Scientific Explanation, New York: The Free Press, & London: Collier-Macmillan.
Langdon, W. B. (1998). Genetic Programming and Data Structures. Dordrecht: Kluwer.
Price, G. R. (1970). Selection and covariance. Nature, 227:520-521. [doi: 10.1038/227520a0]
ReDIF documentation. http://www.repec.org/
RePEc (Research Papers in Economics) documentation. http://www.repec.org/
Santa Fe Convention for the Open Archives initiative. http://www.openarchives.org/sfc/sfc_entry.htm
'Uncover Reveal' service. [formerly http://uncweb.carl.org/]