Pediatrics is the official journal of the American Academy of Pediatrics. Published continuously since 1948, the journal has achieved a circulation of 62,500, with broad additional distribution via international translations, proprietary computer systems, and online services. Pediatrics has an impact factor of 3.487 (1999), the highest ranked clinical journal (as opposed to research journal) in the specialty. In 1996, the editors were facing a growing backlog of quality articles, longer times to publication, and prohibitive and increasing print publication costs. They needed a viable publishing option that avoided the high variable costs and capacity constraints of print, and the Web beckoned.

In January 1997, an online-only section of peer-reviewed articles debuted as part of Pediatrics. This online-only section, named Pediatrics electronic pages, allows the editors to publish ten to fourteen additional articles every month and make these articles freely available on the Internet (access to the print contents is limited to subscribers or pay per-article users).[1]Abstracts of the online-only articles are included in the print edition on special green pages. Each article passes the same peer-review process as all other original articles in the journal. Selection of which articles appear in the online-only section is done by the editor, with preference given to articles of broader international interest.

Initially, the online extension was viewed skeptically by authors and other editors, but initial resistance soon disappeared as authors gained confidence in this approach to publication. Also, tenure committees and other groups apparently looked approvingly upon online publishing, providing additional reassurances to authors on academic tracks. Like the print articles, the online-only articles are indexed in MEDLINE and PubMed.

Since July 1998, both the print contents and the online-only contents have coexisted online. The print contents require a subscription or payment to access them; the online-only contents remain freely available, as they have been since inception.

In January 2000, after three years of experience with this publishing model, the publisher began an assessment of the articles published in Pediatrics electronic pages. Primarily, we were interested in assessing whether these articles had been cited; if so, how heavily compared to their print counterparts; and how authors perceived the publication of these online-only articles

Methodology

We studied articles published in 1997, 1998, and 1999. Three major measurements were used for both print and online articles: Web usage statistics, citations within the biomedical literature, and author perceptions. One of us was responsible for tallying the citations from the literature, another collected online usage statistics and analyzed these, and the other interviewed eleven authors (selected to represent a mix of topics, years, and citation levels) for their perceptions.

Once all the raw data were collected and the initial analyses were completed, the primary author conducted further analyses in consultation with others on the project.

We also established a priori criteria for successful online publishing: the article should be read, cited, used in tenure deliberations, and used on author curricula vitae . Articles meeting these criteria would, in our eyes, be considered successful without regard to medium.

Findings

In brief, online-only articles met our criteria for successful publication: the content was read, cited, used successfully in tenure deliberations, and used on author curricula vitae.

Online-only articles published in Pediatrics were cited in a manner similar to citation patterns for print articles (see Figure 1). It is interesting to note that for both print and online usage, the patterns are similar to those identified in larger studies of citation patterns.[2] The difference between the mean citation levels for print and online was 3.09 ±0.93 in favor of print (95% CI), meaning that an online article could expect to receive 2.16 to 4.02 fewer citations in the literature than if it had been printed. The peak number of citations for online-only articles was thirty-eight cites, compared to the peak for print citations of fifty-eight citations.

Interviews revealed that authors use online-only articles with their tenure committees and include the articles on their curriculum vitae, but do both with some doubts about their perceived relative strength, likely due to the novelty of the publishing modality and its lower barriers to entry. Finally, and most significantly, authors currently perceive online-only publication as second-tier publishing, subordinate to print.

Citations. In addition to an aggregation of all three years (1997-1999), we divided the citations into two batches: The two older years (1997-1998) and the most recent year (1999). The first subset (1997-1998) yielded the most solid citation data, having had more time in the literature to accrue citations. The difference in mean citation levels for 1997-1998 articles was 3.09 ± 0.89 (95% CI). Considering 1999 separately allowed us to estimate whether online articles are cited faster than print articles. We did not find any difference. For 1999 articles, the difference in mean citation levels was 0.256 ± 0.209 (95% CI) between print and online.

Of the print articles published over all years, 7.25% had received no citations as of January 1, 2000, compared to 23.22% of online-only articles. As shown in Figure 1, 163 print articles had three or fewer citations compared to 187 online articles.

Figure 1: Frequency of Citation Counts by Medium

Online Usage. We next compared the online usage of the articles from 1997-1999 (inclusive). Online-only articles were accessed via the Web at nearly four times the rate of print articles (62.2 downloads per article on average vs. 17.8 for print articles online), and print-friendly portable document format files (PDFs) were downloaded at nearly seven times the rate of print (16.0 downloads per article vs. 2.5 for print articles online). Online usage of online-only articles was consistently above that of print articles, never being exceeded in any given month (see Figure 2), and these differences were found to be statistically significant (ANOVA, P < 0.001, 95% CI).

Figure 2: HTML Accesses By Article Type, 1999

These findings suggest that readers are willing to cross over from the print medium to the online for valuable content and that possibly the free online-only articles are reaching a larger non-subscribing audience for the journal. Our anecdotal evidence also suggests that readers are using the PDFs as substitutes for printed pages to facilitate reading. The lack of fees for online-only articles may well account for a good deal of the difference in access levels.

We also compared decay curves or the curves generated by usage rates from initial publication (highest usage) throughout the life of the article online (the curve resulting as usage falls). There was no apparent difference (see Figure 3), meaning that both print and online usage trended downward at roughly equivalent rates and stabilized in a similar manner.

Figure 3: Online Usage Decay Curve, Online and Print Articles, By Months From Publication

These data are exactly what one would expect. Because online-only articles are available only online and access is free, readers would use them online much more. However, the exact magnitude of this difference was the unknown. With HTML usage being roughly double that of the print articles, but PDF usage seven times greater, it is tempting to infer that readers are still behaving in a relatively traditional way in using articles — i.e., printing them out and reading them (and/or storing them) off line and using the online journal to locate elusive content, then returning to their print stacks.

Author Survey. A survey was sent to the 266 authors of online-only articles published in 1997-1998 which yielded a response rate of 39.1% (104/266). Questions asked authors for their perceptions of reprint requests, email contacts, citation levels compared to print, reference to the work in curricula vitae and before tenure committees, acceptance of the work when cited before tenure committees, overall online publication experience, and preference of publication in print vs. online. A scale of one to ten was used to rate each question, with one equating to a strong preference for online publication and a ten equating to a strong preference for print publication.

Modified Leikert Scale (1=Online Better, 10=Print Better)

The results revealed that authors still have an overall preference for print publication with the weighted-average scores yielding a pooled average of 7.01. The question that directly addressed a preference for print generated a weighted-average score of 8.13. No score in the survey provided a weighted-average of less than 5.0 which would have indicated some preference for online-only publication.

Figure 4: Trends Among Responses from 104 Authors of Online-Only Articles

In general, the data, when graphed, showed two spikes in Figure 4, one around the score of 6/10 and a strong upswing toward the score of 10 (strong print preference). This may indicate a bifurcation among authors and their perceptions of online-only publication and possibly a trend in either direction (toward greater acceptance of online-only publication, or toward greater resistance to online-only publication). Surveys in the future will need to assess whether there is a trend and, if so, which direction it is moving.

Of the respondents to the survey, 16.5% (44/266) indicated that they had applied for tenure since publication of their online-only article. Of these forty-four respondents, 100% (44/44) indicated that their online-only articles had been accepted by their institution's tenure committee.

Also of interest was the fact that 100% (104/104) authors included their online-only publication on their curricula vitae, despite some strongly stated views that online-only publication is "second-tier" to print.

In an aside, one respondent to the survey indicated that a study published by his group (Dietrich AJ, Olson AL, Sox CH, et al. A community-based randomized trial: encouraging sun protection for children. Pediatrics 1998;102(6);e64) was selected as "best paper" of 1998-1999 by the Society of Teachers of Family Medicine on May 6, 2000, an award given annually to the most important paper published by a member. In summary, the strongest trends found in the survey included a strong preference for print publication over online publication and an expected trend for fewer reprint requests for online articles. We will keep our eyes on subsequent activity to see whether online-only publication becomes even more accepted or whether resistance to it stiffens.

Author Interviews. One of us, trained as a communications professional and journalist, contacted authors of online-only articles from various years and with various citation profiles (some with highly cited articles, some with non-cited articles) to discern the authors' perceptions of online publication. A total of eleven lead authors were contacted and interviewed.

Interviews centered on the authors' perceptions of online-only publication, experiences related to the online publication of their particular article compared to any print publication experiences, use of the publication in tenure considerations and/or on their CV, and media contact surrounding publication.

In general, the authors contacted felt strongly that online-only publication is perceived as second-tier publication. As one author said, "There is a misperception that the electronic pages [online-only section of Pediatrics] is a second-prize." Another called publication in an online-only section "disheartening." Yet other authors felt that the results were so positive that they requested (a year after their original article was published online) that another study they had been involved with be published online. And another author noted that "everyone is living at their computers these days," so online publishing was natural and proper. Negative perceptions were as common among highly cited authors as among non-cited authors, but positive perceptions were not.

Other general trends noted from these interviews included: reprint requests were lower than expected (likely due to the availability of PDFs online); authors listed these publications on their curriculum vitae and used them in tenure proceedings, where they were accepted; media coverage seemed greater; e-mail inquiries were noticeably more frequent; and authors felt the scholarly publishing community was the early stages of online publication and our work at Pediatrics was a useful experiment.

Authors were also asked if they would publish an online-only article again. Of the eleven contacted, two (18%) said "No," three (27%) said "Maybe," and six (55%) said "Yes." However, all of the "yes" respondents stated they would prefer print publication to online-only.

The Case of the 38 Citations

The author interviews revealed an interesting case study, which we dubbed, "The Case of the 38 Citations."

One article, published when the online-only section of Pediatrics was only six months old (Hassink SG, et al. Placental leptin: an important new growth factor in intrauterine and neonatal development? Pediatrics 1997;100:e1), garnered the highest number of citations of any online-only articles (thirty-eight cites). This total gave it the fourth most citations of any article from Pediatrics from 1997 through 1999.

When the lead author of this study was contacted for her perceptions of online publishing, her feedback was jolting. Her perception (and that of her colleagues, she said) was that publication in the online-only section "the worst mistake [we] ever made" and she felt that "nobody referenced the study." She also noted that a study with similar findings was published some months later and received many more citations and much more media coverage. Online publication was blamed as the cause of this relative obscurity.

We identified this second study (Masuzaki H, et al. Nonadipose tissue production of leptin: leptin as a novel placenta-derived hormone in humans. Nature Med 1997;3(9):1029-33) and compared article citations, journal impact factors, and circulations. We found that while the article in Pediatrics (3.466 impact factor, 1998) had been cited thirty-eight times, the article from Nature Medicine (27.9 impact factor, 1998) had been cited 136 times, most often in Science, Nature, and Nature Medicine. In essence, this relatively minor article within Nature Medicine (strong, but certainly not their largest-impact study by far), garnered more citations than the highest-cited print article in Pediatrics during the same time frame.

Our conclusion is that while online publication was blamed for the perceived relative obscurity of the article in Pediatrics, other factors were likely more important. First, an article in a journal with an impact factor eight times greater than Pediatrics will be more likely to have heavier patterns of citation. Second, Pediatrics is not part of a larger publishing constellation of the magnitude of the Macmillan journals or of the Science/Nature spectrum. Having a constellation like this may help to drive impact factors,[3] and two of four of Nature Medicine's top citing journals are part of their publishing family (Nature, Nature Medicine.) Finally, while online-only publication may have limited the citation level, there are no data to support this inference.

When the author of the study from Pediatrics was shown the data, it was clear that, had we been able to supply this data to the authors on a periodic basis, author perceptions would have been different. However, the amount of work need to assemble such comparative data (and knowing what to compare), and in finding a way to inform authors of citation and usage levels in a convenient, sustainable manner makes this option difficult. By keeping authors in the dark about the use of their online-only publications, publishers may be subverting their own attempts to successfully leverage online publication.

Conclusions

Our conclusion for online-only journal publishing ventures is mixed. The studies summarized in this paper seem to indicate that online-only peer-reviewed content is cited in the literature at a fairly normal rate (range, 2.1-4.1 more citations on average for print articles), and that users access, download, and use the content in a sustained manner. However, authors remain reticent, and many are not yet enthusiastic participants in a strong movement toward a networked journal-publishing environment. This may be an effect of a lack of information about the success of their online publications.

Online-only publication offers many advantages relative to cost for publishers (significant savings, especially in variable costs, for each article published in this manner),[4] and potentially increased speed to publication (some journals are now publishing online within hours of submission. [5] However, there are currently no other publishers experimenting with online-only peer-reviewed sections of this magnitude, and authors remain concerned about the viability of the medium and perceptions of online publishing among their peers.

As the only biomedical journal with more than three years of experience with a hybrid print/online publishing model, we need to assess the rate of citation of online-only articles across multiple years to see if acceptance is growing, declining, or remaining stable. We need to evaluate whether author perceptions of the Internet as a venue of primary publication changes year-to-year. And we need to see if increases in Web usage overall will drive citations and author perceptions in certain directions.

Perceptions of online-only publication are a major concern. If authors perceive online-only publication as solidly second-tier, a publisher's ability to achieve faster, cheaper publication of important materials will be restrained. A troubling and continuing trend is that when online-publishing initiatives are discussed in meetings or among large groups of authors, often superficial commonalties and convenient business platitudes about "dot-coms" are emphasized instead of crucial distinctions and meaningful differences. (This is true of nearly all Web initiatives.) The particular strengths of the Pediatrics approach — a peer-review process identical to that of its print parent, free access to online-only articles, prominent display of article titles on the journal's cover, and a host of other details — can be glossed over when authors or other publishers consider taking advantage of the Web in a similar manner. Tincture of time, abetted by perseverance by innovators, is likely the solution to this problem.

Those who are publishing (or considering publishing) articles online only should pay careful attention to their authors' needs and to their attitudes toward electronic publication — especially their views of electronic publication's validity and rankings. That means countering misperceptions with facts, as we did with the author in the case study. Authors are the key to acceptance of online-only publishing endeavors and will serve as its most effective advocates — or its greatest detractors.

Our survey confirmed that tenure committees, at least in the pediatric field, apparently accept online-only publication for tenure decisions; 100% of those surveyed told us so. However, it must be noted that the relative paucity of online-only publication makes blending the sole or infrequent online publication with many other print publications easier, so online publications can be hidden in the publications list. The paucity of online publications in any single academic's publications list also may provide tenure committees room to argue that the online-only publication is an eccentricity if the candidate is otherwise acceptable.

For Pediatrics, where only 21% of the readership is actively engaged in research (the remaining 79% consists of practice-based pediatricians, academicians, and others), the issue of whether citation impact factor is a valid measure of the utility and relevance of the information is uncertain. In fact, the journal is meant to be a practitioner's journal rather than a researcher's journal which naturally limits citation rates. However, Pediatrics is significantly larger in circulation than other pediatric journals, and 70% of the readership reads nearly all issues published within one week of receipt. Certainly, these measures of usage — circulation, speed to access — are of no less importance than citation rates. In fact, as online statistics begin to interweave with traditional measures of utility such as impact factors and citation rates, new and possibly more relevant measures of a journal's value should emerge. Garfield has documented the limitations of citations as a validation of the literature,[6],[7] and these findings suggest that similar, if not new and expanded, limitations to citations as a measure of the utility of a literature will be encountered online.

Finally, emphasizing studies with broader international interest may explain some of the difference between print and online citation rates, as the Science Citation Index does not include a number of smaller clinical journals and is generally regarded as favoring U.S. journals. However, this is only speculation, and only a completely randomized study of highly comparable articles would answer whether international bias alone produces a significant difference in citation patterns for online articles.

We plan to continue online-only publication in Pediatrics. It allows us to publish 700-800 more pages per year at less than 20% of the variable cost of publishing in print. Our worldwide impact is greater, and readership has increased. However, we will also begin working more closely with authors to ensure that they realize that online-only publication is valid, citations occur at a healthy rate that is comparable to the rate for print articles, and media and reader attention may be greater. We also plan to repeat these measurements on an annual basis to gauge the rate of citation in both print and online media, whether one is accelerating, and how authors' perceptions may be changing as the networked journal becomes more and more integrated into biomedical publishing efforts.

Kent R. Anderson is Publishing Director, The New England Journal of Medicine. Prior to joining the NEJM, he was Director, Medical Journals & Professional Periodicals, at the American Academy of Pediatrics. Kent has worked in biomedical publishing for 13 years, as an editor, writer, designer, managing editor, and publisher. He has launched numerous publications, both in print and online, and developed a number of successful online continuing medical education sites. His education includes a BA in English and an MBA. You may reach him by e-mail at kentanderson@charter.net.

Over a 25-year career in information systems, John has been responsible for architecting and leading the delivery of campus-wide and nation-wide information systems and systems infrastructure. In the 1980s this included development of systems infrastructure such as database management and electronic forms and signature systems that brought transaction processing to nearly every desktop in Stanford University. In the early 1990s the work focused on planning the move from proprietary and locally developed systems to open, marketplace solutions. With the founding of Stanford University's HighWire Press in 1995, John shifted his focus back to information-delivery and publishing technologies that can be thoroughly rethought in light of new distributed technologies. HighWire's goal is to change the economic model of scholarly communication, especially for scientific, technological, and medical information and, in doing so, strengthen the scholarly societies who foster communication. John is expert at the introduction of new technologies into large organizations, and at managing the organizational-change cycles related to such technology introductions. You may reach him by e-mail at sack@stanford.edu.

Lisa Krauss began her career at Stanford University in 1980 and over the past twenty years has performed a variety of administrative tasks. In 1995, with the founding of Stanford University's HighWire Press, her focus changed to include more research, statistical and database work. Today she is the new-business coordinator for HighWire press. She very much enjoys being part of a team on the cutting edge of scientific Web publishing. You may reach her by e-mail at lkrauss@sulmail.stanford.edu.

Lori O'Keefe is an independent public relations consultant and journalist. Previously, Lori was a public relations manager at the American Academy of Pediatrics, and from 1990 to 1996, she was employed at public relations firms in Chicago and St. Louis. She earned her B.S. in journalism from Northern Illinois University. You may contact her at jlmokeefe@aol.com.

Footnotes

1. Anderson K and Lucey J. Pediatrics electronic pages: Looking back and looking ahead. Pediatrics 1998;102(1):124-135. [doi: 10.1542/peds.102.1.124]

2. Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314:497.

3. Meenen NM. The impact factor — a reliable sciento-metric parameter? [Abstract] Unfallchirurgie 1997;23:128-134. [doi: 10.1007/BF02630217]

4. Anderson K. From print to electron: How an STM journal can survive the transition to the Internet. J Am Med Informatics Assoc 2000;15:234-245.

5. Anonymous. JBC Papers in Press. Journal of Biological Chemistry. Web site: http://www.jbc.org/pips/index.dtl.

6. Garfield E. Long-term vs. short-term journal impact: Does it matter? The Scientist 1998;12:10.

7. Garfield E. Which medical journals have the greatest impact? Ann Intern Med 1986;105:313-320.

Links from this article

A community-based randomized trial: encouraging sun protection for children http://www.pediatrics.org/cgi/content/full/102/6/e64

MEDLINE http://www.nlm.nih.gov/databases/freemedl.html

Pediatrics http://www.pediatrics.org/

Pediatrics electronic pageshttp://www.pediatrics.org/current.shtml#ELECTRONIC_ARTICLE

Placental leptin: an important new growth factor in intrauterine and neonatal development? http://www.pediatrics.org/cgi/content/full/100/1/e1

Pubmed http://www.ncbi.nlm.nih.gov/entrez/query.fcgi

Society of Teachers of Family Medicine http://stfm.org/

Society of Teachers of Family Medicine "best paper" of 1998-1999 http://stfm.org/awards/abstract.html