Abstract

Peer reviewing is mandatory for scientific journals as quality control of submitted manuscripts, for universities to rank applicants for scientific positions, and for funding agencies to rank grant applications. In spite of this deep dependency of peer reviewing throughout the entire academic realm, universities exhibit a peculiar lack of interest in this activity. The aim of this article is to show that by taking an active interest in peer reviewing the universities will take control over the management and policy shaping of scientific publishing, a regime that is presently largely controlled by the big publishing houses. The benefits of gaining control of scientific publishing policy include the possibility to implement open access publishing and to reduce the unjustifiably high subscription rates currently charged by some of the major publishing houses. A common international clean-up action is needed to move this pivotal element of scientific publishing from the dark hiding places of the scientific journals to where it should be managed: namely, at the universities. In addition to the economic benefits, we postulate that placing peer reviewing at the universities will improve the quality of published research.

Keywords: Peer reviewing, scholarly publishing, publishing houses, funding agencies, grant applications, universities, open access

1. Introduction

1.1. Major stakeholders

Science and research have enabled humankind over the last few centuries to develop ideas and solutions that have greatly improved welfare. Governments of most countries appreciate this and allocate considerable resources to enable continued scientific research. On this basis it is clearly of great importance to understand the ecosystem of scientific research: who are the key players and stakeholders, and what are the key processes involved in organizing and driving scientific research?

Four major players and stakeholders can be identified: (1) the scientific journals, or publishing houses; (2) governments, research councils and other public and private funding bodies; (3) universities and other R&D institutions; and (4) the researchers. It may be argued that the researcher should be viewed as part of the university and not as a distinct stakeholder; however, a meaningful analysis of this issue requires identification of the researcher as one of the major stakeholders. To discuss the role of peer reviewing in scientific publishing it is pertinent to first clarify the roles of these four stakeholders, and the ways that they are interconnected.

Table 1. Key players in scientific research and publishing: their tasks, interests, and motives. They all depend on peer reviewing.
Scientific journals/publishing houses Government, national, and international research councils Universities and similar R&D institutions Researchers
Receive manuscripts from researchers Fuel universities (positions, infrastructure, research education) Take care of research education: production of PhD candidates Generate ideas, design projects, and carry out research
Coordinate peer reviewing of submitted manuscripts Fuel research councils that establish research programs, allocate resources to these programs, and organize ways of distributing resources to universities, research centers, and researchers. This is done by competition based on the outcome of peer reviewing, which depends to a large extent on prior journal-coordinated peer reviewing, resulting in acceptance of applicants’ publications. Organize research, including (1) employing researchers and, (2) awarding local research grants, both activities to a large extent depending on prior journal-coordinated peer reviewing resulting in acceptance of applicants’ publications Produce publications (write manuscripts, carry out peer reviewing of peers’ manuscripts, supervise PhD candidates)
Finalize the publishing in print and/or online Write grant applications and carry out reviewing of peers’ grant applications

1.1.1. Scientific Journals

The scientific journals need to be clearly identified and understood as players in the ecosystem of scientific research and publication. The utmost importance of journal-coordinated peer reviewing is appreciated by realizing that this level of peer reviewing has a major impact also on the processes underlying the ranking of grant applications and applicants of research positions. Thus, journal-coordinated peer reviewing plays the pivotal role of “primary peer reviewing”, whereas peer reviewing making use of journal-coordinated peer reviewing (used in evaluation of grant applications and applicants of faculty/research positions) may be referred to as “secondary peer reviewing”. These relationships, along with the fact that the scientific journals, notably those published by the large commercial publishing houses, represent the only commercial actor among the major stakeholders in scientific publishing and research, justify a detailed analysis of the role of the scientific journals not only in scientific publishing, but also in how they impact the policy and management of academic research as a whole.

1.1.2. Governments

Governments decide the amounts of money to be allocated to the R&D sector and represent by far the largest source of research funding. Thus, public money is the major financial source to fund research. Research councils establish programs to make sure that the money is distributed to cover prioritized sectors and topics. These topics are decided both by governmental political guidelines, and through processes within the research councils. The research councils announce their programs and invite researchers to apply for research funding, and researchers write grant applications. The process of grant application is normally a highly competitive undertaking. In addition to a requirement that grant applications must describe research activities that fit within programs predefined by the research council, the evaluation of submitted applications is based on a meticulous peer review system determining the research quality proper. Peer reviewers read the applications and rank them according to overall quality. Through this peer reviewing process it is thought that the money allocated from the government will be used in the best possible way. It should be noted that the publication record enclosed by the grant applicant plays a major role in evaluating the scientific credibility and quality of the applicant. Thus, peer reviewing of research grant applications represents a “secondary reviewing” that is based to a large extent on the “primary peer reviewing” coordinated by scientific journals.

1.1.3. Universities

Universities are increasingly urging their researchers to apply for external grants to finance their research activities and at the same time contributing to the general economy at the university. The competition for these funds is very high, thus ensuring that only the best projects are funded. This process is thought to maintain research production at a high level. The term research production in this context means both amount and quality of research. A simple way to measure the amount of research produced is to count the number of candidates, PhD theses, and publications produced per year. Hence, staff at any administrative level can readily assess the amount of research produced. However, research quality can only be evaluated by scientific experts, i.e., peer reviewers. Universities frequently appoint prioritized researchers or research groups, which receive strategic grants. The process underlying such decisions are commonly based on rate and quality of research publication, based largely on journal-coordinated peer reviewing. Universities also use peer reviewing when employing scientific staff. Similar to the peer reviewing of grant applications, the reviewing of research job applications represents a “secondary” type of peer reviewing, for which the “primary,” or journal-coordinated peer reviewing, is already in place in the form of publication lists.

1.1.4. Scientific researchers

Scientific researchers, who publish and depend on external grants are, as a rule, university employees. In addition to taking care of administrative and teaching duties, they publish in scientific journals and submit grant applications. Both of these activities are normally extremely competitive undertakings. It is common knowledge that the researchers are evaluated not purely based on their published research, but also based on where they publish. In applications for tenure and for funding, the CV of the applicant needs to include publications in prestigious journals to stand a chance. There is thus a stiff competition between researchers to have their manuscripts accepted for publishing in the highest possible ranking journals. Thus, the researchers are heavily dependent on a fair and reliable peer review system to have their research production evaluated. Again, this may be divided between the primary peer reviews to get published in the “right” journals, and next the secondary peer reviews when applying for positions and grants. Their success in these latter reviews are to a large extent dependent on how their publication list appears.

By analyzing the role of these four major stakeholders it appears that peer reviewing is a common denominator of major importance at most stages of the money flow and production of research (Fig 1):

Figure 1.: Interconnection between peer review and the major stakeholders of scholarly publishing. Peer review is a major common denominator.
Figure 1.
Interconnection between peer review and the major stakeholders of scholarly publishing. Peer review is a major common denominator.

Research councils depend on peer reviewing to manage a fair and acceptable distribution of public money allocated by the government. The universities and their researchers in turn depend on peer reviewing to facilitate publication. And last but not least, the scientific journals depend on peer reviewing as an instrument to verify the quality of the research and decide on acceptance of submitted manuscripts. In this analysis it is important to understand that the journal-coordinated peer reviewing serves as the basic and primary peer reviewing processes underlying both allocation of research grants and employment of scientific staff. It is noteworthy that the journal-coordinated peer reviewing is beyond the control of the universities and research councils. This is one of the great paradoxes of scientific publishing.

2. The cost of peer reviewing

In the eyes and budgets of scholarly journals peer reviewing is the least expensive part of the publishing process. The statement that “The cost of peer review is only a small proportion of the total budget of the journal” [1] is undoubtedly correct. So is the calculation by The British Medical Journal [2] that the direct cost of reviewing an article amounts to approx. £100. Of note, neither of these estimates includes the time spent by the scientists doing the reviewing. In reality peer reviewing is a rather costly affair. Peer reviewing is performed by university-employed experts who spent many years of education and scientific work to be eligible to take on this task. Accordingly, instead of articulating that the journals pay close to nothing for peer reviewing, it would give a more honest and realistic understanding to communicate the total cost of peer reviewing. In a survey most referees reported 7–10 hours spent to review a manuscript. [3] The Research Information Network (RIN) estimated the unpaid non-cash costs of peer review undertaken by academics at £1.9bn globally each year. These academics are as a rule university employees, and, accordingly, the time spent on peer reviewing ought to be in the interest of the universities. Taxpayers fund the salaries of the scientists, whose research and publication costs are funded by grants based largely on public money. The same scientists rely on the university libraries to pay publishers so they can access their own work. The following quotation appropriately sums up this relationship: “In essence, public funds pay twice for the work: to do it and then for access to it . . . The latter cost is being paid not to public servants or universities, but to companies.” [4] In reality taxpayers pay not only twice, but three times for the work, since peer review, mainly carried out by university-employed scientists, is a pivotal part of the process. It is noteworthy that peer reviewing represents the very foundation in this peculiar alliance of academia and commercial publishing houses. Hence, it is rather odd that when the journals uphold the low cost of peer reviewing, the universities, who clearly spend big money for this central activity, do not pay attention to this at all. A study estimating the time spent on peer reviewing for scholarly journals by researchers at UiT The Arctic University of Norway indicates that 15,000 to 20,000 hours are used annually on peer reviewing at this university alone, with its 1251 researchers [5] at the time when this study was carried out (2010). Extrapolating this figure to universities worldwide would indicate that university salaries are used to provide scholarly journals with a colossal number of work-hours. This labor (as well as infrastructure) used for peer review is given away for free to the scholarly journals, in the same way as labor and infrastructure used to write journal articles are provided without charge. This arrangement might have been an agreed deal yielding mutual benefit, if the journals were all run by the scholarly community at large (societies and universities). However that is, as we all know, not the situation. Lortie [4] compared this situation with running a relay race, where the scientists run every step of a long race, and the publishers receive the baton just meters from the finish line.

The universities’ ignorance of peer reviewing is not limited to their giving away for free the work-hours used by their employees to the journals: A recent informal survey carried out by us to establish to what extent universities in different countries take an interest in peer reviewing revealed that most of the universities included in the survey cared very little—if at all. One of the questions asked to representatives of major universities in Europe, the United States, and Japan was: “Does your university regard reviewing as an important activity of its scientific staff, by incentivizing or controlling this activity in any way (similar to how teaching and research are incentivized)?” Most of our survey participants replied that their university does not have any policy whatsoever regarding the issues mentioned in the question. A few of the respondents reported that their university instructs their employees to limit their time used for peer reviewing to a certain maximum, to make sure that peer reviewing activities do not stand in the way of other tasks that are expected from scientific personnel. One of the respondents replied that his university recently ordered its scientific employees to refuse any peer reviewing tasks for scholarly journals (!). This directive was later cancelled due to massive protests by the scientific staff. All in all the answers clearly indicated that none of the universities included in the survey showed any active interest in ensuring that peer reviewing carried out by their employees follow certain standards; neither did any of the universities identify peer reviewing as an activity worthy of being incentivized or mentioned as an obligatory work task.

3. Advantages gained for universities and research by identifying peer reviewing as an activity that should be coordinated and owned by the universities

3.1. Increased power to negotiate subscription price and implementation of true open access

Scholarly journals cannot do without peer reviewing. Any scientific journal that was to abandon peer reviewing would soon lose its credibility and perish. This crucial activity is supplied for free mainly by professors and researchers at higher education and research institutions. Hence, peer reviewing may be viewed as in-kind payment from the scholarly community to the journals and publishing houses. We suggest that the scholarly community, in negotiations with the publishers, includes this as part of what they pay for access to the services of the publishers, and thus also as a bargaining chip. To prepare for this, each institution should obtain information on the extent, and for which journals (and publishers), their employees do peer reviewing. Today, to our knowledge, this information is generally lacking in the universities’ administration. When the universities realize that peer reviewing service should be considered as part of their payment to the publishers, the universities will be in a much better position to negotiate down the unjustifiably high subscription prices demanded by some of the big publishing houses. Further, article processing charges (APCs) for open access publishing may also be subject to negotiations. Or, instead of negotiating APCs, a bulk payment for unlimited access to publish open access in the publisher’s journals may be an issue to negotiate. We have seen examples of such negotiations lately—more on this below (section 4.1). And in section 4.2, we describe an alternative strategy, where we apply the bargaining chip of the peer review labor.

3.2 Increased power to improve the standards and performance of peer reviewing

More than 28,000 scholarly peer-reviewed journals published almost 2 million articles in 2012. [10] With the growing number of scholarly publications, it could be argued that it is simply not possible to offer all submitted manuscripts a reliable and objective review process. On this background, it appears justified to question the soundness of a regime where journals have the sole responsibility to coordinate and oversee the peer reviewing. Would it not be more logical to leave this academically important activity to a university-based system? The university, but not the journals, may decide on matters such as: (1) Should all scientific staff do their share of peer reviewing?; (2) Should the quality of peer reviewing be reviewed, in much the same way as research itself is reviewed?; (3) Should peer reviewing be regarded as being just as important as doing research and writing papers?; and (4) Should peer reviewing be implemented as part of the curriculum? Although peer reviewing is presently the most reliable process available to ensure that publications meet certain quality requirements of the science, it is also generally agreed that it is full of flaws. [6], [7], [8], [9] Many studies on peer reviewing reliability have been undertaken by journals. It is understandable that the journals themselves wish to perform such studies, since they are so dependent on the peer review service. However, it would appear more beneficial to academic research if studies of peer reviewing be carried out by the scholarly community. If the universities take an interest in and responsibility for peer reviewing it would be possible to implement international good standards of peer reviewing, agreed upon by some coordinating university organization.

Universities and research councils are increasingly focusing on ethics and fraud in research. These issues are included in compulsory courses given to all research students, and it is expected that research is conducted with high quality and ethics. So why is it that the universities take no responsibility in overseeing ethics and fraud in peer reviewing, a cornerstone in the process of scholarly publishing? This question is highly relevant since recent research (conducted by several individual journals) have disclosed both unethical behavior and fraud associated with peer reviewing. [6], [7], [8], [9]

By organizing the peer review job in accordance with internationally agreed upon standards, instead of a multitude of single scientific journals or in any of the recently established web-based initiatives offering payment or other credits for peer review activities, it would appear that some of the known flaws of the peer review regime could be remedied, which would result in improved quality of the published research.

4. In which direction should the scholarly community go?

Scholarly publishing used to be an activity run by the scholarly community, where societies ran their journals. In a regime like that, it is not a problem that peer review is carried out as a cooperative voluntary activity within the scholarly community, where each contributor commits as much effort and time as he or she can spare. However, scholarly publishing has increasingly become an undertaking of commercial enterprises. Therefore, the scholarly community should act accordingly, and treat any flow of money and labor from the scholarly institutions to the publishers as payment for services supplied by business enterprises. And any kind of payments should be subject to negotiations: What are acceptable payments, and which services do the university want to purchase?

There is an increasing agreement among research policy makers and funders that publishing should be done with an open access model (e.g., [11],[12]). This would be beneficial for the scholarly community, as well as for society at large. The challenge is how to move from the subscription or licensing for access (toll access) model to an appropriate open access model. As we have shown, the commercial publishers are powerful stakeholders, and many of them benefit heavily from the toll access fees and also open access publishing fees. It is fully understandable that these publishers will make all possible efforts to maintain this regime that is so profitable for them. How can the scholarly society take back the power to shape publishing policy from the publishers? Or how can the universities get a more fair price for the services supplied by the publishers?

4.1 Big deal negotiations

One strategy is for the universities to insist that open access publishing be incorporated at an affordable price when big deal agreements with the publishers are negotiated.

These agreements are commonly reached with strict nondisclosure clauses, especially with respect to the agreed price, making them impossible to analyze from the outside. But critique has been raised that these agreements will leave publishers in the same position as before, draining loads of money from the universities. The prestigious journals and the powerful publishers will still be able to take very high tolls for access, and also for releasing publications to become openly accessible.

The universities have basically one negotiating power in the big deal bargaining. They may threaten to say no, refusing to sign the deal. So the publishers’ best strategy will be to offer the highest possible price that still makes the universities agree to the deal.

At the time of writing (January 2017), the German consortium “Alliance of Science Organizations in Germany” has turned down the big deal offer from publisher Elsevier, and as of January 1, 2017, the researchers within this consortium no longer have access to the journals published by Elsevier. This is thus rather dramatic for the researchers affected, and will obviously obstruct their research.

Other interesting examples of big deal agreements are found in the Netherlands. The Association of Universities in the Netherlands (VSNU) has entered into agreements with publishers that give both access to the journals’ content, and also freedom to publish open access in the same journals. Such agreements were reached with publishers Springer (before they merged with Nature) and Wiley. [13] And VSNU also reached an agreement with publisher Elsevier [14], which included, at a limited scale, the option for Dutch researchers to be able to publish open access in their journals.

As part of their negotiation strategy VSNU threatened publisher Elsevier by opening up the possibility that Dutch editors and reviewers would stop doing their editorial and reviewer jobs for Elsevier, and ultimately for Dutch researchers to stop publishing in Elsevier journals. [15] So the Dutch negotiating strategy included more elements than merely the option of saying no to the deal. This strategy was meant to force Elsevier into a deal that would yield 100% open access publishing from Dutch researchers in Elsevier journals, and at an affordable price. In December 2015, VSNU and Elsevier announced that an agreement was reached, but without the full move to open access publishing. This compromise deal indicates that their strategy was rather complex and difficult to pursue. A major reason that VSNU did not reach the goal of 100% open access publishing in the Elsevier journals, was, we believe, that full compliance to their boycott scheme was unachievable as long as editor tasks and peer reviewing in reality is a private affair between the researchers and the journals.

4.2 Alternative negotiating strategy

The strategy we suggest is much simpler than the Dutch approach. We propose to include the supply of labor used for peer reviewing in negotiations between the universities and the publishing houses. How can this be achieved? Today researchers choose freely which journals and publishers to peer review for. Our suggestion is that the universities should threaten to withdraw the free supply of peer review labor from those publishers who are unwilling to enter into agreement at a reasonable price. The withdrawal of the free labor will thus be a way to hold back part of the payments.

This strategy will effectively give the universities the power to favor the less costly publishers and the journals run by scholarly societies, which are normally much less costly than the journals run by commercial publishers.

With this negotiation strategy the toll access journals may still be available to the institution's researchers. And there is no need for the universities to tell their scientists to stop publishing in any journal. All that is needed is for the institutions to use their right as employer to threaten to instruct their researchers not to do peer review with publishers they find too unwilling to enter a fair deal. In this way the researchers’ choice of where to publish, which is at the core of academic freedom, will not be undermined through the negotiation between universities and publishers. And the researchers may continue to access their journals as before, as long as it is only the in-kind payment of peer reviewing that is withheld.

It may be claimed that choosing where to do peer review is an important part of academic freedom. Admittedly, through peer review tasks researchers get access to peers’ manuscripts, which can be a potentially important way to “read news” about the research front. Nevertheless our proposed strategy will not take away the freedom for the researchers to decide how and where to publish their articles.

A journal’s status as a trustworthy publisher of science will be in jeopardy if access to expert peer reviewers is limited. Thus, establishing the threat of withdrawing the peer review effort will be a forceful bargaining power, even if the act of withdrawal will seldom be put into effect. We therefore maintain that the disadvantage for the individual scholar will be rather small. And the risk of scientists not complying should be much less than in the Dutch strategy and with similar strategies where scientists ultimately would be required to stop submitting manuscripts. It should also be noted that the current publishing market, dominated by for-profit publishing houses, where scholarly articles are barred by heavy payments for access, and likewise heavy payments for open access publishing, is a real and serious threat to academic freedom, and a fight worth fighting.

Some may argue that it is the right of each individual scientist to decide on the extent and for what journal to perform peer reviewing. However, if an employer issues directives on the amount of time used by employees to do peer reviewing, this does not interfere with the academic freedom to do research and to choose freely where and how to publish. After all, work contracts include instructions on how to perform a certain amount of teaching, administration, and research. The option of directing where to do or not to do peer review should not be very controversial.

By taking control of, and organizing, peer reviewing universities would obtain a means to regain the academic freedom that was lost when commercial enterprises took over the society driven journals, introducing heavy payment bars to get access to papers and publishing open access.

4.3 Coordinated international action

The directorate-general for research and innovation (RTD—EU) recently launched a new policy initiative: the establishment of an Open Science Policy Platform. The communiqué on this initiative states that “The Policy Platform will provide advice to the Commission on open science policy in Europe as well as on the coordination of those policies with stakeholders. The implementation of open science policy will be effectuated by the active advocacy of agreed policies by these stakeholders.” [16] It is noteworthy that “universities” are listed among the stakeholders. This underscores our point that by taking an interest in and control of the peer review task the universities will greatly facilitate development toward an open science regime.

Further, the EU has recently announced their goal of making all European scientific articles freely accessible by 2020. This announcement was made unanimously by the EU ministers responsible for research and innovation. We welcome this announcement from EU. The ministers have not announced what means to use in achieving their announced goal. We suggest that a united approach in taking control of the peer review job could be an interesting road to follow. Such a unified international action among universities and grant agencies would be very beneficial in order to make the changes needed to establish peer reviewing as a truly academically based responsibility.

Cooperation between universities and their coordinated actions are not something new. Universities have since long joined forces in consortia, not the least for negotiation purposes. Each consortium is free to choose its strategies. But there is nothing preventing the consortia from talking together and exchanging experiences, in forums like the European University Association, and discussing how the EU goal may be reached. Thus, a coordinated international approach that moves the universities toward taking control of the peer review should be achievable.

The increasing international agreements and actions to implement open access publishing are indications that necessary changes in scholarly publishing are possible. By standing together universities will be able to break the economic grip that the big commercial publishing houses have on academic research.

5. Conclusions

Journal-coordinated peer reviewing, a hallmark of scholarly publishing, is also a pivotal part of other central academic processes, such as evaluation of research grant applications and ranking of applicants for faculty/research positions. Hence, journal-coordinated peer reviewing may be viewed as “the mother of academic peer reviewing.” With this background it is astonishing that universities and other public R&D institutions take only a very limited interest in the management and policy shaping of this cornerstone of scholarly publishing. The advantages for the universities to awaken and gain an awareness about peer reviewing are manifold: (1) negotiating power that may lead to easier and quicker implementation of open access publishing and/or (2) reducing costs, in particular the unjustifiably high subscription and licensing rates set by the big commercial publishing houses; (3) better control of how scientific staff use their time for the good of the university; and (4) managing a unified policy shaping of peer reviewing, reducing fraud and flaws. This will in turn increase the quality of the research produced by the universities.


Author Contributions: Bård Smedsrød and Leif Longva conceived the ideas and wrote the paper. Eirik Reierth and Lars Moksness contributed to the writing, created the figure in the article, and searched the literature. All authors met on a regular basis to discuss and agree on the design and set-up of the paper.

Conflicts of Interest: The authors declare no conflict of interest.

Abbreviation: VSNU, The Association of Universities in the Netherlands

References

    1. Fletcher, R.H.; Fletcher, S.W. The effectiveness of journal peer review. In: Jefferson T, Godlee F. Eds. Peer review in health sciences. BMJ Publishing Group, London. 2003. p. 62-75 return to text

    2. Smith, R. Classical peer review: An empty gun. Breast cancer research. 2010, 12 Suppl 4, S13.return to text

    3. Grod, O.N.; Budden, A.E.; Tregenza, T.; Koricheva, J.; Leimu, R.; Aarssen, L.W.; Lortie, C.J. Systematic variation in reviewer practice according to country and gender in the field of ecology and evolution. PloS one. 2008, 3, e3202.return to text

    4. Lortie, C.J. Money for nothing and your referees for free. Ideas in Ecology and Evolution. 2011, 4, 43–47.return to textreturn to text

    5. Refsdal, M. Peer review at the university of Tromsø: A study of time spent on reviewing and researchers' opinions on peer review. Master thesis. 2010.return to text

    6. Bohannon, J. Who's afraid of peer review? Science (New York, N.Y.). 2013, 342, 60–65.return to textreturn to text

    7. Barbash, F. Academia's seamier side: Lying, cheating and fraud. Published in: The Washington Post - Morning Mix, 09.06.2014. Last time accessed: 11.03.2016.return to textreturn to text

    8. Bouter, L.M. Commentary: Perverse incentives or rotten apples? Accountability in research. 2015, 22, 148–161.return to textreturn to text

    9. Oransky, I. Publisher discovers 50 manuscripts involving fake peer reviewers. retractionwatch.com. 25.11.2014. http://retractionwatch.com/2014/11/25/publisher-discovers-50-papers-accepted-based-on-fake-peer-reviews/. Last time accessed: 11.03.17.return to textreturn to text

    10. Manchikanti, L.; Kaye, A.D.; Boswell, M.V.; Hirsch, J.A. Medical journal peer review: Process and bias. Pain physician.2015, 18, E1–E14.return to text

    11. 194/39 commission recommendation of 17 July 2012 on access and preservation of scientific information (2012/417/ue), http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32012H0417. Official Journal of the European Union, 2012. Last time accessed: 11.03.2017.return to text

    12. NIH public access policy details. https://publicaccess.Nih.Gov/policy.Htm, 2009. Last time accessed: 11.03.2017.return to text

    13. Combined open access and subscription agreement between wiley and dutch universities. http://www.vsnu.nl/en_GB/news-items/nieuwsbericht/243-combined-open-access-and-subscription-agreement-between-wiley-and-dutch-universities.html, 2016. Last time accessed: 11.03.2017.return to text

    14. Dutch universities and Elsevier reach agreement in principle on open access and subscription, Elsevier press release. https://www.Elsevier.Com/about/press-releases/corporate/dutch-universities-and-elsevier-reach-agreement-in-principle-on-open-access-and-subscription, 2015 December 10th. Last time accessed: 11.03.2017.return to text

    15. Wijkhuijs, J. Dutch universities start their Elsevier boycott plan. https://universonline.nl/2015/07/02/dutch-universities-start-their-elsevier-boycott-plan, 2015. Last time accessed: 11.03.2017.return to text

    16. (RTD), D.-G.f.r.a.i. New policy initiative: The establishment of an open science policy platform: https://ec.europa.eu/research/swafs/pdf/pub_open_science/new_policy_initiative.pdf, 2015. Last time accessed: 11.03.2017.return to text