Abstract

Systematic reviews (SR) are comprehensive and reproducible literature reviews that provide evidence for practice and identify gaps in research. This paper describes the work of a group of librarians at Carnegie Mellon University that is collaborating with faculty on a proof-of-concept management SR for the Campbell Collaboration. Discussions of SR training, developing methodical search strategies, and the role of librarians is included. This service is an example of the evolving role of business librarians and an opportunity to collaborate with our clientele in an innovative way.

Keywords: systematic reviews, collaboration

Systematic Reviews: A Brief Overview

Systematic reviews are methodical and comprehensive literature syntheses that are focused on well-formulated research questions. Their purposes are to identify and synthesize all scholarly research on a particular topic in order to provide evidence for practice and policy making or identify gaps in research and reduce bias. Most research attempts to prove a point in one direction or another. This practice leads to potentially biased results that skew evidence. By ingesting and analyzing all evidence on a particular topic, systematic reviews help address this problem and lead to more complete analyses on complex questions and topics. Systematic reviews started primarily in the health and medical sciences but are now slowly gaining traction in the social sciences. There are efforts to make the systematic review methodology more of a norm within several areas of the social sciences, including business and management. The momentum is growing, but there is still work to be done to make this a more accepted practice.

There are several ways in which systematic reviews differ from traditional literature reviews, but two stand out the most. The first is in the generation of the review question or topic. In traditional literature reviews, topics may be broad in scope and the goal may be to place one’s knowledge within an existing body of knowledge. Thus, there is the tendency to gather information, data, and studies that help support a particular viewpoint. In a systematic review, the topic is a well-defined research question that is answered by the review itself. The goal is to find all existing evidence in an unbiased, transparent, and reproducible way. A second difference is in the search strategy. An ad hoc approach to searching for studies based on what the author is already familiar with is usually the search strategy approach in traditional literature reviews. They are rarely exhaustive or fully comprehensive. Systematic reviews ensure that attempts are made to find all published and unpublished literature on the research question and that the process is well documented and reproducible. There are seven major steps in the systematic review process. The first is to identify the research question, a step that can often take a significant amount of time. The second is to define the inclusion and exclusion criteria, followed by searching for studies (a key area where librarians play a significant role), selecting studies for inclusion based on pre-defined criteria, extracting the data from the included studies, evaluating the risk of bias in the included studies, and lastly presenting the results and assessing the quality of evidence on the initial question. These steps can take a significant amount of time to complete. Most systematic reviews can take anywhere from 12 to 18 months to finish. A major reason for this is that systematic reviews are often side projects for faculty, researchers, and librarians that are not supported by grant dollars.

It is highly encouraged that systematic reviews are done by a team that includes content specialists, statisticians, and a librarian or search/information retrieval specialist. Although it is possible for the team to produce systematic reviews on their own, there are two overarching organizations that can help with the management, training, and execution of an SR. They are the Cochrane Organisation (https://us.cochrane.org) and the Campbell Collaboration (https://campbellcollaboration.org). Cochrane, founded by Iain Chalmers in 1993, focuses on developing systematic reviews of the strongest evidence available about healthcare interventions for the health and medical sciences areas. Campbell, founded in 1999, has the same focus for the social sciences. Our Carnegie Mellon University (CMU) SR team chose to work with the Campbell Collaboration for our proof-of-concept management systematic review that focuses on executive compensation, firm financial performance, and inaccurate financial reporting.

Campbell Collaboration: SRs for the Social Sciences and Training

The Campbell Collaboration is the primary organization devoted to systematic reviews and meta-analyses in the social sciences. According to their stated mission, “the Campbell Collaboration promotes positive social and economic change through the production and use of systematic reviews and other evidence synthesis for evidence-based policy and practice” (Campbell Collaboration, 2020a). Campbell is organized into 9 coordinating groups that focus on specific topics/research areas within the social sciences: Crime & Justice, Disability, Education, International Development, Knowledge Translation & Implementation, Methods, Nutrition, Social Welfare, and Business & Management (the newest coordinating group). Finished systematic reviews that are done through the Campbell Collaboration are openly available through the Campbell Collaboration Online Library. Additionally, Campbell and John Wiley and Sons recently announced a publishing partnership whereby Wiley will also publish the openly available systematic reviews in the Campbell Library as an open access journal starting in June of 2019 (Campbell Collaboration, 2019). Our team’s systematic review will be one of the first entries in the new Business & Management group.

Although not required, it is highly recommended to participate in various Campbell training courses on systematic review and meta-analysis methods. There are several training opportunities available through Campbell, including in-person and online trainings (Campbell Collaboration, 2020b). Our team decided to invite trainers from Campbell to come to CMU for a 2-day in-person training that took place in February of 2018. During our training, we learned about principles and methods of research synthesis, common sources of bias and error in research reviews, steps in the systematic review process, steps in conducting meta-analysis, strategies for minimizing bias and error at each step, evidence-based standards for systematic reviews, how to obtain more information and practical evidence on research synthesis, how to identify local and global challenges and partnerships for research synthesis, uses and misuses of research synthesis to inform policy, practice, and further research, and strategies for developing effective and efficient search strategies. This last portion of the training was of utmost importance to the librarians in our group. Here we learned about the best possible literature and database sources for conducting search strategies, the development of complex search strategies, and how to identify and search gray literature sources for unidentified studies that are not represented in the literature and online databases. During the training, there were opportunities for short group activities and quizzes to help ensure key concepts were practiced and learned. Plenty of additional educational materials were provided on all areas covered as well.

Even though training is not required, a protocol is required for conducting a systematic review through the Campbell Collaboration. The steps involved with the protocol were also covered in our training. The protocol is a detailed, long form document that describes every aspect of the systematic review from the research question to the search methods to the inclusion and exclusion criteria to the data extraction method(s) and more. Completed protocols through Campbell are submitted using the Review Manager 5 (RevMan 5) software. RevMan 5 allows for the preparation and maintenance of both the protocol and systematic review. The software provides a centralized place for systematic review team members to collaborate on their documents as well as Campbell reviewers to provide feedback. RevMan 5 software is free of charge to prepare protocols and systematic reviews for purely academic use. If the review is intended for commercial use, a license must be purchased. Once a protocol is completed, it is reviewed by content area specialists, systematic review specialists, and search strategy specialists. After careful review, feedback is provided to the team that either requests modifications to the protocol or gives a green light to proceed with the review. The approved protocol acts as a separate publication for the systematic review team and is also freely available in the Campbell Online Library. Our team is at the protocol completion stage and we are awaiting approval from Campbell to proceed with the actual systematic review.

Our Team’s Proof-of-Concept SR

Our team is focusing on the following question: “Does executive compensation predict publicly traded firms’ financial performance and inaccurate financial reporting?” Our team consists of Denise Rousseau, PhD, joint faculty of the Tepper School of Business and the Heinz College of Information Systems and Public Policy; ByeongJo Kim, PhD student at the Heinz College; Sarah Young, Liaison Librarian for the Heinz College; Donna Beck, STEM Librarian at Duquesne University (formerly Liaison Librarian for Engineering at CMU); and myself. Rousseau is heavily involved with the Center for Evidence-Based Management (https://www.cebma.org), an organization determined to bring evidence-based practices to the management field. Her research specializes in executive compensation within firms. Kim works very closely with Rousseau and is learning evidence-based methods to apply in public policy decisions. Young and Beck both have vast experience with systematic reviews and their methods. Both have served as search strategy experts on many SR teams. I am a novice who is learning these methods. My goal is to spread these methods to business librarianship and the management field for future practice.

As mentioned earlier, our team has completed the protocol for our systematic review and is awaiting approval from Campbell to continue. One of the most important roles that librarians play in the process is the search strategy. Our team met several times to finalize a search strategy for our review. First, we decided where to search for relevant studies to the question we developed. For studies in subscription-based, online databases, we chose to focus on a number of resources devoted to business and management: ABI/INFORM (ProQuest), Business Source Ultimate (EBSCO), Emerald Management Research Collection, EconLit (EBSCO), Web of Science – Core Collection (Clarivate), Scopus (Elsevier), and Dissertations & Theses (ProQuest). The resources for identifying gray literature studies is also quite exhaustive and includes the National Bureau of Economic Research (NBER) Working Papers, Bureau of Economic Analysis, Board of Governors – Federal Reserve, Federal Reserve Economic Data, American Economic Association Papers & Proceedings, Social Sciences Research Network (SSRN), Research Papers in Economics (RePEc), and the Conference Board.

After developing the list of resources to be searched, we then turned to crafting the list of keywords and thesauri terms that will be searched in each database. Based on our question, the keywords and thesauri terms focus on three different subjects and their synonymous meanings: chief executive officer, (firm) financial performance, and financial restatements. Our goal is to generate a long search string of terms around these three areas that will be searched within the title (TI) and abstract (AB) fields of the articles in the respective databases. Once the search string has been crafted it will need to be replicated in the other databases. The search string will change slightly because of the different field codes, terms, and other idiosyncrasies involved with each database. This process is called translation. The search strings for each database are not required for the protocol, but a sample search string that will be used in one of the databases is required. Our team developed a search string for the ABI/INFORM database that we included in the protocol, and it currently has 17 steps to get to a final set of articles. The translation of this search string for the other databases will take place when we get approval to start the systematic review. Gray literature searching strings will not be as thorough because of the restrictions in advanced searching within gray literature sources and databases. For this, we will search for various keywords that align with our developed question in as many fields that are possible within each gray literature source.

The search strategy can be broken down into four parts: developing the search strategy keywords, translation of the search strategy to all databases, documenting the search strategy, and delivering the search results. A recent study in the Journal of the Medical Library Association found that these four parts take an average of twenty-one hours to complete (Bullers et al., 2018). In the end, the overarching goal of the search strategy and the systematic review is to provide an open, transparent, reproducible method of researching and assessing the evidence of a particular research question. Hence, it takes a large amount of time to complete all of these processes.

Role of (Business) Librarians and Benefits

The role of librarians in the systematic review process is pivotal in determining the overall success of the systematic review. Our expertise in the search strategy and information retrieval process yields the relevant studies that will make up the bulk of the content for analysis within the systematic review. Who better to serve in this role than librarians? Additionally, we serve as relevant disciplinary experts for research and resources related to specific areas and fields. This is clearly evident within the landscape of business librarianship. As many of us know, there are dozens of potential business/management literature resources to consult during the systematic review, including EBSCO, ProQuest, Emerald and other traditional database vendors. Our expertise in the gray literature sources (NBER, RePEc, SSRN, etc.) is vital to the success of the systematic review as well. The concept of systematic reviews is still a relatively new one within the field of business/management and within business librarianship. Working to promote these types of studies enables business librarians to increase our value as true collaborators with faculty and PhD students in a new and innovative way. It will also lead to better evidence on a variety of topics and questions within this field.

In working with our team at CMU, I have greatly benefitted from our collaboration in many ways. First, I am becoming a better content expert overall, particularly in the subfield of executive compensation practices. There is a rich literature on this topic that looks at many different types of compensation plans that includes everything from bonuses to golden handshakes (briefly, a clause in an executive’s contract that provides the executive with a significant severance package in the event that the executive loses their job through firing, restructuring, or retirement). Gaining this knowledge about the practices and the culture that forms around executive compensation packages at large companies enhances my understanding of core business principles that I can apply to my liaison work within the business school. By obtaining knowledge like this, I will be seen as a better collaborator with faculty and PhD students in this space because I will be better able to “speak their language” and understand complex business practices that I never knew before.

Second, the SR process has made me a better librarian. One of the core skills librarians possess is advanced searching strategies in a variety of different discipline-specific resources and databases. A key strategy to use when searching these resources is to rely on the built-in thesaurus when crafting search strings. Because of the work I have done with this team, I have strengthened my advanced searching skills in all business/management-specific literature databases. Additionally, due to the nature of changing terms in thesauri from resource to resource, I have enhanced my knowledge of how Business Source Ultimate, ABI/INFORM, and other business databases are structured, which will lead to better search strategies and results in future consultations and projects.

A third benefit is the potential for scholarly publication credit. Many librarians that work on systematic review teams do get authorship credit for the finalized systematic review. Although there is no guarantee for this in general, it is a point that librarians can negotiate for since our contributions are vital to the process in more ways than one. Many of us have designated faculty status and are expected to produce and partake in the scholarship process. Systematic reviews are an ideal way for us to fulfil that criteria while also building long-lasting collaborations with our constituents and colleagues.

As mentioned before, one of the ultimate goals of systematic reviews is reproducibility. The entire process is documented and spelled out so that others in the field can replicate every step. This plays very well into the ethos of librarianship, where we value transparency and openness to research and processes that produce scholarship. It is also a way for librarians to practice open access principles during the publication phase. By working with the Campbell Collaboration, SR teams will produce protocols and systematic reviews that are open to the public at large for consumption through their Online Library or with John Wiley and Sons. Additionally, copies of both can be housed in our respective institutions’ institutional repositories without any worry about author’s accepted manuscript versions or forced embargos by journal publishers. In the end, this is a win-win situation for open, collaborative scholarly research.

Hopefully, the systematic review movement will continue to gain momentum across all disciplines, including business and management. Librarians are key collaborators in this process as we continue to try to find new, innovative, and creative ways to connect with our researchers. This is a perfect opportunity to do so that benefits everyone in the end.

References