Benchmarking Librarian Support of Systematic Reviews in the Sciences, Humanities, and Social Sciences

Mê-Linh Lê, Christine J. Neilson, and Janice Winkler*

Systematic reviews, along with other types of knowledge synthesis, are a type of research methodology that attempt to find all available evidence on a topic to help answer specific questions. Librarian involvement in systematic reviews is well established in the health sciences, and in recent years there has been growing awareness of, and literature about, librarians outside of health supporting systematic reviews. This study benchmarks librarian support of systematic reviews in the sciences, humanities, and social sciences (SHSS) by looking at the growth of demand for support, the disciplines requesting this kind of librarian support, and the specific types of support needed. It also examines what SHSS librarians need to be successful in this type of work, including administrative support and workload adjustments.

Background

Knowledge synthesis is a collection of secondary research methods that use the systematic collection, evaluation, and integration of previous research to answer a research question. Over 40 types of reviews which go by a variety of names (sometimes used interchangeably by researchers) fall under the knowledge synthesis category, including systematic reviews, scoping reviews, integrative reviews, and meta-analyses.1 For simplicity, we use the term systematic review (SR) in this article to refer to all types of knowledge synthesis. Different areas of research adopted SR methods at different rates. We often hear anecdotally that SRs originated in the health sciences and have since spread to other disciplines, but this is not accurate. SRs took hold and spread widely in health along with the Evidence-Based Practice Movement in the mid-1990s, but the social science disciplines of education, psychology, and business and economics have continuously used the SR methodology since the mid to late 1970’s, albeit without the same fervor seen in the health disciplines.2

Library literature has discussed library support for SRs since at least the mid 1990s.3 Health librarians found roles on SR research teams, likely due to well-established methodological guidelines from the Cochrane Collaboration and the Institute of Medicine, that advise reviewers to consult with an experienced health sciences librarian to ensure a high-quality literature search for the project.4 As secondary research, the literature search that identifies the existing knowledge base forms the foundation for a SR’s data collection. Research has demonstrated that health librarian participation on SR teams is associated with better quality search strategies, lower risk of bias, and better reporting of search methods.5 Health sciences librarians increasingly contribute to SRs in roles that extend beyond searching, including protocol development, source selection, and teaching.6 The number of SRs published by health researchers is high and continues to grow. Hoffman and colleagues used the PubMed database to estimate that 80 SRs alone—that is, not including other types of knowledge synthesis reviews like scoping reviews, rapid reviews, etc.,—were published per day in 2019. According to their analysis, this publication rate is 20 times greater than it was 20 years earlier, in 2000.7 Unsurprisingly, the ever-increasing popularity of SRs, as well as the demand for health librarian involvement, have led to concerns over librarian workload and the capacity to provide SR support along with other library services.8

While SRs and librarians’ role in that research have long occupied a large amount of the health librarianship discourse, SR research—and library support for it—within the humanities, social sciences, and sciences has only started to gain more attention in the library literature relatively recently.9 Outside of health, the reported involvement of librarians in published SRs in the social sciences and sciences is low. Premji found that two percent of SRs on business topics published from 2014 to 2019 mentioned consulting a librarian, and only one percent credited the librarian with running the search itself.10 Similarly, Slebodnik et al. found that 3.3 percent of the science and social science SRs they examined reported that a librarian was consulted for the review.11 Given this evidence, one might assume that even though SRs are being conducted in these fields, librarians are typically not involved in the process. However, we know that librarians regularly work on SRs without receiving credit for their contribution in a resulting publication.12 Indeed, Kogut and colleagues reported in 2020 that the number of librarian consults at their institution for SRs in education increased over a period of six years, from fewer than 20 consults per year to more than 100 per year. This increase in demand threatened to exceed the library’s capacity to provide support services and required training additional education librarians to provide SR support to maintain the service.13

We know that a variety of disciplines use SRs, but existing library literature has not provided an overview of librarian involvement in SRs outside of health. This article starts to fill this gap by benchmarking Association of Research Libraries (ARL) and Canadian Association of Research Libraries (CARL) SHSS librarians’ involvement in SRs. We collected the data presented here as part of a larger survey regarding SHSS librarian involvement in SRs, their comfort and competence with SR processes, and their perception of library administrators’ level of support for SHSS librarian participation in SRs.

Methods

Survey Design

We created an open survey using SurveyMonkey software. The survey included 29 potential questions in total, over 14 “screens” or pages; 23 questions were closed-ended and six open-ended. The survey used conditional logic so that respondents were only asked applicable questions. Responses to all questions were optional. Participants could return to previous questions using the back button on their browser to revisit their responses, if desired. The survey began with background questions about participants’ subject responsibilities and years of experience. Next, it asked respondents details about involvement or requests for involvement with SRs. The next set of questions asked about supports available to respondents in their provision of SR support. The final set of questions related to training involvement and preferences, which we will explore in a forthcoming C&RL article. We piloted the survey with five colleagues, using their feedback to refine questions for clarity. The University of Manitoba Research Ethics Board approved the final questions and study design (JFREB J2020:062). We translated the recruitment materials and survey instrument into French to obtain responses from librarians who speak both official Canadian languages.

Study Population and Recruitment

Academic librarians working at ARL and CARL institutions were invited to participate in the survey in a convenience sample using 22 listservs maintained by Canadian and American library associations. Due to restrictions from our research ethics board, we could not contact libraries or librarians directly and were only permitted to use listservs. We obtained consent through a form at the beginning of the survey. We offered no incentives in exchange for completing the survey.

To be included in the study, respondents had to be librarians at an ARL or CARL institution, who currently, or within the last 5 years, provided direct library services and support to faculty, staff, or students within the SHSS disciplines. The survey did not provide definitions for which subject areas fell under these categories because programs can be interdisciplinary and their points of focus can vary, leaving this open to interpretation. Instead, it provided a definition of health sciences to allow respondents to determine whether their subjects fell under the category of health sciences and allowed them to use their judgment as to the categorization of their liaison areas. The provided definition was as follows:

For the purposes of this study, the Health Sciences is deemed to include programs or disciplines where health or health care is the primary focus and includes: Allied Health, Dentistry, Dental Hygiene, Nursing, Medicine (including Public Health), Pharmacy, and Rehabilitation Sciences (Physical Therapy, Occupational Therapy, Rehabilitation Therapy). We urge those librarians working with other disciplines that may include strong health components (e.g., kinesiology, psychology) to fill out this survey.

We excluded responses from librarians serving the health sciences, except in cases where respondents supported SHSS disciplines as well as the health sciences.

Librarians who identified themselves as not meeting these inclusion criteria were rejected prior to beginning the survey. The survey was open for seven weeks, with an initial recruitment email sent out in November 2020 and a reminder email sent three weeks later in December 2020. Supplementary materials, including the survey instrument, a list of listservs the survey was sent to, and anonymized data, are available via Open Science Framework (OSF) at https://osf.io/mqxf2.

Data Analysis

We downloaded all responses from SurveyMonkey to a private group in Microsoft Teams, which our institution approved as a secure location for research data. We discarded responses that did not meet the inclusion criteria.

Original data files are stored in password-protected files, accessible only to the authors. We anonymized responses prior to data sharing via OSF in the following manner. We categorized reported liaison areas into four broad disciplinary areas for analysis: Humanities, Sciences, Social Sciences, and Health Sciences (see the OSF page (https://osf.io/mqxf2) for the categorization scheme). As there is no definitive list of liaison areas assigned to a specific discipline, our standard for those liaison areas (e.g., kinesiology) that could be part of different disciplines depending on the institution was to categorize based on where they sat within our home institution. We separated references to specific employers from the data set that we analyzed, and we completely removed them from the shared dataset to maintain participant anonymity. Open-ended responses also presented the possibility of including identifiable information, so we removed them from the shared dataset. The statistical consultant who conducted data analysis signed the required oath of confidentiality, as per research ethics board requirements.

We conducted descriptive analysis for each close-ended question included in the survey. Because of the conditional logic used in the survey and the optional nature of the questions, percentages reported below are based on the number of responses received for individual questions, rather than the total number of individual respondents. We coded responses provided in the free-text questions into broad themes that indicated common issues brought up by multiple respondents.

Because we were interested in relationships between specific variables, we created a list of these variables and hired a statistical consultant to complete more sophisticated statistical analysis. Depending on the type of data gathered and quantity of responses, the consultant determined whether and how to best complete the analyses. In most cases, this involved cross-tabulations (shown in Table 1). Spearman’s correlation coefficient could be calculated to measure the correlation between variables in two cases: the correlation between librarians’ years of experience and their confidence in all aspects of SR support; and their years of experience and the number of SRs they had supported in the last 5 years.

Results

The survey received 379 responses. After filtering for the inclusion criteria, the total number of usable responses was 161. Not all respondents answered every question, so the true number of responses is indicated for each individual question below. Anonymized study data is freely available online at OSF.

Due to our recruitment strategy of library listservs, calculating a response rate was not possible, however, respondents were asked to provide the name of their CARL/ARL institution. There were 98 total responses for this question, but multiple respondents came from the same institution. In total, 42 out of 108 US-based ARL institutions (38.8 percent) and 20 out of 31 CARL institutions (64.5 percent) are represented. Of the 42 ARL institutions represented, 38 (90.4 percent) are R1 institutions, three (seven percent) are R2 institutions, and one (two percent) is an M1 institution. Of the CARL institutions, 13 (87 percent) are U15 institutions.14

Liaison Area

As outlined above, we categorized liaison areas into four broad disciplinary categories: Humanities, Sciences, Social Sciences, and Health Sciences. Individual respondents may support more than one discipline, faculty, or department. As a result, a total of 521 liaison support areas were indicated; 17 percent were classified as the Humanities; 41 percent were classified as the Social Sciences; 32 percent belonged to the Sciences; and ten percent of liaison areas were classified as the Health Sciences (see Figure 1). As noted in the inclusion criteria above, all included respondents who had liaison responsibilities in the Health Sciences also had liaison responsibilities in non-health disciplines. Of the 151 respondents, 84 (52 percent) had liaison responsibilities within a single category; 58 (36 percent) had liaison responsibilities across two categories; and nine (5.5 percent) had liaison responsibilities across three categories.

Total Years as a Librarian

To analyze based on years of experience and other factors, respondents were asked how many years they had worked as a librarian. Of the 157 responses, eight (five percent) answered zero to one years; 25 (15.9 percent) answered two to five years; 35 (22.3 percent) answered six to ten years; 20 (12.7 percent) answered 11 to 14 years; and 69 (43.9 percent) answered 15+ years. Spearman’s correlation coefficient analyses were done to look for potential relationships between years of experience as a librarian, and either confidence in SR support or the number of SRs completed in the last five years, but no correlations were found.

Systematic Review Requests from The Past 5 Years

Respondents were asked whether they had been asked by a faculty member, researcher, or student to participate in a systematic review in the past 5 years. Of the 149 responses received for this question, 104 (70 percent) indicated yes; 38 (26 percent) said no, and seven (five percent) were unsure. When asked how many systematic reviews they had supported during the last five years, out of 139 respondents, 34 (25 percent) had not supported any SRs; 37 (27 percent) had supported one to four; 32 (23 percent) supported five to nine; and 26 (25.9 percent) had supported ten or more (see figure 2 below). Statistical analysis between the number of SRs performed in the last five years and years of experience as a librarian, using Spearman’s rank correlation, found no significant correlation between the two variables. Our analysis also did not find a difference in the number of reviews completed based on whether respondents were given time away from regular duties to complete reviews. For example, 29.7 percent of those who were not given time away from regular duties to work on SRs completed five to seven SRs in the last five years, compared to 34.5 percent of those who were given time away from their regular duties. When asked if the frequency of SR support requests has changed over the past five years, just over half of respondents (56.6 percent, n=81) indicated that they had experienced a change, 31.5 percent of respondents reported no change, and 11.9 percent were unsure if there had been a change in the frequency of requests. Of the 81 respondents who reported a change, one reported a decrease in the number of requests, while the remaining 80 reported an increase in the number of requests received. This translates to 55.9 percent of respondents indicating an increase in the frequency of requests overall.

Most respondents (93, or 67.9 percent) received requests for SR support from only one of the broad disciplinary areas. Close to one third of respondents (44, or 32.1 percent) indicated that they received requests from a combination of two or three disciplinary areas. Over the past five years, Social Science and Health Science disciplines were the main source of requests for SR support, making up 39.3 percent and 37.7 percent of total requests respectively (see figure 3). Respondents received 65 requests (18 percent) from Science disciplines, and six requests (1.7 percent) from Humanities disciplines.

Types of Systematic Review Support

Respondents were asked to indicate the type of SR support they had provided, with the ability to select as many support types as applicable. As Table 2 shows, the areas with greater number of responses were those that are typically seen as an area of expertise for librarians, such as search development, search execution, and search translation. Consultation, which was not explicitly defined, had the highest number of responses.

Acknowledgement

Respondents were asked to indicate the types of acknowledgement they had received for their support of SRs; respondents could select as many answers as were applicable. A thank you via email/in-person/phone had the most responses (70; 50.7 percent); followed by: co-authorship of a paper or presentation (55; 39.3 percent); no acknowledgement (42; 30 percent); mention of working with a librarian in a paper or presentation (29; 30 percent); mention of working with a librarian by name in a paper or presentation (29; 20.7 percent); mention of working with a librarian by name in the formal acknowledgement of a paper or presentation (26; 18.6 percent); co-investigator/collaborator/investigator status on a grant application (seven; five percent).

Administration Communication, Administration Attitude, and Librarian Workload

Most respondents discussed participation in SR projects with their library administrators or otherwise received communication from administration on the topic (100; 68 percent). Of those who had discussed SR participation with their administration, 80 (54.4 percent) said administration was supportive of participation; 34 (23.1 percent) believed their administration to be neutral on participation; 5 (3.4 percent) perceived their administration as discouraging of participation; 21 (14.3 percent) were unsure of administration attitude; and seven (four percent) marked not applicable.

Statistical analysis shows that administrative attitudes toward SRs (discouraging, neutral, or supportive) did not appear to correlate with the actual numbers of reviews the respondents completed over the last five years. Of those who had had discussions with administration, 27.1 percent of respondents had supported five to nine SRs; 11.5 percent supported ten to 14 SRs; and 13.5 percent supported 20 or more SRs. The majority of respondents (52.6 percent) with no discussion with their administration did not support any systematic reviews.

The survey included two questions related to workload adjustments. It asked respondents whether they were or would be given time away from regular duties, and/or granted a reduction in their existing workload to accommodate the work involved in participating in SRs. A small number indicated that they were, or would be, granted protected time (29 or 19.9 percent) and/or a reduction in workload (nine or 6.1 percent). The majority indicated that they would not receive time away from regular duties (77 or 52.7 percent) and/or reduction in workload (98 or 66.7 percent) to participate in SRs. The remaining respondents were unsure if these accommodations were possible (24 or 16.4 percent for time away from regular duties; 22 or 15 percent for reduction in workload) or responded with “not applicable.”

Interest in Supporting Systematic Reviews

When asked to rank their interest in participating in SRs on a 100 point scale, the largest number of respondents indicated a high level of interest (see figure 4 below). Forty-three (28 percent) respondents indicated an interest level ranging from 91 to 100; 24 (16 percent) participants entered a ranking in the 81 to 90 range; and 25 (16 percent) entered a ranking in the 71 to 80 range. Thirty-six participants (23 percent) indicated interest in the middle range (41 to 70), and 26 participants (17 percent) registered their interest in the low range (zero to 40).

Statistical analysis indicated that respondents’ interest in working on SRs was not related to administration attitude towards support of SRs, whether that attitude is perceived as discouraging, neutral, supportive, or if the respondent was unsure of administration attitude. However, the data suggests that interest in supporting SRs is highest among those who are now, or would be, given time away from regular duties (average of 88 on a 100 point scale) or have their workload adjusted to accommodate SR support (average of 91). Those who are not, or would not be, given time away from regular duties showed an average interest of 68, and those who do not have other aspects of their workload decreased showed an average interest of 70. Respondents who indicated they were unsure if they are or would be eligible for time away from regular duties indicated an average interest score of 56, and those who were unsure if they are eligible to receive a reduction in other aspects of their workload indicated an average interest score of 66.

Additional Comments

Respondents received the option to provide additional comments following the closed-ended questions. A common theme from the respondents’ comments was the need to create new positions for SR support, such as a SR librarian, or evidence synthesis librarian, to meet increasing demand for SR support coming from areas outside of the health sciences. Unfortunately, this increase in demand is not always matched with the commensurate support needed. One comment stated:“The requests are repeated and overwhelming […] We’ve talked about how to support this need but honestly, without a reduction in workload, it’s just unsupportable.” One reason noted for an increase in SR requests was the disruption of in-person research caused by the COVID-19 pandemic. A different respondent noted that, even though researchers are increasingly turning to SRs, faculty do not necessarily understand what SR research involves, writing: “They don’t really know what it [systematic reviews] means but it sounds good and could get their work published so they want it – whatever it is.” Lack of support for librarians participating in SRs, whether through training or workload reduction in other areas, was also emphasized in the comments. While the ability to support SRs was seen as an important way to demonstrate value to faculty and the academic community, the need for educating library administrators on this type of research and its importance was also noted.

While most additional comments on SRs were positive, notwithstanding noted areas requiring improvements, some respondents did not see a need for this kind of support in their liaison areas, stating that their faculty do this type of work themselves. For example, one respondent stated: “The other, more newfangled aspects of systematic reviews that you mention here I’d not heard of and do not seem to apply yet to doing comprehensive literature reviews in my collection areas.” For some librarians, there was simply a lack of interest in this type of work, stating that “If poorly implemented, [SRs] represents one of the lesser inspiring and also quite robotic activities in which librarians might engage.”

Discussion

Librarian support of SRs in the SHSS disciplines can be viewed positively in many ways, including acknowledgement of librarian expertise, demonstrating value in library services, and relationship building between research teams and librarians. Our data indicate that SHSS librarians are facing an increasing number of requests for SR support, and many are rising to the challenge regardless of the number of years of experience as a librarian they may have. This supports what the growth in literature over the last ten years,15 development of robust training programs,16 and anecdotal evidence have indicated. Broadly, disciplines within the social sciences are more likely to request librarian support than the sciences or humanities. This aligns with a strong history of SR in psychology, business, and education.17

Consultation was the most frequently reported form of SR support requested. Consultation, whether provided over email, video, or in-person, is a chance for a librarian to provide guidance and answer any questions that a research team may have. Consultation as the most common form of SR support aligns with a thank you via email/in-person/phone as the most common form of acknowledgement (50.7 percent). The percentage of librarians who receive co-authorship of a paper or presentation (39.3 percent) is high and similar to numbers reported by health science librarians,18 although it does not reach the numbers reported by health science library administrators.19 As noted by Ross-White20 authorship on SRs is an important indicator that researchers value librarians’ work and contributions to the team. Consults alone do not typically lead to authorship, but can be formally acknowledged in a paper; they are often the first step to a librarian taking on a more substantive role on a SR team, such as search strategy development, translation, and execution, which were the next most frequently requested support that respondents reported. These types of requests are unsurprising, as they draw upon fundamental areas of librarian expertise.

One of the most striking themes to emerge from our data is the high level of interest in supporting SRs among SHSS librarians. Library administrators should recognize that there is both demand for SR support in SHSS and an appetite by librarians to take on this work. Administrators who wish to encourage SR services should note that librarians who reported knowing that they had organizational supports in place reported higher levels of interest in participating in SR work, while those who were unsure of the organizational supports available to them expressed a similar level of interest to those who said they had no organizational supports. Providing resources for SHSS librarians involved in SR research is important to prevent burnout21 and to ensure the long-term viability of offering this type of service to SHSS researchers and faculty. As such, consideration of SR services and organizational support for SHSS librarians should be a part of library planning.

Librarians and administrators must exercise caution regarding scalability and capacity to provide SR services. Providing high levels of SR support requires extensive training and education by the librarian and a significant time investment. One study found that the average time a librarian spent on a single SR is 26.9 hours (median 18.5 hours).22 Adding support for multiple SRs to an already full workload can lead to burnout; one study found that SR-related burnout is a problem for health sciences librarians, but those who are assigned to spend more than 80 percent of their time on SRs had lower personal burnout scores.23 This suggests that employing dedicated SR librarians or providing time away from regular duties for SR work can help reduce burnout. SHSS librarians who are already involved in SRs or are interested in it for the future, must pay attention to how they manage their workload. SHSS administrators or department heads planning on offering or expanding SR services in the SHSS disciplines should carefully examine existing workflow and capacity to ensure there is adequate staffing, tools, and resources to ensure the long-term feasibility of that type of service. Researchers in SHSS disciplines where SR are a new research methodology will likely need even higher levels of librarian support and training to successfully complete the process.

Limitations and Future Research

The findings of this research are limited by the small sample size and use of convenience sampling. Due to restrictions from our research ethics board, we were unable to directly contact ARL or CARL libraries or librarians to increase the sample size and instead had to rely on the use of listservs to recruit respondents. Based on the actual number of ARL or CARL academic libraries that support institutions or researchers in the SHSS disciplines, we believe there is a larger number of librarians doing this work than is indicated in this study. The original design of this study only included librarians working at ARL or CARL institutions to narrow a suspected large potential number of respondents; future research should include librarians working at any academic institution who support SRs in the SHSS disciplines. Another possible limitation is that SHSS librarians who have not been asked to support SRs in their liaison areas, or who serve disciplines where this methodology is rarely used, may not have considered this survey on SRs relevant, which may have biased the sample. However, the fact that 26 percent of respondents had not yet been asked to participate in a SR demonstrates that at least some librarians who had not completed a SR filled out the survey, providing a small amount of information about the views of this group.

Another limitation of this study relates to data analysis. Most of this analysis is descriptive, as more sophisticated analysis was not possible with the data available. While this prevents us from drawing firm conclusions about the data that can be generalized more broadly, it does provide a snapshot of SR support in an area that has been under-examined to date.

A final limitation is the process used to categorize a librarian’s liaison area(s) into one or more broad categories (e.g., a librarian who listed psychology as their liaison area was categorized into “Social Sciences” ). This is an imperfect process and is open to interpretation, particularly in multidisciplinary areas (e.g., biomedical computing) or those that may be classified differently at different institutions (e.g., kinesiology). However, broad categorization was necessary to maintain participant anonymity and facilitate data analysis. Fortunately, liaison areas that could potentially have been assigned to multiple disciplines were in the minority (see OSF for the categorization scheme). Future surveys could, instead of asking respondents to list their liaison areas in free-text format, provide standardized liaison areas or disciplines and ask respondents to select those they feel most closely aligns with their liaison areas.

This study is among the earliest investigations into SHSS librarians’ involvement in SRs. There are many potential avenues of research and inquiry into library support of SRs in the SHSS disciplines. Examples could include subsequent benchmarking studies that document growth in demand for library support and librarian participation, development of discipline-specific SR standards, and documentation of time spent supporting SHSS SRs.

Conclusion

This study provides evidence that demand for SR support from librarians serving the sciences, humanities, and social sciences at ARL and CARL institutions has increased, and that many SHSS librarians are rising to the challenge of providing that support. However, SR support is a labor-intensive endeavor that is not sustainable as an “off the side of the desk” activity. Library administrators should consider both the benefits and challenges of providing such service and should take steps to adequately provide for the development of library SR expertise and ensure the scalability of library services.

Supplemental Material

All supplemental material, including survey instrument, listservs contacted, and anonymized data are available on the Open Science Framework at https://osf.io/mqxf2.

Acknowledgements

This study was funded by the Canadian Association of Research Libraries’ (CARL) Research in Librarianship grant, and we are extremely grateful for their support. We are also indebted to our colleagues at the University of Manitoba for testing early forms of the survey. We acknowledge the help of Loring Chuchmach from the George & Fay Yee Centre for Healthcare Innovation (CHI) for statistical consulting services.

Contributorship

M.L., C.J.N, & J.W. contributed to the design and implementation of the research, the analysis of the results, and the writing of the manuscript.

Notes

1. Anthea Sutton, Mark Clowes, Louise Preston, and Andrew Booth, “Meeting the review family: exploring review types and associated information retrieval requirements.” Health Information & Libraries Journal 36 (2019): 202-222. https://doi.org/10.1111/hir.12276. David Gough, James Thomas, and Sandy Oliver, An Introduction to Systematic Reviews (London: Sage Publications, 2nd Ed., 2017).

2. Laura Sheble “Macro-Level Diffusion of a Methodological Knowledge Innovation: Research Synthesis Methods, 1972–2011.” Journal of the Association for Information Science and Technology 68, no. 12 (2017): 2693-2708.

3. Thomas L. Mead. “Librarian Participation in Meta-Analysis Projects.” Bulletin of the Medical Library Association 83, no. 4 (1995): 461-64; Jack T. Smith, “Meta-Analysis: The Librarian as a Member of an Interdisciplinary Research Team.” Library Trends 45, no. 2 (1996): 265-79.

4. Institute of Medicine (US) Committee on Standards for Systematic Reviews of Comparative Effectiveness Research. Finding What Works in Health Care: Standards for Systematic Reviews. Eds. J. Eden, L. Levit, and A. Berg, (Washington (DC): National Academies Press, 2011). Lefebvre, Carol, Julie Glanville, Simon Briscoe, Anne Littlewood, Chris Marshall, Maria‐Inti Metzendorf, Anna Noel‐Storr et al. “Searching for and selecting studies.” Cochrane Handbook for Systematic Reviews of Interventions (2019): 67-107.

5. Jonathan B. Koffel, “Use of Recommended Search Strategies in Systematic Reviews and the Impact of Librarian Involvement: A Cross-Sectional Survey of Recent Authors.” PLoS ONE 10, no. 5 (2015): 1-13; Deborah Meert, Nazi Torabi, and John Costella. “Impact of Librarians on Reporting of the Literature Searching Component of Pediatric Systematic Reviews.” Journal of the Medical Library Association 104, no. 4 (2017): 267-77; Melissa L. Rethlefsen, Ann M. Farrell, Leah C. Osterhaus Trzasko, and Tara J. Brigham. “Librarian Co-Authors Correlated with Higher Quality Reported Search Strategies in General Internal Medicine Systematic Reviews.” Journal of Clinical Epidemiology 68, no. 6 (2015): 617-26.

6. Angela J Spencer and Jonathan D. Eldredge. “Roles for Librarians in Systematic Reviews: A Scoping Review.” Journal of the Medical Library Association 106, no. 1 (2018): 45-56.

7. Falk Hoffmann, Katharina Allers, Tanja Rombey, Jasmin Helbach, Amrei Hoffmann, Tim Mathes, and Dawid Pieper. “Nearly 80 Systematic Reviews Were Published Each Day: Observational Study on Trends in Epidemiology and Reporting over the Years 2000-2019.” Journal of Clinical Epidemiology 138 (2021):1-11.

8. Michelle R. Demetres, Drew N. Wright, and Antonio P. DeRosa. “Burnout among Medical and Health Sciences Information Professionals Who Support Systematic Reviews: An Exploratory Study.” Journal of the Medical Library Association 108, no. 1 (2020): 89-97; Genevieve C. Gore and Julie Jones. “Systematic Reviews and Librarians: A Primer for Managers.” Partnership: The Canadian Journal of Library and Information Practice and Research 10, no. 1 (2015).

9. Allison Campbell, Brian Taylor, Jessica Bates, and Una O’Connor-Bones. “Developing and Applying a Protocol for a Systematic Review in the Social Sciences.” New Review of Academic Librarianship 24, no. 1 (2018): 1-22; Karen Chapman, “Characteristics of Systematic Reviews in the Social Sciences.” The Journal of Academic Librarianship 47, no. 5 (2021): 102396; Megan Kocher and Amy Riegelman. “Systematic Reviews and Evidence Synthesis: Resources Beyond the Health Sciences.” College and Research Libraries News 79, no. 5 (2018): 248-52; Ashlynn Kogut, Diana Ramirez, and Margaret J. Foster. “Systematic Review Training Model for Education Librarians: A Case Study.” New Review of Academic Librarianship (2020): 1-22; Diana Papaioannou, Anthea Sutton, Christopher Carroll, Andrew Booth, and Ruth Wong. “Literature Searching for Social Science Systematic Reviews: Consideration of a Range of Search Techniques.” Health Information and Libraries Journal 27, no. 2 (2010): 114-22; Zahra Premji, “Exploring the Involvement of Librarians in Systematic Reviews in Business” (Ontario Library Association Super Conference, January 2020), https://www.olasuperconference.ca/wp-content/uploads/2020/01/Premji_OLA2020_BusinessLibrarianInvolvement_StandardPoster.pdf [accessed 21 April 2022]; Zahra Premji, Ryan Splenda, and Sarah Young. “An Exploration of Business Librarian Participation in Knowledge Synthesis Reviews.” College & Research Libraries; Vol 83, No 2 (2022): 314; Amy Riegelman and Megan Kocher. “A Model for Developing and Implementing a Systematic Review Service for Disciplines Outside of the Health Sciences Amy.” Reference & User Services Quarterly 58, no. 1 (2018): 22-27; Nedelina Tchangalova, Eileen G. Harrington, Stephanie Ritchie, and Sarah Over. “Working across Disciplines and Library Units to Develop a Suite of Systematic Review Services for Researchers.” Collaborative Librarianship 11, no. 4 (2019): 267-81.

10. Premji, “Exploring the Involvement,”11. Maribeth Slebodnik, Kevin Pardon, and Janice Hermer. “Investigating Systematic Reviews Outside Health Sciences.” ALA Annual STS Poster Session 2018. (2018), https://repository.arizona.edu/handle/10150/628518 [accessed 21 April 2022].

12. Koffel, “Use of Recommended,” 1-13.

13. Kogut, Ramirez, and Foster. “Systematic Review Training,” 1-22.

14. “U15 - Group of Canadian Research Universities.” U15, accessed April 29, 2022, https://u15.ca/.

15. Smith “ The Librarian as a Member,” 265-79. Diana Papaioannou, Anthea Sutton, Christopher Carroll, Andrew Booth, and Ruth Wong. “Literature Searching for Social Science Systematic Reviews: Consideration of a Range of Search Techniques.” Health Information and Libraries Journal 27, no. 2 (2010): 114-22; Premji, Splenda, and Young. “Business Librarian Participation,” 314; Slebodnik, Pardon, Hermer. “Systematic Reviews Outside Health Sciences.” ALA Annual STS Poster Session (2019). Jennifer Evans and Pauline Benefield. “Systematic Reviews of Educational Research: Does the Medical Model Fit” British Educational Research Journal, 27, no. 5 (2001): 37-41; Melanie Nind. “Conducting Systematic Review in Education: A Reflexive Narrative.” London Review of Education 4, no. 2 (2006): 183-95; Mark Petticrew and Helen Roberts. Systematic Reviews in the Social Sciences. (Oxford, UK: Blackwell Publishing Ltd, 2006); Robin A Paynter, “Evidence-Based Research in the Applied Social Sciences.” Reference Services Review 37, no. 4 (2009): 435-50; Georgina Guilera, Maite Barrios, and Juana Gómez-Benito. “Meta-Analysis in Psychology: A Bibliometric Study.” Scientometrics 94, no. 3 (2013): 943-54; Jacqueline Davis, Kerrie Mengersen, Sarah Bennett, and Lorraine Mazerolle. “Viewing Systematic Reviews and Meta-Analysis in Social Research through Different Lenses.” SpringerPlus 3, no. 1 (2014): 1-9.

16. Kogut, Ramirez, and Foster. “Systematic Review,” 1-22.

17. Margaret Foster, T. Derek Halling, and Catherine Pepper. “Systematic Reviews Training for Librarians.” Journal of EAHIL 14, no. 1 (2018): 4-8. 17. Sheble “Macro-Level Diffusion,” 2693-2708.

18. Amanda Ross-White. “An Environmental Scan of Librarian Involvement in Systematic Reviews at Queen’s University: 2020 Update.” Journal of the Canadian Health Libraries Association 42, no. 2 (2021): 110-117.

19. Robin Desmeules, Sandy Campbell, and Marlene Dorgan. “Acknowledging Librarians’ Contributions to Systematic Review Searching.” Journal of the Canadian Health Libraries Association 37, no. 2 (2016).

20. Ross-White. “ Environmental Scan,” 110-117.

21. Demetres, Wright, and DeRosa. “Burnout among Medical,” 89-97.

22. Krystal Bullers, Allison M Howard, Ardis Hanson, William D Kearns, John J Orriola, Randall L Polo, and Kristen A Sakmar. “It Takes Longer Than You Think: Librarian Time Spent on Systematic Review Tasks.” Journal of the Medical Library Association: 106, no. 2 (2018): 198.

23. Demetres, Wright, and DeRosa. “Burnout among Medical,” 89-97.