01_Editorial

Editorial

Considering Developmental Peer Review

This editorial is a collaborative discussion of College & Research Libraries’ open peer review experiment, representing the unique perspectives and voices of those playing roles.

Authors: Sarah Hare and Cara Evanson

Reviewers: Emily Ford and John Budd

Editor: Wendi Arant Kaspar

The past couple of years have seen high level discussions and concerted efforts in ACRL on open peer review, as a more inclusive and transparent process for assessing research and best practice. Emily Ford, Chair of ACRL’s Publication Coordinating Committee (PCC), raised level of awareness and worked with the Editorial Board of C&RL to investigate how open peer review (OPR) might look for the journal.

Reviewers: It is hard to say when the idea of open peer review at C&RL, ACRL’s flagship journal, was initially hatched. Before Wendi’s tenure at the journal, the Editorial Board had reviewed and discussed a proposal to experiment with open peer review, but conversations continued into the beginning of her editorship.

To say that there has been a little concern about open peer review in academia is an understatement. In fact, it remains a controversial process, largely due to its evolving nature, and due to communities not understanding its purpose or impact.

There were a couple of conversations happening at the same time—one was the desire to try open peer review for purposes of, as I like to say, “opening the black box” and making the process more transparent. In addition, the Board was also discussing the desire to be able to provide more guidance for authors beyond review feedback. There are many submissions with great ideas and lots of potential, but needing more advice and guidance that is appropriate for a either the traditional reviewer or editor role. This is not unusual for authors who may be new to research or writing; they may benefit from some mentoring in specific areas.

While most peer review is evaluative in nature, ne type of open peer review, namely developmental peer review, offers “generous feedback provided by reviewers, coupled with editors’ guidance in navigating feedback from multiple reviewers, [which] can produce papers that are clearer, more impactful, and more effectively executed.”1 In his blog, the editor of the American Sociological Review address common misconceptions about developmental review and provides very even handed guidelines.2 There are concerns raised about the amount of time needed, the concern about a potential lack of objectivity,

Authors: When we began the process of writing the article that would eventually become “Information Privilege Outreach for Undergraduate Students,”3 we had little prior experience with publishing. Over the previous year, we had focused on information privilege outreach in various ways with our students, including a survey, an experiential scenario, a panel discussion, and a workshop for graduating seniors. We were excited to share our work in this area with a broader audience. We ultimately decided to address the lack of scholarly literature on the topic of information privilege by drafting our own article.

We spent nine months working through an initial draft and rewrites. The part we dedicated the most time to re-working was the literature review, but each subsequent draft never quite seemed to work. Without a clear base of scholarly information privilege literature to build on, the possibilities for the framing we could use were numerous. We cycled through multiple iterations of the literature review and tried out a new emphasis for each re-write, including a focus on the intersections with scholarly communication work and the socioeconomic divisions in information access. Beyond the literature review, we were worried we did not have enough quantifiable data. We did the best we could to draft the article and analyze the data on our own for the first few months, and then reached out to those in our professional circles for feedback. Ultimately, though, we felt stuck.

When I receive a query or a submission that addresses an original topic or unique perspective, it is a thrill. An editor has the responsibility to steward the journal, maintaining the quality of the research while help to evolve scholarship in the field. As a journal affiliated with the largest professional association in ALA, which implicitly has an education mission, I feel that there is also some obligation to help develop rigorous research and professional writing.

Reviewer (John): I will add that only about half of the accredited master’s programs offer a research methods course, and in most instances where one is offered, it is usually an elective course. Beginning researchers do frequently need assistance with the intricacies of writing for publication. Open peer review is an alternative that seems to hold considerable promise. I have to echo the benefits that Emily enumerates; the conversation can be invaluable.

Authors: We were not sure if our manuscript was suitable for publication but we were equally unsure how to move forward to make it into a publishable document. In Spring 2017, we contacted the College and Research Libraries editor about our article. We thought it could be of interest to a broad audience, including both information literacy and scholarly communication practitioners. However, this made selecting a journal with the appropriate scope and readership challenging. After several e-mails, the editor agreed that our article was appropriate for submission to the journal and suggested OPR as a potential option. We agreed to participate in a C&RL OPR pilot.

It is not unusual for an editor to receive initial queries about the suitability of a topic or a study outside of the system. When the authors queried the editor about their paper topic, it was when we were thinking about these issues of mentoring authors and discussing what open peer review might look like for C&RL. Given the topic of their study, information privilege, it seemed a natural choice as it filled a gap in the literature and was aligned with the issues we were discussing.

The C&RL Editorial Board were also committed to developing potential authors and willing to jump in and explore the possibilities. When I queried Emily Ford, Chair of PCC and John Budd, one of our long-standing Editorial Board members, they both enthusiastically agreed to help spearhead a developmental peer review pilot.

Reviewer (Emily): First, in my own experience with our professional literature and community, my experiences with open peer review had been incredibly robust and allowed me to create better work than my experiences with closed review. Second, the open ethos of librarianship and library scholarly publishing, in my view, should be a leader in scholarly communication. As academic librarians we work with researchers publishing in various scholarly communities. Some of those communities, such as STEM fields, have a greater acceptance and adoption of open access generally, and are pushing the proverbial envelope when it comes to publishing processes. Opinion pieces as well as the research literature have surfaced many issues with blind or double-blind peer review, unveiling issues of reviewer abuse, gratuitously long timelines from submission to review and acceptance, and the general opacity of review and editorial processes. In my view open peer review addresses all of these issues and affords us the opportunity to approach peer review in a more robust way.

Reviewer (John): I’ll confess that I have something of an advantage in this process; I have served as an editor of a journal and of a monograph series. So, I have had the opportunity to work closely with authors with a clear view to providing constructive advice and helping to make the works publishable. While my experiences have not involved an open process, the end goal has been similar—to assist writers in improving their work in all ways. Admittedly, when the research questions and methodology are sound, the editorial work becomes easier. That is definitely descriptive of this specific experience.

I chose reviewers that had experience with reviewing, doing research, could assess the topic of the study and were willing to be guinea pigs in this experience. It was important to me that this process was organic, derived from the expectations and negotiations between the authors and reviewers. I deliberately minimized the instructions, indicating only that we wanted to maintain the standard of rigor of the journal and, thus acceptance was not guaranteed.

Authors: The editor selected two peer reviewers, John Budd and Emily Ford, with whom we communicated about timeline and respective responsibilities before moving forward.

Reviewers: When Wendi approached us with the idea of experimenting with OPR at C&RL using Sarah and Cara’s article as a test case, we were both excited to participate. In collaboration with Wendi we formed a review process, outlining communication practices, technology nitty-gritty, and discussing appropriate timelines for reviewer feedback and revision timelines.

Then, when Sarah and Cara agreed to this trial experiment, we were able to review and tweak our guidelines using their input. Collaboratively creating a timeline of due dates for drafts and feedback, we were able to keep the project moving forward.

Authors: We submitted our draft to John and Emily and they sent us their feedback. The four of us then met to discuss the article over Zoom. The call was extremely helpful and provided clear direction for us on how to move forward.

Reviewer (Emily): Direct communication between reviewers and authors enables discourse, a genuine back and forth exchange of ideas, a real-time conversation. Finally, open peer review is a way to work toward equity in scholarly publishing. The scholarly publishing landscape suffers from elitism and issues of hierarchy, as many have argued, and open peer review can be a way to alleviate some of these problems. It can surface potential bias and abuse, as well as make transparent reviewer and editorial processes and decisions. It can begin to open a space to allow students, early career researchers, and community members from various life experiences and backgrounds to engage in open conversation.

Reviewer (John): The fundamental structure of the submitted paper was sound, so the interaction that we had with authors could be along the lines of asking for (and offering particular suggestions) for clarification and greater detail in certain sections of the paper. We were able to respond directly to the authors when they had questions so as to facilitate the completion of a publishable paper.

I made a deliberate choice, as the editor, to play a minimal role (at this point) in the feedback to authors and the communication back and forth. This was to maintain arm’s distance from the review process, allowing the reviewers to bring their own expertise into play and determine that methods of engagements. It also allowed me to be in a more objective position to review the “revised” submission without knowledge of the coaching and detail that went into it.

Reviewers: Part of this back and forth conversation is the ability not just for reviewers to provide input to authors, but to allow reviewers to engage with one another. In this instance, John was able to send comments to authors prior to Emily. In her comments, Emily was able to corroborate points that John made and add her own twist on them. The ability for Sarah and Cara to ask us directly to clarify our points was incredibly helpful. Reviewers are people, too, and our written comments can also suffer from lack of clarity in language. Instead of having clarifying questions mediated through an editor, were both able to respond to clarifying questions with Sarah and Cara via e-mail and during our Zoom conversation. In this experience we were on the same page.

Authors: We are thankful for the opportunity we had to have a dialogue with our reviewers. There were several instances in which asking John and Emily questions about their initial comments gave us a chance to explain what we had meant to say (but had not clearly articulated) in a certain passage. Then, with more clarity about our intentions, they gave us more targeted feedback. For instance, talking together on the Zoom call helped us understand how certain terminology could be interpreted in ways we did not intend.

Reviewers: We both saw the need for Sarah and Cara to more explicitly present a theoretical foundation or perspective tied to their work, and we were able to mentor them in doing so. The perspective was in their experience and thinking, it just hadn’t translated to the written page. We also both agreed and pointed out that the article could be tied a bit more closely with ACRL’s Framework for Information Literacy. In fact, we seconded most of one another’s comments, which offered Sarah and Cara clarity and direction in their revisions.

Authors: Our conversation also led John and Emily to suggest critical information literacy as a theory to undergird our writing, which enabled us to finally find a focus that worked for the literature review. They helped us reframe our article focus, identify places where we were making assumptions or generalizations, and understand that our article could take a more qualitative instead of quantitative approach.

Through this effort, the reviewers and authors communicated clearly and negotiated various stages and timelines. The result was a draft process to define how developmental peer review might go (appended).

Authors: The OPR process was a transformative experience for us. We are uncertain that our article would have made it through peer review to publication without the guidance, rigor, and transparency of OPR. As a result, the article is now out in the world, connected to the reputation of C&RL, to be published in September 2018 with a pre-print version already available. The full text of the pre-print was downloaded over four hundred times between October 2017 and June 2018. One result of the article is an information privilege class that is being developed at the University of Tennessee, which is slated to be taught in Fall 2018 by Charissa Powell. This success is a direct result of OPR: in addition to the benefits of the process and our resulting development as authors, the article itself improved, contributing to the impact that it will be able to have on the profession.

A typical criticism of OPR is that it breeds bias, and perhaps in some contexts it might, but it also flattens hierarchies and builds authorial voice and confidence. Moreover, when responsibilities, roles, and timelines are clearly defined, OPR is transparent and still constructive and rigorous. Dispelling myths about OPR is the first step to giving early career librarians options and venues for pursuing OPR and diversifying the LIS publishing ecosystem. We both benefited from OPR and see the importance of making similar models accessible for other early-career librarians and new researchers.

So where do we go from here?

Overall, the experience was positive and productive. The authors experience as an affirmation to the editor, the reviewers and the Editorial Board that developmental review was a direction the journal should continue to explore:

Authors: As early career researchers and librarians, the process of open peer review has been both encouraging and educational for us. After drafting our article, we knew that we wanted it to go through the peer review process. Peer review is often synonymous with the rigor and reputation we both have been working to build as librarians. More importantly, we knew that the peer review process would enhance the article, refine our writing, and strengthen the ideas we presented. We are thankful that College and Research Libraries, a top open access journal in Library and Information Science, was open to experimenting with open peer review. We believe that C&RL’s willingness to pilot OPR demonstrates the journal’s enthusiasm for and commitment to the scholarship and growth of early career librarians. OPR has been immediately useful for us as authors and we know that it has the potential to have an even larger impact on the profession as a whole.

The response from ACRL’s Executive Board was also affirming with some points to consider: “Overall, the Board supports open peer review and believes this aligns with the ACRL Plan for Excellence. The Board expressed some concern regarding the quick turnaround for the process, but was pleased to hear that PCC is willing to adjust authors’ timeline if needed. The Board was also pleased to hear that the open peer review would be an “opt in” process so that librarians would not be required to do this. The Board recognized there might be some cases where open peer review may not be acceptable in a tenure review.”

The Editorial Board discussed the next steps in the process at ALA Annual in New Orleans. Given the positive experience and the outcomes of the pilot process, there is a desire to formalize a process of developmental peer review process while still maintaining a process of double blind review. The Board is committed to developing authors in the discipline and in the profession while maintaining the standards of quality and objectivity that authors, readers and practitioners expect in C&RL.

In discussions with the Editorial Board, it was also acknowledged that a small percentage of submissions would actually be appropriate for developmental peer review. The most likely candidates would be innovative studies, those that were filling a needed gap, where the study method was rigorous (and thus would not have to be redone); however, there needed to be improvement in the framing, literature review, explanation, findings, analysis, implications for the profession—enough that more than the usual feedback from reviewers would be necessary.

The primary question is the identification of potential submissions to go through developmental peer review. In that case, the editor or, more likely, one of the reviewers might indicate that a submission going through the established double-blind process was original and had high potential but needed substantial work. The reviewer could suggest the potential for developmental peer review with the review comments and indicate interest in serving as a reviewer for that process.

The editor would look at all of the reviewer feedback and consider whether it was a good candidate for developmental peer review. If it was determined that the submission was a suitable candidate for developmental peer review, the editor would contact the author(s) and explain the offer and the process:

  • If the authors declined to participate, the editor would make a determination about the submission and send the blind reviewer feedback to the authors who might revise or not, depending on the decision.
  • Should the authors agree, the editor would then contact the other reviewer on the manuscript and ask if they were amenable to serving in the developmental peer review. If not, the editor would identify another reviewer qualified to do so and facilitate the initial communication between author(s) and reviewers, clearly defining the expectation and the model process.

This process of developmental peer review would not be as straightforward as blind review. It would indicate additional rounds of review as well as more time and engagement from both the authors and the reviewers. Because of the added investment and the commitment to the development process in terms of guiding the authors and evolving the submission toward its best version, this would not be the standard review process for the journal.

By maintaining the traditional blind process, which seeks to be impartial and candid, while also identifying candidates for the developmental peer review process, it allows College & Research Libraries to meet two significant and diverse goals: 1) maintain the rigor of the research in the profession; and 2) play a mentoring role in shaping future scholars and the evolution of scholarship and practice in the profession.

Many thanks to Sarah Hare and Cara Evanson for their courage and willingness to be the “first,” allowing C&RL to try out this model and share their story. I also want to commend Emily Ford and John Budd for engaging at a high level of professionalism and expertise, with care and with candor.

Open and Developmental Peer Review for C&RL (June 2017)

Process

  1. Editor queries author for potential developmental review process.
  2. Author chooses developmental peer review process.
  3. Reviewers for developmental peer review are identified and accept review.
  4. Editor shares the submission with authors and reviewers via Google Docs allowing track changes and comments.
  5. Reviewers have 2 weeks to make comments and suggest edits to the document. Narrative suggestions may be added at the end of the document. The purpose is to make sure that both reviewers and the author can see all of these comments. During this 2-week period, the author should not respond to reviewer comments or questions, though they will be curious, so are free to look at cogitate on what they see.
  6. After the 2-week period of reviewing ends, authors have 1 week to respond to comments on the document, requesting reviewers for clarification of points, posing questions to the reviewers and generally engaging in dialogue on the document. During this 2-week period it might be appropriate for the author to speak with reviewers via telephone, if desired.
  7. The author then has another 2 weeks to revise the document and share again with reviewers. Revisions should be clearly labeled (easy to do with commenting functions in Google Docs), and may also add narrative about changes at the end of the document.
  8. Reviewers then have 2 more weeks to review revisions, and then send their collaborative recommendation to the Editor.

Guidelines for Authors

In addition to C&RL’s Author Guidelines,4 authors engaging in open and developmental peer review should approach the review process as an opportunity to improve their work with support from reviewer colleagues. As with other review processes, not all suggestions by reviewers or conversations with reviewers may result in changes to the submission. An author’s revised submission should directly address reviewer comments, including their reasoning for not taking suggestions, or moving the submission in a different direction.

Guidelines for Reviewers

In addition to C&RL’s Review of Manuscripts, reviewers engaging in open and developmental peer review should be prepared for a more hands-on reviewing experience. In open and developmental peer review, reviewers should provide thoughtful suggestions and pose critical questions to authors with the intent to support authors in further developing and presenting their ideas in article form. Feedback given to authors should be direct, clear, and concise. In this review paradigm, reviewers are encouraged to engage in conversation with other peer reviewers via commenting functions, as well as with article authors in the interest of supporting and strengthening the scholarship under review.

Notes

1. Rory McVeigh, “The Developmental Peer Review: Clearing Up Some Misunderstandings,” (2016), available online at http://speak4sociology.org/2016/09/19/the-developmental-peer-review-clearing-up-some-misunderstandings/ [accessed 6 August 2018].

2. “Reviewer Guidelines.” American Sociological Review (July 2015), available online at http://journals.sagepub.com/page/asr/help/reviewer-guidelines [accessed 6 August 2018].

3. Sarah Hare and Cara Evanson. “Information Privilege Outreach for Undergraduate Students.” College & Research Libraries 79, no. 6 (2018): 726–36.

4. See: https://crl.acrl.org/index.php/crl/about/submissions [accessed 13 August 2018].

Copyright Wendi Arant Kaspar, Sarah Hare, Cara Evanson, Emily Ford, John M. Budd


Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Article Views (Last 12 Months)

No data available

Contact ACRL for article usage statistics from 2010-April 2017.

Article Views (By Year/Month)

2020
January: 7
February: 14
March: 13
April: 16
May: 8
June: 26
July: 11
August: 14
September: 17
2019
January: 18
February: 13
March: 18
April: 9
May: 29
June: 15
July: 12
August: 56
September: 11
October: 9
November: 14
December: 10
2018
January: 0
February: 0
March: 0
April: 0
May: 0
June: 0
July: 0
August: 8
September: 380
October: 99
November: 43
December: 30