What do reviewers primarily focus on in order to determine whether a study should be published?

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally that become part of the lore of field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Types of Literature Reviews

Argumentative Review
This form examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review
Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review
Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review
A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review
This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem. Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review
The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

Baumeister, Roy F. and Mark R. Leary. "Writing Narrative Literature Reviews." Review of General Psychology 1 (September 1997): 311-320; Mark R. Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper. 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination. Thousand Oaks, CA: Sage Publications, 1998; Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147; Petticrew, Mark and Helen Roberts. Systematic Reviews in the Social Sciences: A Practical Guide. Malden, MA: Blackwell Publishers, 2006; Torracro, Richard. "Writing Integrative Literature Reviews: Guidelines and Examples." Human Resource Development Review 4 (September 2005): 356-367; Rocco, Tonette S. and Maria S. Plakhotnik. "Literature Reviews, Conceptual Frameworks, and Theoretical Frameworks: Terms, Functions, and Distinctions." Human Ressource Development Review 8 (March 2008): 120-130; Sutton, Anthea. Systematic Approaches to a Successful Literature Review. Los Angeles, CA: Sage Publications, 2016.

Scientific findings and discoveries can have far-reaching implications for individuals and society. This is one reason why they undergo a process of quality control known as 'peer review' before they are published.

Peer review involves subjecting the author's scholarly work and research to the scrutiny of other experts in the same field to check its validity and evaluate its suitability for publication.
A peer review helps the publisher decide whether a work should be accepted.

How the peer review process works

When a scholarly work is submitted to a scientific journal, it first undergoes a preliminary check known as a desk review. The editor decides if the manuscript should be sent for peer review or be immediately rejected. The next step is to select experts from the same field who are qualified and able to review the work impartially. Ideally the work is evaluated by multiple experts.

The primary goals of a peer review are to determine whether a scholarly work falls within the journal's scope, to check whether the research topic has been clearly formulated, and to decide if a suitable approach has been taken to address the scientific issues involved. The reviewer also examines the methodology to determine whether the author's results can be reproduced, and he or she assesses the novelty and originality of the research findings. If a work involves patients or animals, then the peer review will also cover ethical aspects. Finally, the reviewer will also rate the 'readability' of the work, assessing how logically the argument has been constructed and whether the conclusions are well-founded. In addition, the author of the work will generally receive useful advice on how to improve their work.

Peer reviewers normally provide their assessment in the form of a questionnaire which they return to the editor. This forms the basis for deciding whether the work should be accepted, considered acceptable with revisions, or rejected. Submissions with serious failings will be rejected, though they can be re-submitted once they have been thoroughly revised. In principle, editors have the final say in how many reviewer reports they obtain and to what extent they follow the recommendations therein.

If a work is rejected, this does not necessarily mean it is of poor quality. A paper may also be rejected because it doesn't fall within the journal's area of specialisation or because it doesn't meet the high standards of novelty and originality required by the journal in question. Another reason a paper may be rejected is that the reviewers do not agree that the approach taken by the author is innovative. There are also some journals which take a more relaxed stance in regard to originality and focus more on the extent to which the author has followed correct scientific procedures. It is therefore common for authors to submit their paper to a different journal after receiving a rejection.  Some prestigious journals reject over 90 percent of papers submitted to them, while the rejection rate across all scientific journals is somewhere in the region of 50 percent. If an article is rejected, authors usually resubmit to another journal. Whether a resubmission to the same journal is possible is indicated in the respective journal's policy and should be discussed with the editor if necessary.

Different types of peer review

The term peer review actually encompasses a number of different approaches, the most common of which are the following:

• single-blind peer review: the name of the reviewer is hidden from the author;
• double-blind peer review: both the reviewer and the author remain anonymous to each other.

There are also considerable differences in the level of detail with which papers are evaluated. For example, some journals make additional use of anti-plagiarism software, organise separate reviews of the author's methods and statistics, or examine the submitted illustrations to detect whether they have been manipulated. There is also an increasing number of journals which focus on scientific software or research datasets, and the peer review process has been adapted to fit these contexts, too.

Peer review is also used by conference organisers to select which contributions to include in their programme. And funding bodies even use peer review methods to assess the eligibility of research proposals for funding.

Criticisms of peer review

All the methods mentioned above have their advantages and disadvantages. For example, critics of the double-blind method argue that reviewers can guess who the authors are by looking at the references they cite. They suggest that this could undermine their neutral stance. The evaluation of an academic paper is also affected by the reviewer's scientific beliefs and by the care and effort they choose to invest in the process.

Peer review has recently come in for major criticism following cases where reviewers failed to spot serious errors in the author's methodology. The reasons for 'failures' in the peer review process include peer reviewers' heavy workloads as more papers are published and poor selection of reviewers by editorial boards.

Another objection that is frequently raised is that peer review is not transparent enough, not just because the reviews are inevitably subjective (especially if reviewers are unable to separate themselves adequately from their particular schools of thought), but also because reviewers may not appreciate the value of a new idea or may withhold – or simply not be asked to provide – relevant information on conflicts of interest.

Another key criticism of peer review is that the process may stretch over a long period of time, generally weeks or months, but occasionally even years.

Some people also suspect that journals which claim to have implemented peer review actually carry out very superficial assessments, or none at all. For further information see the FAQ on "Predatory Publishing".

It is generally accepted that peer review cannot completely eliminate cases of fraud and the publication of low-quality papers. Nevertheless, peer review continues to be favoured despite all the criticism because it has ultimately proved its worth and shown that in most cases it can help improve the quality of publications – especially if authors are able to view the report and work through the comments. In the end, of course, responsibility lies with the authors who are required to demonstrate rigour, probity and scientific reproducibility as part of the scientific process. The peer review concept is also constantly being adapted to counter criticism such as the points mentioned above.

Alternatives to the standard peer review process

The variants described above are all forms of pre-publication peer review, in which an article is peer-reviewed prior to its formal publication. But efforts are also being made to experiment with different alternatives, with the hope that these may address some of the criticisms mentioned above:

• Open peer review (or crowd-sourced peer review) is a collective term for various methods that seek to open up the peer-review process, for example by revealing the identity of authors and peer reviewers or even providing access to the reviews themselves.
• Post-publication peer review is a generic term for all forms of feedback and commentary on an article that take place after publication. It can therefore be combined with open peer review. The basic idea is to publish a manuscript after nothing more than a rough, preliminary review – or no review at all – and to leave the corresponding evaluation and assessment to the scientific community. The F1000 platform is a well-known example of this approach. In addition, post-publication peer review generally allows people to publish comments and feedback on articles that have already been published. These comments are submitted within a formalised process, either on the publishing platform itself or on external platforms such as PubPeer.
• Other alternatives include redirecting rejected manuscripts to another, potentially more suitable journal together with the review reports, thereby shortening the subsequent peer-review process. This is typically referred to as cascading peer review when the manuscript is submitted to another journal from the same publisher, or portable peer review when the manuscript and its peer-review reports are also submitted to a different publisher. The goal is to avoid needlessly duplicated review effort. A similar approach is taken by the Review Commons platform, an initiative that works in affiliation with a consortium of life science journals. Review Commons makes manuscripts available on a preprint server before submission to a journal, where they can be peer-reviewed independently of any particular journal. The authors then use the peer-review reports as a basis for deciding which of the participating journals is the best choice for their manuscript. They then submit this to the journal together with the peer-review reports.

All of these alternatives seek to address specific problems of the peer-review process, yet some of them may raise new challenges of their own.

Informal methods of obtaining feedback

Another option is to seek informal feedback on a manuscript, a process that is typically initiated by the author themselves. One possibility is to simply send the manuscript to colleagues and ask for their feedback; another is to upload the manuscript to a preprint server and share the link on a blog, via email, or on social media channels such as Twitter. The advantage of this approach is that the author can quickly get feedback on their manuscript before they submit it. The downside is that the author cannot be sure that their manuscript has been thoroughly reviewed, and the advice and comments they do get may offer little help in improving the manuscript. What is more, this approach is not recognised as a valid form of peer review in the strictest sense of the word.

Rapid peer review or fast-track peer review

The traditional peer-review process generally takes weeks or months. Some life-science journals have now introduced a fast-track peer-review process for submissions on specific topics, for example those relating to public health. This “accelerated” process draws on a pool of  reviewers who have agreed to provide quicker reviews while maintaining the same degree of thoroughness. This fast-track process provides the fastest possible access to scientific findings in the area concerned without compromising on quality assurance. To find out whether a journal offers this service, please consult their website.

Acknowledging peer review as a service

As a rule, peer reviewers do not get paid for their work, because peer review is an integral part of the scientific process. Some publishers “reward” their reviewers by granting them free access to the publisher’s archive for a limited period of time.

Peer reviewers can also increase the visibility of the services they provide by making use of platforms such as Publons, which give researchers an opportunity to present their contributions to the scientific quality-assurance process. Publons allows researchers to indicate how many manuscripts they have peer-reviewed for each journal. However, the peer-review reports themselves can only be viewed if the journal has installed an open peer-review process and if the authors and reviewers consent to their publication.

Researchers can also refer to their peer-review activities on their CV.

How to become a peer reviewer

In order to evaluate other researchers’ manuscripts, you need a certain amount of expertise in the subject area. Editors mostly become aware of potential reviewers through publications or lectures. In addition, major publishers also maintain reviewer databases that contain each person’s specialist subject areas and contact details. Some journals also allow authors to suggest reviewers when they submit their manuscript. This makes the editor’s work easier, and they are not obliged to accept the suggestion.

How many peer reviews should I write?

There is no standard answer to this question because it largely depends on each individual’s interests, resources and available capacity. As a rule of thumb, you should aim to write as many peer reviews as you have received for your own work.

Peer reviews and good research practice

Since peer review is an integral part of the scientific process, it is also subject to the rules of good research practice. This means candidates are expected to disclose any lack of expertise or possible conflicts of interest to the editorial board. It also means that any manipulation of the peer-review process or the submission of incorrect or discriminatory peer-review reports is regarded as scientific misconduct.

See also

Journal quality and standing: which aspects are relevant to open access?

Faster dissemination of research findings – key facts about preprints

An introduction to good research practice, research misconduct and academic integrity

Disclaimer

Important note: The information and links provided here do not represent any form of binding legal advice. They are solely intended to provide an initial basis to help get you on the right track. ZB MED – Information Centre for Life Sciences has carefully checked the information included in the list of FAQs. However, we are unable to accept any liability whatsoever for any errors it may contain. Unless indicated otherwise, any statements concerning individual statutory norms or regulations refer to German law (FAQ updated 03/2022).