the logo of Lindexer, the literature review software.

Peer review of search strategy for systematic literature reviews 

Literature Review Software - PRESS

Systematic literature reviews are key methods for evidence synthesis and are crucial for evidence-based practice in medicine, as well as for evaluation of medical devices and health technology assessments. The search strategies employed in systematic reviews to retrieve evidence from literature search databases must reconcile high recall, to ensure completeness and avoid missing key publications on the one hand, and reasonable precision on the other hand, to prevent having to screen extremely large numbers of records. 

Why having your search strategy peer reviewed is important

The quality of the searches in systematic literature reviews is of crucial importance since the searches determine the evidence base that will be used for subsequent review and analysis. Therefore, an additional pair of eyes examining your search strategy can be the key to a high-quality literature review. 

A valid, comprehensive and reliable search strategy can be considered the heart of any systematic literature review as this determines the evidence base on which the analysis is based. Validation of the search strategy before finalizing the systematic literature review protocol is an important step to ensure the correctness and comprehensiveness of the searches.  

Peer review of the search protocol of a systematic review by a knowledgeable person, well-versed in searching literature databases as well as familiar with the topic or domain of the systematic literature review at hand can be a good way to ensure a thorough and comprehensive search strategy, well-tailored to the objective and scope of your systematic review. Peer review of the final searches is, however, not a replacement of the informal testing you should do while developing a search strategy and fine-tuning your searches. 

In this blog post, we provide an overview of the PRESS guidelines which provide a framework for peer reviewing electronic searches for systematic reviews. 

Guidelines for reviewing literature database search strategies

The Peer Review of Electronic Search Strategies (PRESS) Guidelines Statement 2 aims to provide an evidence-based guideline for the peer review of literature searches to identify search errors and improve the quality of search strategies for systematic literature reviews and health technology assessments.  Over the years, several checklists and instruments have been developed to check and validate some aspects of the review process. But a structured process for peer reviewing the overall search plan was missing until Sampson et al. in 2009 published what would become the initial version of the PRESS guidelines 1.  These guidelines were developed using an evidence-based process, consisting of three steps: 
  1. A systematic literature review to identify 
    • Instruments to evaluate or validate the quality of literature searches
    • Which elements of an electronic search strategy have an impact on search performance
  2. Surveys to gather expert opinion 
    • For each of the elements identified in the systematic literature review 
    • Experts were asked to evaluate:
      • Potential impact on recall 
      • Potential impact on precision  
      • Whether problems might indicate insufficient expertise of the searcher 
      • Importance in a peer review process 
      • Level of research evidence available
  3. Evidence synthesis and formulation of recommendations 

The peer review process

The PRESS guidelines propose a structured peer review process based on 6 elements, that span the whole process of search strategy development, from conceptualization and structuring the research question over using the correct set of keywords and subject headings to correct use of Boolean operators and correct translation to the query syntax of the database used.

Elements of the PRESS guideline 

PRESS guidelines for peer review of electronic literature searches
1. Is the search question correctly translated into a structured search?
2. Are Boolean and proximity operators used correctly?
3. Are relevant subject headings used?
4. Does text word searching take into account spelling variants, synonyms, acronyms in the appropriate fields?
5. Are spelling, syntax and combinations of searches correct?
6. Are appropriate limits and filters used?
Translation of the primary search to other databases was part of the initial version of the guideline 1, but this element was removed in the 2015 revision, as expert consensus indicated this should be left to the expertise of the searchers

How to use the PRESS guidelines

The PRESS guidelines stipulate that the primary search in a systematic review should be peer reviewed. Both the requestor and the reviewer are expected to be skilled in performing systematic literature searches.  The six elements of the PRESS guideline listed above, as well as the search strategy as a whole are evaluated by the peer reviewer, who may determine that no changes are needed or suggest or even require a revision of the search strategy.  When the peer reviewer indicates that major changes are needed, a second review round should be held. The guideline further recommends that the peer review process should be documented, and that the peer reviewer acknowledged in any resulting publication. They suggest a turnaround time of 5 days for peer review a search strategy.  From a practical point of view, the easiest way to get started with search reviews according to the PRESS statement is to use the accompanying word document published as an appendix 3. 

Information to be supplied by the requestor 

The first part of the assessment serves for the requestor to provide the necessary details on the search to the reviewer: 
  • Title of the systematic review 
  • Type of search strategy: 
    • Primary or secondary 
    • First review or follow-up in response to prior peer review round 
  • Database(s) 
  • Interfaces to access databases 
  • Research question 
  • Inclusion criteria 
  • Exclusion criteria 
  • Search filters 
  • Notes useful for the peer reviewer 

Evaluation by the peer reviewer 

In the second part, the reviewer can then document his/her findings and recommendations and provide details of the revisions needed and/or explain the rationale for the given score.   The six elements of the PRESS guideline and the search strategy as a whole are evaluated using the following scoring system:  
Score
Result
A
No revision needed
B
Revision(s) suggested
C
Revision(s) required

When reviewers suggest or require a revision, they should provide an explanation or example to guide the revision. 

Are the PRESS guidelines useful for MDR-compliant literature reviews?

Search strategies for the literature review of medical devices need to be objective, comprehensive and documented 4,5 but neither the MDR nor any of the available guidance documents requires a formal peer review of the search strategy per se. One could, however, consider the approval of the literature review protocol or clinical evaluation protocol as documentation of review and approval of the documented search strategy. 

For complex literature reviews of medical devices, adding a formal peer review step according to the PRESS guidelines and including peer review documentation to the literature review could be a worthwhile strategy to provide additional evidence on the quality of the search strategy employed in order to convince the notified body that solid, well documented literature searches were used. 

The quality of the search strategy for the literature review can be a deciding factor in the clinical evaluation process in cases where either clinical data from the literature form a substantial part of the amount of the available clinical evidence, or in contrast, when the amount of literature data is very limited. 

In the first scenario, if the clinical evaluation heavily relies on literature data, documented peer review of the search strategy demonstrates the comprehensive, rigorous and objective methodology of the search, lending support to the quality of the evidence base used for clinical evaluation. In the second scenario, when literature evidence is scarce, substantial documentation is provided that the search strategy was wide and comprehensive enough, and the results thus reflect a true paucity of published evidence.  

Whether or not to use search peer review is a strategic decision of the clinical evaluation team, but in case you opt for performing and documenting peer review of your search strategy, the PRESS guidelines provide an easy to use and transparent tool that could be a useful addition to a manufacturer’s clinical evaluation toolbox. 

References and Reading

  1. Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009;62(9):944-952. doi:10.1016/j.jclinepi.2008.10.012
  2. McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016;75:40-46. doi:10.1016/j.jclinepi.2016.01.021
  3. PRESS Guideline — Search Submission & Peer Review Assessment. https://ars.els-cdn.com/content/image/1-s2.0-S0895435616000585-mmc1.docx. Accessed November 27, 2023.
  4. European Commission. MEDDEV. 2.7.1 Rev.4: Clinical Evaluation: A Guide for Manufacturers And Notified Bodies Under Directives 93/42/EEC and 90/385/EEC. MEDDEV 271 Rev4. 2009;(April 2003):1-65.
  5. International Medical Device Regulators Forum. Clinical Evaluation Document. Imdrf Mdce Wg/N56Final2019. 2019;2019.

 

Share This :

Register for a demo.

Pick the most convenient time for you from the following options below: