AI is transforming the way we search for, locate, and organize scientific information, but the available tools still do not offer sufficient guarantees to replace systematic literature searches conducted by specialists. This is one of the main conclusions of the analysis “AI in Literature Searches: Constraints and Opportunities,” edited by Concepción Campos Asensio, head of the Library, the Continuing Education and Library Committee, and the Clinical Practice Guidelines Committee at Getafe University Hospital in Getafe, Spain, as well as a member of the Board of Trustees of its Foundation for Biomedical Research.

The analysis found that AI platforms can help identify primary studies for systematic reviews, but the takeaway is clear: They are not yet a substitute for a systematic, multidatabase search strategy developed by information specialists.

Higher Precision, Lower Comprehensiveness

One of the main risks identified is the so-called “precision mirage.” AI tools can place highly relevant documents among the top results, which facilitates exploratory searches, but this apparent effectiveness may obscure a significant portion of the available evidence.

According to the studies reviewed by Campos Asensio, this shortcoming affects sensitivity, or the ability to capture all relevant studies. In an assessment by the Methodology Committee of the European Academy of Allergy and Clinical Immunology, AI-based platforms identified 85.7% of articles with a digital object identifier (DOI), but sensitivity fell to 56.3% when the full set of eligible trials was considered. They also missed some studies in gray literature and unpublished trial registries.

The results were even more limited in some fully automated workflows. In reviews on diabetic retinopathy, GPT-4-based systems identified between 6.1% and 32.7% of the articles that were originally included.

Another analysis compared Elicit with the original literature searches of four systematic reviews and found an average sensitivity of 37.9% for the AI tool compared to 93.5% for conventional searches.

Gray Literature: A Weakness

Coverage of gray literature is one of the weaknesses highlighted in the analysis. Conference abstracts, clinical trial registries, and other documents without a DOI may be excluded from the results of AI platforms.

This limitation is significant in systematic reviews in healthcare, which aim to comprehensively identify all studies that meet the inclusion criteria. Therefore, a tool capable of quickly locating relevant references does not, by itself, guarantee a comprehensive search or eliminate the risk for selection bias.

AI as a Complement, Not a Substitute

Rather than completely replacing traditional methods, Campos Asensio proposed a hybrid approach. AI tools can perform various functions within the systematic review process, provided their role is clearly defined in advance and professional oversight is in place.

Potential uses include helping researchers get a better handle on a field’s concepts, terminology, and structure before finalizing a search strategy; uncovering studies missed by conventional searches or broadening citation networks; spotting possible thematic gaps or missing references; and finding new studies or candidates for living reviews and periodic updates.

The author cautions, however, that the terms suggested by these tools may not include controlled vocabularies such as MeSH or Emtree — standardized indexing terms used in biomedical databases like PubMed and Embase to improve search accuracy — so they must be reviewed and validated by specialists.

Expert Searching Remains Central

Literature searching for a systematic review requires combining natural language and controlled vocabulary, adapting strategies to different databases, and considering complementary sources, including gray literature. The goal is to maximize sensitivity and reduce the risk of excluding relevant studies.

In this context, the role of the health sciences librarian remains essential. Campos Asensio’s own experience at biblioGETAFE highlighted the importance of planning the search strategy, selecting sources, and ensuring the reproducibility of the process.

Transparency and Traceability

Another key point highlighted in the analysis is the need to thoroughly document the use of these technologies. When a systematic review incorporates AI tools, it should include, among other details, the platform used, its version or model, date of access, prompts, stopping criteria, sources used, and export processes.

Transparency is essential because models and platforms evolve rapidly, which can make it difficult to exactly replicate a search in the future.

This story was translated from El Médico Interactivo on Univadis, part of the Medscape Professional Network.