Publication
SNOOP Method: Faithfulness of Text Summarizations for Single Nucleotide Polymorphisms
Wolfgang Maaß; Cicy Agnes; Maxx Richard Rahman; Jonas S. Almeida
In: Proceedings of the 2nd Symposium on Human Partnership with Medical AI: Design, Operationalization, and Ethics at the Association for the Advancement of Artificial Intelligence (AAAI) Summer Symposium 2023. AAAI Summer Symposia, July 17-19, Singapore, Singapore, AAAI, 2023.
Abstract
Time pressures and a heavy workload often limit a physi- cian’s ability to keep up with the increasing number of scientific publications. It is hoped that text summarization by large language models (LLM) can help practitioners quickly identify essential publications. However, it is un- known whether LLMs have been trained on scientific publica- tions in medicine and whether summaries are faithful or even caused by hallucinations. We present the SNOOP method, which uses transformer model embeddings to assess fidelity and hallucinations for different types of LLM summaries and provides an integrated view of results that can be quickly as- sessed by physicians. In the context of genomic medicine, we present results on the performance of SNOOP-enhanced LLMs.