Publication
Had enough of experts? Elicitation and evaluation of Bayesian priors from large language models
David Antony Selby; Kai Spriestersbach; Yuichiro Iwashita; Dennis Bappert; Archana Warrier; Sumantrak Mukherjee; Muhammad Nabeel Asim; Koichi Kise; Sebastian Vollmer
In: NeurIPS 2024 Workshop on Bayesian Decision-making and Uncertainty. Neural Information Processing Systems (NeurIPS-2024), Workshop on Bayesian Decision-making and Uncertainty, located at NeuIPS-2024, December 14, Vancouver, BC, Canada, NeurIps Foundation, 2024.
Abstract
Large language models (LLMs) have been extensively studied for their abilities to generate convincing natural language sequences, however their utility for quantitative information retrieval is less well understood. Here we explore the feasibility of LLMs as a mechanism for quantitative knowledge retrieval to aid elicitation of expert-informed prior distributions for Bayesian statistical models. We present a prompt engineering framework, treating an LLM as an interface to scholarly literature, evaluating responses in different contexts and domains. We discuss the implications and challenges of treating LLMs as `experts'.