A. Rahimi, M. Wimmer, M. Gusenbauer: AISA: A Web-Based Platform for Evaluating the Literature Search Capabilities of LLMs. In: Mauri, A., Burgueño, L., Tommasini, R. (eds) Web Engineering, Proceedings of 26th International Conference, (ICWE) 2026, Lyon, France, June 9-12, 2026, Lecture Notes in Computer Science, vol 16625. Springer, Cham. Doi: 10.1007/978-3-032-29372-5_30
The huge amount of academic data is challenging for research processes, but a rigorous review of prior studies remains a must for scientific progress. While Large Language Models (LLMs) are seen in several studies as potential help, it is evident that they have limitations, including the generation of fake references and misclassifications of research. Existing ways to test LLMs, such as manual checks or text comparisons, are tedious and costly. Therefore, automation support to systematically assess them for this particular context is needed.
To shed more light on this area, this paper presents a methodology and a Web-based platform for assessing LLM-based literature search. The methodology starts by using human-composed systematic literature reviews (SLRs) as a benchmark. The platform provides search prompt templates, sends instantiations of them to LLMs, and then systematically validates if the generated references actually exist in authoritative databases. Subsequently, the platform compares the LLM’s findings with the set of included studies in the benchmark SLRs we use as ground truth. We specifically measure three metrics: how often LLMs generate fake references (existence check), how relevant the reported papers are compared to the ground truth (precision), and how well the LLMs have found all the important papers (recall). Ultimately, this work provides a flexible methodology, a practical platform, and reliable benchmarks to determine the current status of LLMs for academic literature search.