Transformation of Information Services Using Retrieval-Augmented Generation and Large Language Model

Abstract

The openness of access to public information is regulated in Law No. 14 of 2008 on Public Information Disclosure as a key element in realizing transparency and good governance. At the University of Lampung, Pejabat Pengelola Informasi dan Dokumentasi (PPID) provides various public data through its official website. However, users often face difficulties in finding specific information due to the dispersed nature of documents and the limitations of search features. To address this issue, a Telegram chatbot powered by Artificial Intelligence was developed using the Large Language Model (LLM) Qwen2.5 VL 72B Instruct integrated with the Retrieval-Augmented Generation (RAG) architecture. Data were collected from the University ofLampung’s PPID website and organized into a structured dataset of 375 curated question–answer pairs derived from 47 public information entries. These were processed through tokenization, embedding, and indexing using the FAISS vector database to support semantic search. The evaluation was carried out using RAGAS metrics (faithfulness, answer relevance, and context recall)with a minimum threshold of 0.8, along with a performance comparison against a pure LLM. The results show that the developed system successfully surpassed the RAGAS minimum threshold with a remarkably low average response latency of 0.69 seconds and achieved higher accuracy compared to the pure LLM, with a significant reduction in answer hallucinations. Furthermore, usability testing using the Chatbot Usability Questionnaire (CUQ) produced an average score of 90.62, indicating an excellent user experience. This study concludes that integrating LLM and RAG in a chatbot effectively improves accuracy, relevance, and the ease of real-time access to public information.