Authors (including presenting author) :
HO CM (1), Guan S (2), HUI V (2)(5), LAM C (3), MOK P (4)
Affiliation :
1. Department of Neurosurgery, Queen Elizabeth Hospital, Hong Kong 2. Center for Smart Health, School of Nursing, The Hong Kong Polytechnic University, Hong Kong 3. Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong 4. Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong 5. Health and community systems, School of Nursing, University of Pittsburgh, PA, USA
Introduction :
Perioperative education is crucial for optimizing patient outcomes in neuroendovascular procedures. Inadequate understanding can heighten patient anxiety and hinder adherence to care plans. Traditional education approaches, which rely on face-to-face consultations and printed materials, often constrained by limited availability of healthcare professionals and lack of personalization. While AI-powered chatbots have demonstrated efficacy in various healthcare fields, their role in neuroendovascular perioperative support remains underexplored. This study introduces NeuroBot, a large language model- driven chatbot designed to offer tailored perioperative guidance and improve communication between patients and healthcare professionals.
Objectives :
This study aims to evaluate the feasibility, usability, and validity of NeuroBot in providing perioperative support for neuroendovascular procedures. The study will validate the accuracy, relevance, and completeness of chatbot responses through evaluations by neurosurgical domain expert panels, and explore expert perceptions to investigate the utility and clinical applicability of NeuroBot. Additionally, the study will identify barriers and facilitators to adoption into clinical care setting.
Methodology :
This study employed a mixed-methods approach, which encompassed three phases: 1. Internal Validation: 6 neurosurgical nurse specialists conducted independently rated responses generated from three models of AI- chatbot, Assistant API, ChatGPT, and Qwen, across 306 bilingual neuroendovascular-related questions, using 6-point (accuracy and relevance) and 3-point (completeness) Likert scale. 2. External Validation: Ten neurosurgical domain experts independently rated NeuroBot responses using the same evaluation metrics. The Intraclass correlation coefficient was evaluated to assess the consistency of scoring from these experts. 3. Qualitative Study: A total of 18 neurosurgical nurses and doctors were recruited to participate in individual semi-structured interviews. Interviews were recorded, transcribed, and analyzed using an inductive thematic analysis approach.
Result & Outcome :
Findings: The Assistant API outperformed ChatGPT and Qwen, achieving a mean accuracy score of 5.7/6. External expert ratings for NeuroBot showed significant improvements, with accuracy (5.7/6), relevance (5.58/6), and completeness (2.7/3). Qualitative insights from healthcare staff highlighted NeuroBot’s potential to reduce staff workload, improve patient education, and deliver validated and highly accurate responses. There are several areas for enhancement identified, which include expanding domain coverage, communicating in empathetic language, and human- computer interface improvements. Conclusion: NeuroBot demonstrates the promise of AI chatbots in perioperative neuroendovascular care, offering scalable, continuous support. Further research is necessary to explore the impact of NeuroBot on patient knowledge psychological distress, and overall satisfaction. By continuously refining NeuroBot based on user feedback and clinical evaluations, we can enhance its capacity to foster patient-centered communication, optimize clinical outcomes, and advance AI-driven innovations in healthcare delivery.
