Feasibility Study on LLMs for Preoperative Education
PatientEd and the University of Applied Sciences and Arts Northwestern Switzerland (FHNW) have successfully completed the Innocheque project funded by Innosuisse. The study investigated the possible applications of large language models for preoperative patient education in ophthalmology. The project is relevant for professionals because it addresses key questions about quality assurance and scalability of AI-supported dialogue systems.
Evaluation of Automation-Driven Test Pipelines
In the project, an automated test pipeline with an associated annotation tool was developed. This allows the configuration and evaluation of different models, prompting strategies and parameters. Complementary modules for retrieval, translation and quality assessment support a systematic analysis of several hundred model responses to numerous ophthalmic Q&A pairs. Evaluated criteria included factual correctness, security and empathetic communication. RAG strategies showed weaknesses due to indexation limits, which highlighted the importance of structured reference data.
Classification in current AI research in healthcare
The results make it clear that LLM-based systems offer potential in clinical patient information, but have clear limitations. The moderate agreement between human scoring and automated LLM judge shows that pure model scoring is not a sufficient basis for safety-relevant applications. Professional expertise remains central, especially in sensitive areas such as preoperative discussions, where miscommunication can have a direct impact on treatment success.
PatientEd's contribution and implications for future applications
Dr. med. Amr Saad was responsible for the ophthalmological orientation of the evaluation and the derivation of professional requirements for digital educational dialogues. The project results form a basis for PatientEd's further development of a virtual physician avatar to support standardized clinical information processes. The close cooperation with the FHNW enabled a practical assessment of the technical and medical challenges.
Further development and next project phases
Building on the results, PatientEd plans to expand the pipeline to multi-step dialogs as well as improved integration of retrieval mechanisms for clinical use cases. In addition, future work will focus on bias reduction and validation by subject matter experts in order to further strengthen patient safety and reliability.
