Hello fellow readers! We are Jason Tan and Vaness Chew, Year 3 Pharmacy students from the National University of Singapore (NUS), currently undertaking the SCI-PHI Project. This initiative has provided us with the exciting opportunity to explore real-world challenges through creative and academically rigorous research.

Our project is titled “Navigating the Human-AI Interface: A Qualitative Exploration of Clinicians”. We are investigating clinicians’ opinions and experiences of Explainable AI (xAI) as well as their feedback on hypothetical mock screen scenarios displaying examples of xAI outputs. Our interest in the relevance of emerging technologies in healthcare sparked our curiosity and inspired us to delve into how AI is reshaping the healthcare landscape. Under the strong guidance from our advisors, Dr Jasmine Ong and Dr Alexandra Pyle, we successfully interviewed 20 clinicians from Singapore General Hospital. It was a privilege to be able to interact with professionals from SGH, and it was refreshing to hear about the vast opinions on xAI.

Coming into this project, we were optimistic but admittedly unsure of our ability to interview clinicians, especially given that we had never encountered xAI before. At times, we found ourselves confused about the distinction between conventional AI and xAI, which made us anxious about the possibility of clinicians asking in-depth questions that we might not be prepared to answer.

When reviewing our findings, it allowed us to rethink some of our initial assumptions about xAI adoption. While we initially believed that technical performance and mathematical accuracy would be the primary drivers of adoption, the interviews revealed that trust, usability, and contextual integration are equally, if not, more important to clinicians. We also underestimated the extent to which workflow realities, data infrastructure, and cognitive burden shape clinicians’ receptiveness to new technologies. This highlighted that the successful implementation of xAI is not just a technical challenge, but a deeply human and systemic one. It has shifted our perspective to view xAI less as a plug-and-play tool and more as a solution that must be carefully embedded within the clinical ecosystem, with input from its users at every stage.

However, after completing this project, we have a few key takeaways to highlight. First, the importance of framing complex technologies like xAI within a practical, clinical context. We found that clinicians were most engaged when we discussed how xAI could directly impact their workflows, decision-making, and patient care. Second, the significance of adaptability, particularly during the interviews. While we had prepared an interview guide, we often found ourselves needing to improvise and ask follow-up questions based on the clinicians’ responses, in order to delve deeper into their perspectives.

Recently, we had the opportunity to participate in a Mayo Clinic presentation, which is part of Mayo Clinic’s monthly rounds hosted by their Center for Digital Health team / AI Enablement team. It was an eye-opening experience and a wonderful opportunity to share our work and engage with professionals from diverse backgrounds on how AI is shaping the future of healthcare. The insights gained from these discussions have deepened our appreciation for the transformative potential of AI in clinical practice.

Additionally, if we had the opportunity to do the interview with MayoClinic again, we would dedicate more time to the Q&A segment. Some of the questions raised by the audience highlighted perspectives and concerns that we had not fully anticipated, particularly around automation bias and the lack of formalized frameworks for AI technologies in healthcare.

Nonetheless, we are incredibly grateful for this opportunity and would like to extend our sincere thanks to our advisors, Dr Jasmine and Dr Alex, for their unwavering support and guidance throughout this journey.