Navigating Accountability and Oversight in AI-Integrated Clinical Systems
DOI:
https://doi.org/10.5281/zenodo.20442468Keywords:
Artificial intelligence, health, healthcare ethics, ethical theory, beneficence, non-maleficence, patient autonomy.Abstract
AI holds the potential to transform healthcare delivery by improving decision-making and operational efficiency. However, it is important to address the ethical, governance, and operational issues associated with AI-enabled applications before they can be safely and effectively deployed. AI-enabled healthcare presents unique challenges for beneficence, non-maleficence, and patient autonomy. The AI development lifecycle is often not under the control of healthcare institutions, nor are the outputs of AI systems properly understood. Consequently, the true impact of AI on patient outcomes, equity, and justice cannot be adequately evaluated.
Governance frameworks play an essential role in establishing an initial level of assurance. A well-conceived but imperfectly implemented implementation governance framework can help reduce harm and increase public trust. IRBs, in combination with government-sponsored risk management and safety assurance measures, can address most of the requirements of the Safe and Effective Product Regulations. These agencies are best placed to prevent harm emanating from the use of AI-enabled interventions. The next operational development steps focus on building the evidence needed to inform and guide healthcare AI. Proactive information sharing, together with proper documentation and knowledge capes, can mitigate some of the consequences of working without feedback or clinical validation.
References
1. Abràmoff, M. D., Lavin, P. T., Birch, M., Shah, N., & Folk, J. C. (2018). Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digital Medicine, 1, Article 39.
2. Alderman, J. E., Arsenault, C., & Weng, W. H. (2024). Tackling algorithmic bias and promoting transparency in health datasets: The STANDING Together recommendations. The Lancet Digital Health, 6(2), e95–e104.
3. Allen, L. N., et al. (2025). Artificial intelligence in primary care: Frameworks for categorising applications and implications for practice. The Lancet Primary Care, 2(1), 14–26.
4. Bajwa, J., Munir, U., Nori, A., & Williams, B. (2021). Artificial intelligence in healthcare: Transforming the practice of medicine and the delivery of healthcare. Future Healthcare Journal, 8(2), e188–e194.
5. Beauchamp, T. L., & Childress, J. F. (2019). Principles of biomedical ethics (8th ed.). Oxford University Press.
6. Bouderhem, R., et al. (2024). Shaping the future of AI in healthcare through ethics and regulation. Humanities and Social Sciences Communications, 11, Article 2894.
7. Calvert, M. J., et al. (2020). The CONSORT-AI extension for randomized trials involving artificial intelligence: Checklist and explanation. The Lancet Digital Health, 2(10), e537–e548.
8. Cerdá-Alberich, L., et al. (2023). MAIC-10 (Must AI Criteria-10): A brief quality checklist for publications using artificial intelligence in healthcare. International Journal of Environmental Research and Public Health, 20(3), 2808.
9. Collins, G. S., et al. (2024). TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ, 385, e078378.
10. Cross, J. L., et al. (2024). Bias in medical AI: Implications for clinical decision-making. Frontiers in Medicine, 11, 1321145.
11. Davenport, T., & Kalakota, R. (2019). The potential for artificial intelligence in healthcare. Future Healthcare Journal, 6(2), 94–98.
12. Faes, L., et al. (2020). Automated deep learning design for medical imaging classification by healthcare professionals with no coding experience: A feasibility study. The Lancet Digital Health, 2(5), e232–e242.
13. Fehr, J., et al. (2024). A trustworthy AI reality-check: The lack of transparency of approved medical AI tools. NPJ Digital Medicine, 7, Article 10919164.
14. Flanagin, A., Bibbins-Domingo, K., Berkwits, M., & Christiansen, S. L. (2024). Reporting use of artificial intelligence in research and scholarly publication. JAMA, 331(11), 983–984.
15. Gerke, S., Minssen, T., & Cohen, G. (2020). Ethical and legal challenges of artificial intelligence-driven healthcare. Artificial Intelligence in Healthcare, 295–336.
16. General Medical Council. (2022). Regulating doctors in a digital world: Patient safety and the use of AI in clinical care. General Medical Council.
17. Ibrahim, H., Liu, X., Rivera, S. C., Moher, D., Calvert, M. J., & Denniston, A. K. (2021). Guidelines for clinical trial protocols and reports involving artificial intelligence interventions: SPIRIT-AI and CONSORT-AI. Nature Medicine, 27(9), 1472–1477.
18. International Medical Device Regulators Forum. (2023). Good machine learning practice for medical device development: Guiding principles. IMDRF.
19. Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019). Key challenges for delivering clinical impact with artificial intelligence. BMC Medicine, 17, Article 195.
20. Lekadir, K., et al. (2025). FUTURE-AI: International consensus guideline for trustworthy and deployable AI in healthcare. BMJ, 388, bmj-2024-081554.
21. Liu, X., Rivera, S. C., Moher, D., Calvert, M. J., & Denniston, A. K. (2020). Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: The CONSORT-AI extension. The Lancet Digital Health, 2(10), e537–e548.
22. McGenity, C., et al. (2022). Reporting of artificial intelligence diagnostic accuracy study abstracts: An evaluation against reporting guidance. BMJ Open, 12(10), e060839.
23. McMahan, C. J., et al. (2022). Ethical oversight of clinical AI: Practical governance considerations for health systems. Journal of the American Medical Informatics Association, 29(9), 1601–1609.
24. Mennella, C., et al. (2024). Ethical and regulatory challenges of AI technologies in clinical practice. Heliyon, 10(5), e23284.
25. Mihan, A., et al. (2024). Mitigating the risk of artificial intelligence bias in cardiovascular healthcare. The Lancet Digital Health, 6(3), e174–e182.
26. Morley, J., Machado, C. C. V., Burr, C., Cowls, J., Joshi, I., Taddeo, M., & Floridi, L. (2022). Governing data and artificial intelligence for healthcare: Developing an international strategy. JMIR Formative Research, 6(1), e31623.
27. Mongan, J., Moy, L., & Kahn, C. E., Jr. (2020). Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A guide for authors and reviewers. Radiology: Artificial Intelligence, 2(2), e200029.
28. Näher, A. F., et al. (2024). Measuring fairness preferences is important for artificial intelligence in healthcare. The Lancet Digital Health, 6(4), e240–e248.
29. National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). U.S. Department of Commerce.
30. Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453.
31. Panch, T., Mattie, H., & Celi, L. A. (2019). The inconvenient truth about AI in healthcare. NPJ Digital Medicine, 2, Article 77.
32. Rajkomar, A., Hardt, M., Howell, M. D., Corrado, G., & Chin, M. H. (2018). Ensuring fairness in machine learning to advance health equity. Annals of Internal Medicine, 169(12), 866–872.
33. Singhal, A., et al. (2024). Toward fairness, accountability, transparency, and ethics in AI-enabled health information dissemination. JMIR Medical Informatics, 12, e50048.
34. Topol, E. J. (2019). High-performance medicine: The convergence of human and artificial intelligence. Nature Medicine, 25(1), 44–56.
35. Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., Liu, X., Mateen, B. A., Mathur, P., McCradden, M. D., Morgan, L., Ordish, J., Rogers, C., Saria, S., Ting, D. S. W., Watkinson, P., Weber, W., Wheatstone, P., & McCulloch, P. (2022). Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. BMJ, 377, e070904.
Additional Files
Published
Data Availability Statement
The figures and equations presented (confusion matrices, ROC curves, RPN charts) are described as illustrative/simulated, not drawn from a publicly archived dataset.
