Background and definitions: Artificial Intelligence (AI) is a rapidly evolving field with potentially countless applications in palliative care. AI is a division of computer science that focuses on the simulation of intelligent behavior in computers to support decisions or execute tasks (1). Machine learning (ML) is a subfield of AI in which a computer improves its own performance by continuously incorporating newly generated data from its environment into an existing iterative model (2). The technology is currently not mature enough to achieve specific clinical goals with complete autonomy. Most AI/ML tools in palliative care are customized to fulfill a narrow task with specific goals in place (3). This Fast Fact will outline some of the uses and ethical considerations of AI/ML in the care of patients with serious illness.

 Clinical AI tasks can be broadly defined by whether they directly contact patients.

Direct patient contact / “Front of House” clinical tasks:  These are ways AI directly interfaces with patient care. A key example of AI application is patient education. Multiple large language models (LLMs) offer clinicians communication advice through chatbots, with the potential to directly communicate with patients. These platforms assimilate information from internet language databases to offer suggested verbiage. Evidence is evolving and is not yet robust regarding effectiveness (4,5). While LLMs are trained to generate realistic sounding human text, they are not currently designed to produce accurate information. LLMs are known for confabulating when the requested information is not otherwise available (hallucinations). LLMs have also been shown to perpetuate race-based medicine biases in their responses (6). Any individuals who utilize LLMs to ask medical questions (i.e.: what are the risk-benefits of adjuvant chemotherapy for IIb breast cancer) should be aware of these risks.

Indirect patient contact / “Back of House” clinical tasks:  These are ways AI indirectly interfaces with patient care.  Examples of AI application at the time of writing include: 

• Predicting which patients would benefit from palliative care specialists (7).
• Alerting clinicians to patients at high risk of clinical decline (8).
• Identifying patients with high risk for short-term mortality in palliative care cohorts (9,10).
• Identifying goals of care conversations from electronic health record notes (11,12).
• Generating clinical notes (13).
• Communication skill training (14).

Ethical considerations: 

• Risk assessment: Ethical risk in AI refers to chances for an AI system to deceive a human and/or produce a result leading to human harm. In general, tasks which involve direct patient contact carry greater ethical risk compared to tasks that only involve indirect patient contact.  Tasks of recognizing and responding to emotion, communicating life-altering prognostic news, and giving medical recommendations to a seriously ill patient are of highest risk in hospice and palliative care and should be subject to significant scrutiny. Review by an expert-panel or ethics/institutional review board prior to implementation (if possible) can identify and anticipate ways an AI system can lead to harm and create risk mitigation strategies (15). However, tasks completed by AI via algorithms humans cannot understand, such as flagging a patient as high risk for a given outcome, also have an increased chance of inaccurate output harming a patient due to our inability to fully scrutinize the process. Another risk is AI being trained with data that contains biases and then generating erroneous clinical information, more than what can be reviewed by content experts (6). Finally, some experts are concerned that as AI increases in intelligence and the number of tasks it can perform, there could be catastrophic risk (16). Narrow, singular AI like those above likely bear low catastrophic risk compared with entrusting AI with fuller access to patient data, for example.
• Watermarking: Watermarking refers to visibly labeling AI-generated content. Given that patients can be deceived into believing AI-generated content is human-authored, appropriate labeling is important.
• Data management: Patient data used and generated by AI is at risk of breaches and cyberattacks. The best way to prevent patient harm from this is by deidentifying patient data prior to use in an AI system. If this is impossible, health care institutions should develop a confidentiality, integrity, and availability (CIA) plan with collaborating clinicians. CIA plans address issues such as personnel with data access, encryption, and device security. These plans have been successfully implemented and comprehensive checklists have been developed for various medical specialties (17).
• Data ownership: Data generated by AI is often subject to ownership disputes. AI can collect patient data including voice and biometric data or previously published, copywritten information. This remains the subject of ongoing litigation regarding AI being trained on copyrighted materials such as textbooks, newspapers, and scientific journals (18). Health care institutions in conjunction with legal and ethics teams should develop a data-sharing consent process and ownership plan outlining what health information is shared with AI and whether the patient or medical center own the data.

References 

1. Hosny A, Parmar C, Quackenbush J, et al. Artificial intelligence in radiology. Nat Rev Cancer. 2018;18(8):500-510. doi:10.1038/s41568-018-0016-5
2. Rajkomar A, Dean J, Kohane I. Machine Learning in Medicine. N Engl J Med. 2019;380(14):1347-1358. doi:10.1056/NEJMra1814259.
3. Vu E, Steinmann N, Schröder C, et al. Applications of Machine Learning in Palliative Care: A Systematic Review. Cancers (Basel). 2023;15(5):1596. Published 2023 Mar 4. doi:10.3390/cancers15051596.
4. Ayers JW, Poliak A, Dredze M, et al. Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Intern Med. 2023;183(6):589-596. doi:10.1001/jamainternmed.2023.1838.
5. Burry N, Nakagawa S, Blinderman CD. “You Are Not Alone”: The Allure and Limitations of Artificial Intelligence in Serious Illness Communication. J Palliat Med. 2024;27(1):7-9. doi:10.1089/jpm.2023.0471.
6. Omiye JA, Lester JC, Spichak S, et al. Large language models propagate race-based medicine. NPJ Digit Med. 2023;6(1):195. Published 2023 Oct 20. doi:10.1038/s41746-023-00939-z.
7. Wilson PM, Ramar P, Philpot LM, et al. Effect of an Artificial Intelligence Decision Support Tool on Palliative Care Referral in Hospitalized Patients: A Randomized Clinical Trial. J Pain Symptom Manage. 2023;66(1):24-32. doi:10.1016/j.jpainsymman.2023.02.317.
8. Gallo RJ, Shieh L, Smith M, et al. Effectiveness of an Artificial Intelligence-Enabled Intervention for Detecting Clinical Deterioration. JAMA Intern Med. Published online March 25, 2024. doi:10.1001/jamainternmed.2024.0084
9. Avati A, Jung K, Harman S, et al. Improving palliative care with deep learning. BMC Med Inform Decis Mak. 2018;18(Suppl 4):122. Published 2018 Dec 12. doi:10.1186/s12911-018-0677-8.
10. Zhang H, Li Y, McConnell W. Predicting potential palliative care beneficiaries for health plans: A generalized machine learning pipeline. J Biomed Inform. 2021;123:103922. doi:10.1016/j.jbi.2021.103922.
11. Lee RY, Brumback LC, Lober WB, et al. Identifying Goals of Care Conversations in the Electronic Health Record Using Natural Language Processing and Machine Learning. J Pain Symptom Manage. 2021;61(1):136-142.e2. doi:10.1016/j.jpainsymman.2020.08.024.
12. Forsyth AW, Barzilay R, Hughes KS, et al. Machine Learning Methods to Extract Documentation of Breast Cancer Symptoms From Electronic Health Records. J Pain Symptom Manage. 2018;55(6):1492-1499. doi:10.1016/j.jpainsymman.2018.02.016.
13. Wang J, Yang J, Zhang H, et al. PhenoPad: Building AI enabled note-taking interfaces for patient encounters. NPJ Digit Med. 2022;5(1):12. Published 2022 Jan 27. doi:10.1038/s41746-021-00555-9.
14. Stamer T, Steinhäuser J, Flägel K. Artificial Intelligence Supporting the Training of Communication Skills in the Education of Health Care Professions: Scoping Review. J Med Internet Res. 2023;25:e43311. Published 2023 Jun 19. doi:10.2196/43311.
15. Steimers, A.; Schneider, M. Sources of Risk of AI Systems. International Journal Environmental Research Public Health 2022, 19, 3641. doi: 10.3390/ijerph19063641 
16. Turchin, A., Denkenberger, D. Classification of global catastrophic risks connected with artificial intelligence. AI & Soc 35, 147–163 (2020). https://doi.org/10.1007/s00146-018-0845-5.
17. Chintan Shah, Douglas Nachand, Christoph Wald, et al. Keeping Patient Data Secure in the Age of Radiology Artificial Intelligence: Cybersecurity Considerations and Future Directions,Journal of the American College of Radiology, Volume 20, Issue 9,2023, Pages 828-835, ISSN 1546-1440, https://doi.org/10.1016/j.jacr.2023.06.023.
18. Samuelson P. Generative AI Meets Copyright. Science. July 2023; 381, 158-161 DOI: 10.1126/science.adi065.

Authors’ Affiliations: 1Mayo Clinic, Rochester, Minnesota; 2University of Texas Southwestern Medical Center, Dallas, TX.
Conflicts of Interest:  The authors do not have any conflicts of interest to report.
Version History: originally edited by Sean Marks MD; first electronically published in September 2024.