Managing risk and resilience in autonomous and intelligent systems: Exploring safety in the development, deployment, and use of artificial intelligence in healthcare
2024 (English)In: Risk Analysis, ISSN 0272-4332, E-ISSN 1539-6924Article in journal (Refereed) Epub ahead of print
Abstract [en]
Autonomous and intelligent systems (AIS) are being developed and deployed across a wide range of sectors and encompass a variety of technologies designed to engage in different forms of independent reasoning and self-directed behavior. These technologies may bring considerable benefits to society but also pose a range of risk management challenges, particularly when deployed in safety-critical sectors where complex interactions between human, social, and technical processes underpin safety and resilience. Healthcare is one safety-critical sector at the forefront of efforts to develop and deploy intelligent technologies, such as through artificial intelligence (AI) systems intended to automate key aspects of healthcare tasks such as reading medical images to identify signs of pathology. This article develops a qualitative analysis of the sociotechnical sources of risk and resilience associated with the development, deployment, and use of AI in healthcare, drawing on 40 in-depth interviews with participants involved in the development, management, and regulation of AI. Qualitative template analysis is used to examine sociotechnical sources of risk and resilience, drawing on and elaborating Macrae's (2022, Risk Analysis, 42(9), 1999–2025) SOTEC framework that integrates structural, organizational, technological, epistemic, and cultural sources of risk in AIS. This analysis explores an array of sociotechnical sources of risk associated with the development, deployment, and use of AI in healthcare and identifies an array of sociotechnical patterns of resilience that may counter those risks. In doing so, the SOTEC framework is elaborated and translated to define key sources of both risk and resilience in AIS. © 2024 The Authors. Risk Analysis published by Wiley Periodicals LLC on behalf of Society for Risk Analysis.
Place, publisher, year, edition, pages
Hoboken, NJ: John Wiley & Sons, 2024.
Keywords [en]
artificial intelligence, resilience, risk regulation, sociotechnical risk
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:hh:diva-52589DOI: 10.1111/risa.14273ISI: 001145922000001PubMedID: 38246857Scopus ID: 2-s2.0-85182845452OAI: oai:DiVA.org:hh-52589DiVA, id: diva2:1836364
Funder
Wellcome trustVinnovaKnowledge Foundation
Note
Funding: This research was funded in whole, or in part, by the Wellcome Trust [Grant number 213632/Z/18/Z]. This work was also supported by the Swedish Innovation Agency and the Knowledge Foundation through grants to Professor Jens Nygren and Professor Petra Svedberg.
2024-02-082024-02-082024-02-08Bibliographically approved