[ad_1]
This conversation is part of a series of interviews in which JAMA Editor in Chief Kirsten Bibbins-Domingo, PhD, MD, MAS, and expert guests explore issues surrounding the rapidly evolving intersection of artificial intelligence (AI) and medicine.
Designing AI tools for clinical use means making choices. Among the most challenging, experts say, is how to develop AI that won’t preserve biases built over generations into the US health care system. It’s an issue the National Academy of Medicine is facing as it works with national leaders to develop an AI Code of Conduct.
One of those leaders is Kedar S. Mate, MD (Video), president and chief executive officer of the Institute for Healthcare Improvement (IHI). Mate is also a general internist and faculty member at the Weill Cornell Medical College. He recently spoke with JAMA Editor in Chief Kirsten Bibbins-Domingo, PhD, MD, MAS, about the importance of incorporating equity into clinical AI tools. The following is an edited version of their conversation.
Dr Bibbins-Domingo:Please tell us a little about IHI.
Dr Mate:The Institute for Healthcare Improvement has focused on improving quality, patient safety and experience, value, and increasingly, health equity for populations all over the world. It was founded by Dr Donald Berwick, who many in your audience know as the former administrator for the Centers for Medicare & Medicaid Services.
Dr Bibbins-Domingo:When I think of IHI, I think of the triple aim: the focus on the experience of care, improving the health of populations, and doing that at a lower cost per person. But that triple aim—you’ve written about expanding that out and recently wrote in JAMA about the quintuple aim. Tell us about the IHI framework for thinking about quality improvement and particularly your push toward equity.
Dr Mate:When we originally thought about the triple aim in 2017 or 2018, we were looking at what was broadly thought of as issues in American health care that were intentioned—the idea that creating better outcomes or better quality often came at greater expense or greater cost and might also compromise access to care.
In fact, that was how I think American health care conventionally thought of it, that quality, cost, and access were in competition with each other. And the real revelation was to say, “No, these things are not necessarily competing with one another, but in fact, we can make care better, safer, the care experience better, and we can do so at lower cost.”
The general thesis was better quality would result in lower cost of care, not the opposite. And so that was the original contribution of the triple aim. Very quickly, people wanted to add additional aims to the 3-part aim. The first one that got added was the idea of workforce joy: well-being, meaning, purpose, and safety of the workforce.
The idea was that if we could get to those big-picture goals around quality, safety, effectiveness, and better outcomes through an equitable path, that that would be more sustainable, more durable, and in fact much more achievable.
So those were the reasons why we added equity as a fifth dimension to the quadruple aim. And now we talk about this as the quintuple aim or the 5-part aim for achieving better health system performance.
Dr Bibbins-Domingo:I like your reminder that the original intention was that these goals are not in conflict with one another. But equity is a harder piece to add to it.
Dr Mate:If we don’t put equity right into the design of our health systems to begin with, we leave it as an afterthought. We leave it as a side consideration. And when we turn our attention to AI, I think that’s going to be a foundational question for us to think about as we start to see these new technologies enter our ecosystem.
Will we pay attention to the equity considerations right from the get-go, or are we going to try to add them afterward? I can promise you that if we try to do it later on, it’s going be a lot harder than if we start with the premise that in fact we can actually build these systems to be more equitable to begin with.
Dr Bibbins-Domingo:One of the things that has been most compelling in these conversations I’ve been having is people who are talking about AI as the ability to really scale. That if we do it well, we actually are talking about increasing access and reducing variability. The optimistic point of view is that it could actually scale. As somebody who’s thought about scaling improvements, how do you think about this technology—which is clearly going to shape what we do in clinical practice—through your lens of improving health systems?
Dr Mate:There’s almost no question that lots of people are not getting the kind of quality care that they could be otherwise receiving, and that AI-enabled solutions or AI-enabled technologies might allow us to actually reach a number of places that we haven’t been able to reach before. So think about rural, remote, underserved communities—not only in the United States, but also around the world in middle- and low-income countries.
AI-enabled image reading or pathologists could do a lot of good. I think right now already with what we have in those kinds of contexts—anything that’s highly dependent on imagery or repeatable large data sets—can or may ultimately be done better, more efficiently, and in a more timely fashion by AI.
There’s also the notion of timeliness of care, the speed with which these technologies are operating. You can experience that for yourself by using ChatGPT or one of the other chatbots out there and typing in a series of questions. Within seconds you’re getting volumes of information back. Imagine that in a clinical environment that hasn’t had access to a specialist—being able to ask tools like that for knowledge and information just as a starting point to aid clinical decision-making.
That will really help us in the coming years to begin distributing the kinds of knowledge and expertise that we’re looking for. IHI has a leadership alliance and one of our redesign principles is the idea of moving knowledge, not people. I think AI allows us to move knowledge at massive scale and it reduces the potential waste of moving people or infrastructure around, which would be far more cumbersome and costly to do. This idea of moving knowledge not people is put on steroids with AI technologies and tools.
Dr Bibbins-Domingo:So it sounds like you’re optimistic. How do you see that path forward, especially as somebody who’s championed the principles of quality and equity?
Dr Mate:There are lots of opportunities around AI and making care safer, better, higher quality. And of course, there are some risks that we’re going to have to manage and mitigate. One way of thinking about this is to consider the dimensions of quality. The IOM [Institute of Medicine], now the National Academy of Medicine, has defined quality with those 6 dimensions [safe, effective, patient-centered, timely, efficient, and equitable], which are probably well understood by JAMA’s audience.
We could go through each of those and talk about how AI is going to potentially have an effect on the quality of care that we might imagine receiving. For safety, I think the jury is very much still out. This is still pretty early days, even though there’s a lot in the news and a lot of the literature is coming out very rapidly around AI. In JAMA, an article by a colleague and friend, Hardeep Singh, talked about the possibility of AI helping us with diagnostic challenges, diagnostic error, and diagnostic failure. A diagnostic delay is a massive problem in health care.
I don’t even know that we fully appreciate its scope. IHI has a safety think tank called the Lucian Leape Institute, named after Lucian Leape, one of the founding fathers of patient safety in the United States. That institute has decided to tackle this problem of trying to understand what the risks and opportunities of AI might be with regard to patient safety.
On the opportunity side, people think of better handoffs, better communication, information not falling through the cracks, fewer missed details, better differential diagnoses—maybe for the first time the opportunity to fundamentally eliminate drug-drug interaction problems or adverse drug events around drug-drug interactions. There’s some really interesting work that’s been going on for a while now on sepsis, still a primary cause of death in hospitals.
But work that Suchi Saria and Bayesian Health have done on early warning scores for sepsis is pretty compelling in prospective studies, large studies, 600 000 patients. They saw reductions in mortality from sepsis by using an algorithm to help predict sepsis, anticipate it earlier, and having that verified by clinicians. So the AI is not operating on its own. It’s still working with clinicians to help power their work.
So on the opportunity side, there’s a lot of promise in AI; on the risk side, there’s a lot there too. I think probably one of the biggest risks is complacency. We may find that as it gets better and better, on the one hand we want to trust it more but on the other hand, there’s some risk of us in some ways losing the clinical acumen and skills as we increasingly trust the AI. And then of course, there’s risk of the AI getting it wrong, which at this time it gets it wrong relatively often in common circumstances. And then the final one, and probably the biggest one, is the possibility of introducing bias.
Especially when it comes to this question around equity, I think the possibility of bias is enormous. The training sets—the way that we build these models and how we train them—if they’re built off of existing ways in which we work, existing ways in which our societies and our medical systems are structured, there’s a great risk of it introducing or perpetuating the biases that we’ve been experiencing as a system for generations now and in some ways for hundreds of years. We have to deliberately design that out of AI. That’s a really important part of how we’re going to succeed if we’re going to actually build AIs that are mindful of health equity in the future.
Dr Bibbins-Domingo:When I’ve talked with radiologists who’ve been using systems that are enabled by artificial intelligence, they also describe a little bit of that in some areas—they know better so they’re going to bypass this. From your vantage point, is this technology substantially different than those common ways that clinicians get used to ignoring or avoiding certain technologies that are designed to help us do our jobs better? Is there some reason to believe we’re in a different era now, or is this just more of the same?
Dr Mate:This is a both-and answer to this question. I think it is a little bit more of the same. But there is something quite different about this technology. So you’re right, we’ve had what might be described as analog algorithms for as long as I can remember. They’ve actually made care less safe for specific populations over time, even in the in the digital era—let’s just say pregenerative AI, large language model AI.
In the digital era, we’ve also had algorithms that have been demonstrably biased. The paper by Ziad Obermeyer about how usage of digital algorithms on hundreds of millions of patients systematically underestimated risk among complex and Black patients—because the algorithms were trained on cost data and because of less utilization of health services by those populations, they underestimated the challenge and complexity of risk in those patients.
The difference, I think, between what we’re seeing with generative AI and large language models is in the way in which it expresses itself. So many of our CDS [clinical decision support] tools are explicitly designed to pop up in a window or provide an alert and say, “Hey, pay attention to me, there’s a potential risk of something bad happening here.” The way generative AI talks to you is qualitatively different. It speaks to you with authority. It reaches a conclusion that’s a bit more definitive. The confidence with which it asserts its view, whatever that view may be, it feels different and it also feels enormously useful because of that.
And so you see this sort of bottom-up adoption of generative AI tools, the chatbots, ChatGPT, and other things in part because it feels immediately useful because it’s solving our immediate problem. I think it’s probably best to recognize that the AI tools we have right now, at least for now, are adjunct tools at best. Today they’re capable of perhaps supporting human clinical reasoning and shared decision-making processes that occur between patients and clinicians. But they’re not a replacement for human clinical reasoning and judgment, at least at this time.
So I think that’s the differentiator, this qualitative way in which AI positions itself. But again, there is a challenge around this. We’re going to see that as the training models get deeper and richer, smarter, as more and more training data are entered into it, as the AI gets more and more sophisticated and the error rate declines to fractions of a percentage, our confidence could and probably should grow in the AI tools that are coming into our clinical practice environments.
Dr Bibbins-Domingo:What I hear you saying is that we’re in a rapidly evolving time when the authoritative voice used by many of these tools is both what makes them useful cognitively in the way clinicians think, but also that the risk right now is they are not quite as accurate or designed for all of the uses where we’ve tried to apply them.
Dr Mate:Right. But therein lies one of the many potential ways of mitigating the risk, which is the idea of putting a clear signal on the AI conclusion that says this is an AI conclusion. And by the way, it was created in this way—transparently understanding the inputs into it in a way that we haven’t had, for example, for the VBAC [vaginal birth after cesarean] algorithm or the GFR [glomerular filtration rate] algorithm.
It’s not that that wasn’t available to us; it’s in the literature. You can dig through it and find it, but it was hard to find and for the most part we forgot that history, unfortunately. But we can be a lot better about this in the future. We can say, “This is the algorithm, this is how it’s composed, these are the elements, and this is the training data that was utilized.”
With that level of transparency, the relative value of the AI tool becomes more apparent as it affects our clinical decision-making. And then we leave it to the human clinician and the patient, both parties in the therapeutic dyad, to navigate how useful that recommendation or suggestion or idea set from the AI is. I think that’s how we start to move forward toward a future in which AI is a part of the encounter, but it’s not making all the decisions on our behalf.
Dr Bibbins-Domingo:So the National Academy of Medicine has convened a group of which you are part to develop an AI code of conduct.
Dr Mate:It’s not the only code of conduct or set of principles that is being written. So I don’t want to overstate what it’s going to end up being. I do think it’s going to try to establish or articulate a set of principles, guidelines, and maybe some guardrails, depending on where we end up in our deliberations.
The group is keenly aware of how fast-moving the field of AI really is. Anything we put out is out of date within weeks, so I think the desire is to move what we know out into the world sooner than later. AI is not yet in a position to in any way replace human clinical judgment or shared decision-making, but the idea is that clinicians who use AI and systems that use AI tools will likely have a decisive advantage over those that don’t. And I think those 2 things can live in tension. You know, we’re not ready to replace clinical reasoning, but as an adjunct or support, almost surely it will present those that can use AI successfully with advantages. The other thing is that patients who use AI, there’s actually significant benefits, too.
Dr Bibbins-Domingo:It’s an important point you’re raising, that with the technology that allows scale and accessibility—that this means the accessibility of knowledge for patients as well. I think it’s at that patient interface where it’ll also radically change what happens in the clinical encounter. But all of this is still a work in progress, so what are you thinking about as the most important thing in the next year?
Dr Mate:I’m really eager to see clinical application, more training of AI models or algorithms on clinically relevant information. I know that there are efforts underway with the Mayo Clinic and Google and Microsoft to try to build clinically relevant AI tools. I’d like to see more of that and then I’d like to see us actually putting those into practice and to try to actually solve real clinical challenges.
We have the capability to make sepsis, or at least morbidity and mortality from sepsis, a relatively rare event with the applications of these tools. It would be exciting to see that proven in the literature and through demonstration efforts. So that’s the kind of real application I’d like to start seeing in clinical practice.
Going back to the bias concern and the equity considerations—it may very well be that when the history is written of this time, we’ll think of this like the birth of the internet.
But AI can’t change history. It actually learns or studies history, and from that creates new conclusions. But because it learns from history, it depends on choices about what we tell it is history. This gets back to the bias and equity question because we have choices about what we believe is a true history, a real history. What we feed the AI essentially as training information is a deeply political or social choice. That’s what it really is—it’s a choice.
Furthermore, I’m curious about whether history alone is really sufficient to describe our future. Because an early-stage AI that was analyzing whether, for example, women can be president, is trained on historical data, which suggests it’s not possible, which is obviously not true for our future. And so I think this notion that AI is neutral or some sort of tool that is apolitical is not true.
It depends on the choices we make about what we tell it to learn from. And that is intrinsically a political or social or a cultural choice. And as a result, we have the opportunity to, in some ways, help AI create for us a better future or allow it to perpetuate the challenges, problems, inequities, and structural failures of our past and present. That’s not a short-term consideration but is perhaps more of a philosophical consideration of how we will help to build the AI tools of our future.
Dr Bibbins-Domingo:You’re reminding me of a piece that I saw about prompts for images in global health. The images that it generated, even when the prompts were specifically designed with a more equitable framing in mind, continued to have a White doctor in an African village. That was the image on a global health stage.
Dr Mate:We’ve got to find a way, and this is not easy, to ensure that training data are free of bias; doing the hard work of sampling the training data so that we understand what biases might be present and then to eliminate it. In the case of health care, maybe oversampling specific communities that are disproportionately affected by a particular issue or condition. And also conducting the kind of postmarket surveillance of algorithms so when we deploy something, when an algorithm becomes adopted, we should verify that it’s producing the kind of equitable improvements that we hope it might. And if it doesn’t, retooling the algorithm so that it does and not being afraid to do that as a community of scientists and scholars.
Dr Bibbins-Domingo:I know at JAMA we are trying to apply in many ways the same lens that we usually do to new advances. We want to see the science, to see it in clinical settings, to see the outcomes, and then to interrogate it across the principles that are important—in this case equity.
Dr Mate:I was really excited to see the call for papers that JAMA issued. And I appreciate this balance of sensitivity to speed as the field evolves so fast but also a focus on sound science. I think that’s just a tension we’re all going to have to live with, wanting to make sure that we actually produce good science but also respecting that the field is advancing at breakneck speed at the moment.
Published Online: October 11, 2023. doi:10.1001/jama.2023.19293
Conflict of Interest Disclosures: Dr Mate reported being a member of the scientific advisory board for Spindletop Capital Management, an advisor for Blue Heron Capital, and on the Board of Directors of HIVE Networks.
[ad_2]
Source link
