Why AI ROI Becomes Guesswork Once Systems Scale [PODCAST]

← Back to All Podcasts

Why AI ROI Becomes Guesswork Once Systems Scale

In this episode, Dave Trier, CEO of ModelOp, discusses why AI ROI becomes guesswork once systems scale.

Highlights of this episode include:

• Why there is such a gap between AI investment and AI value
• What metrics to track beyond the typical number of AI models deployed
• How CFOs should think about managing AI as a portfolio
• Industrializing AI delivery and what that means
• How hospital system CFOs can ensure AI risk oversight is actually happening
• How the new phase with generative AI and agentic systems changes the operational complexity for hospital systems
• What will separate hospital systems that successfully scaled AI from those that did not
• One thing this year to improve the ROI of organization’s AI investments

Kelly Wisness: Hi, this is Kelly Wisness. Welcome back to the award-winning Hospital Finance Podcast.  We’re pleased to welcome Dave Trier. Dave serves as CEO of ModelOp, where he leads the company with a clear focus on customer value, product innovation, and enterprise execution. A builder at heart, Dave brings deep technical fluency and real-world operating experience to the challenge of helping global enterprises unlock the transformational power of AI. With more than 20 years of experience spanning data science, AI, analytics, cloud, and enterprise software, and as a named inventor on multiple patents, Dave is known for his pragmatic, transparent leadership style. Prior to becoming CEO, Dave served as ModelOp’s SVP of Product and was foundational in shaping product vision and strategy, working directly with customers to address the operational, delivery, and governance realities of enterprise AI. Before joining ModelOp, Dave held senior technology and business leadership roles across software, consulting, and industry. He served as Vice President of Advanced Analytics Services at Think Big Analytics (acquired by Teradata), where he led a 400-person organization across the Americas. Dave holds a Bachelor of Science in Electrical Engineering from the University of Notre Dame.

In this episode, we’re discussing why AI ROI becomes guesswork once systems scale. Welcome, and thank you for joining us, Dave.

Dave Trier: Thanks for having me, Kelly.

Kelly: All right, well, let’s go ahead and jump in. So, hospital systems have substantially invested into AI over the past few years, yet many CFOs still struggle to see measurable returns. Why is there such a gap between AI investment and AI value?

Dave: Yeah, that’s a great question, Kelly. And I often get it quite extensively. And there’s actually a couple reasons. I guess just first to ground everybody that AI is obviously a new technology. And there is just naturally for large healthcare systems, a bit of a lag between when a new technology is introduced and when you start to see the actual value. And in particular, this one is very nuanced in that the technology itself does have intertwined with it change management. So, I just start with that grounding that AI is a new technology and there’s just a natural lag. However, most of the time what I see with large healthcare organizations is that they jump right into the technology for technology’s sakes, which is not a bad thing, right? It’s very engaging. It offers a lot of opportunity, but they need to step back and really look into a thoughtful analysis of what are the top use cases, what are the top business scenarios, the top processes, areas where we can drive the most value and focus on those first. And then from there, it’s also about, well, how do I then put the right discipline in place to go from, I’ve got this great idea all the way through testing it, testing with different users, whether it’s clinical or back office, and making sure that you have the right approach that works with the existing processes, the existing users, the existing patients potentially as well, in order to actually make the most value out of that particular use case. So again, it really comes down to those couple of fundamental principles, if you will, that we see just time and time again.

Kelly: Yeah, that makes a lot of sense. And I love what you said about focusing on the top use cases. That really resonates with me. So, if a hospital CFO asks you how to determine whether their AI program is actually creating value, what metrics should they track beyond the typical number of models deployed?

Dave: Yeah, that’s a great one. And it happens a lot especially with AI and generative AI, that there’s not exactly a financial metrics tied to it. So, I give some guidance to CFOs and other financial leaders across different organizations, healthcare organizations. Number one, it starts with just usage and making sure that your usage is matching the cost, right? Because there’s a tendency with new technologies, “Oh, let’s just go buy it. It’s a new shiny object.” But is there actual usage behind it? So that’s kind of the fundamental. You can apply that to any particular AI, technology, solution, etc.

Number two, then if you think about these layers of an onion, if you will, number two comes in. What’s the feedback? What’s the feedback you’re getting from customers, from different users? Maybe it’s an agentic system, and you want to have the overseers of these agents to give feedback. Is it meeting the needs? Did it solve the problem? Did it do its job correctly the first time? Or did we have to intervene, in which case maybe it didn’t make sense to use AI in the first place? So that’s number two is getting that feedback and having that as a constant, continuous barometer, if you will, as to whether or not that we are starting to get value.

But then, number three and the most ideal is there direct financial correlation? And I know there’s a lot of buzz around AI around headcount reduction, which is fair in certain parts, but there are other direct financial metrics, reducing time to market for your product, drug discovery, sales, automating back-office tasks that help to save potentially time. And maybe you outsource some processes in the past, you might be able to bring those in-house, if you will. So, there are direct financial correlations around AI that, again, you can start to see and appreciate those benefits over time. So, again, back to your original question, Kelly. It’s really that kind of three-pronged approach of usage, feedback, and then ideally direct financial correlation, which unfortunately is not in every situation.

Kelly: Yeah, no, those metrics make a lot of sense. So, Dave, many organizations treat AI as a collection of experiments rather than a managed portfolio of investments. How should CFOs think about managing AI as a portfolio?

Dave: Yeah, that’s another great question. AI really cannot be thought of as experiments. There’s just too much on the line. There’s too much investment, too much risk around it in certain areas. So, they really need to be thought about and managed with the same discipline that you would manage a portfolio of investments. Yeah, some will fail, some will succeed. But you always approach them with that, like I said, that disciplined process to determine what’s the business benefit? What’s the risk involved? What’s the complexity to manage it? And what’s our ability to deliver, especially from that change management aspect? So, if you think about before you embark on any sort of AI investment, that you’d go and approach it just like you would approach purchasing your portfolio of stocks, that discipline around business versus the costs, and in this case, the risk trade-offs around the usage of AI.

Kelly: I’ve really never thought about managing AI as a portfolio, so that’s very interesting. So, let’s talk about why you call industrializing AI delivery. What does that actually mean from an operational and financial perspective?

Dave: Yeah, that’s a great question. And what we see routinely, we’ve been doing this for over seven years working with obviously large healthcare organizations, but also in other regulated industries as well. And routinely, what we’ve been seeing over the past seven-plus years is that when we come into a large organization, they have a variety of different teams that are developing or buying AI solutions, and each of them are doing it differently. They have manual processes, a lot of ad hoc, a lot of back and forth emails, spreadsheets. And it’s just a repeat of the wheel, a lot of duplicative costs, and unfortunately, a lot of unknown risk that aren’t being managed, unfortunately. So when we talk about this, and really our unique approach and differentiated approach around this, is to turn that cottage industry, those artisanal-type approaches of one-off crafting of an AI solution into something that is repeatable, consistent, regulated, traceable, and ultimately automated so that you produce this industrial-grade approach to how you go from, I’ve got an idea for using AI to actually using it into business. So again, it’s just turning that, like I said, manual ad hoc approach into something that is consistent, repeatable, and of course traceable, especially in the regulated industries.

Kelly: Yeah, no, I never really thought about it that way, but thanks for explaining that for us. So how can hospital system CFOs ensure AI risk oversight is actually happening, not just documented in governance committees?

Dave: Yeah, I see this time and time again that when you see the word governance as it relates to AI, a lot of spreadsheets come to mind. A lot of checklists. I mean, if you want to go back 20 years, you got clipboards, right? Unfortunately, governance gets a bad rap because people think, “Oh, man, it’s just another checklist that I have to do,” if you will. But it’s there, and it’s very important to have this in place because AI is inherently risk-bearing. It’s very tied into data, especially in Gen AI, it can start to hallucinate when starting to produce what look like really, really good answers, but are just flat out wrong, right? So, governance is imperative to be able to do this effectively and safely across, especially healthcare organizations. Now, the keyword that I would say in response to your question is enforcement, is how do you enforce those governance processes consistently and proactively? So, it’s not just an after-the-fact clipboard and checklist, but rather it’s just ingrained. It’s just a part of the process. It’s just part of going from, like I said, AI idea, if you’re developing it, part of your development process, if you’re buying it, it’s part of that procurement and testing process. It’s just ingrained and enforced as they go through what we call the life cycle of an AI solution, from idea through the usage and eventual retirement. So, to answer your question succinctly, I would just say to CFOs, make sure that you have an enforcement policy around it, that you’re able to trace it and track it, and have visibility into what’s happening across the entire organization, regardless of what type of AI is being used..

Kelly: Yeah, that enforcement policy, that makes a ton of sense to me. So, Dave, we’re entering a new phase with generative AI and agentic systems. How does that change the operational complexity for hospital systems?

Dave: Yeah, especially a generative AI and agentic system, there’s really a variety of different areas that you have to now think about. Number one, it starts with just the variety of different technologies themselves, right? So, in the past, if you think about how people used AI and ML, there was kind of some typical approaches that are used. And there may have been some open-source and some proprietary technologies. But now there’s such a variety of new AI technologies, both development-wise, and there are just, it seems like hundreds of companies that start every week related to AI. So, you just have this massive variety that you’re trying to manage. And then you couple that with the pace of change of the technology itself. So not only is you got this wide variety of different AI technologies, but then they themselves are constantly changing. So how do you keep pace with it? And that’s where it introduces, like I said, operational complexity that you have to manage. Then finally, if that wasn’t enough, you got a combination of it’s not just the “AI models themselves,” they’re actually integrated into different applications. They’re affecting or a part of existing processes, like I said, whether they’re clinical, or back office, or supply chain. And lastly, they have humans involved as they should, right? And so, it’s no longer just, I’ve got a model that gives it some data and it predicts an output, but rather you are entwining these, like I said, within existing processes that are involving different teams and users. Again, all adding up to the operational complexity. But don’t worry, right? That’s why companies like ModelOp exist, right, to help you to overcome some of those complexities, to help you to have that consistent approach to how you go and make sure that the AI is going to be working as you would expect, that it’s not going to drive undue or unbearable risks, that you have oversight and visibility into what’s happening overall. So hopefully that gives you a little bit of context on that one.

Kelly: Yeah. I mean, I know things are changing rapidly. So, if we look ahead three to five years, what will separate hospital systems that successfully scaled AI from those that did not?

Dave: Yeah. Again, we’ll just start from more of the business value side of things. You’ll see that there is better patient care, right, because they’re able to use AI to help to inform not only the clinicians and the staff, but also for consumers, patients themselves as well. Second, you’ll help to really take a lot of load off the plates of especially the clinicians and those that work within the hospital organizations because it’s just helping to run some of the routine tasks, some of the menial tasks, ones that you kind of don’t want to do, right? So, you’ll see that there’s a lot of time freed up for patient care as opposed to doing some of the back-office work that nobody, frankly, wants to do. So, I wanted to start in that front.

But for those, as to your question, say, what will separate them is that they’re driving those type of outcomes day in, day out. It’s just part of what they’re doing with AI ingrained into that process. So they’ll just, like I said, take that transformational step to start to look at what are those high value business processes or areas where AI can help to drive some of those types of improvements that I mentioned, and they’ll start to implement those to the point where it’s just, like I said, ingrained in how they do day-to-day operations overall.

Kelly: Who knows what’s going to be ahead in three to five years? But I know things are rapidly changing. They’re going to keep changing, right?

Dave: Absolutely.

Kelly: Yeah. So, if a hospital CFO listening today could do one thing this year to improve the ROI of their organization’s AI investments, what should it be?

Dave: Yeah. Very simply, it’s helped to introduce some of those standards that provide that discipline we talked about upfront. Discipline to have the process industrialized, refined, automated that first checks upfront, “Hey, what’s that business value versus risk, as we mentioned?” and then has that industrialized process to do the checkpoints all along the way. It’s during development, if they’re developing it or buying it, they’re testing it. “Are we seeing the value that we want to see? Great. Let’s let it proceed.” Then you go into a pilot phase. Awesome. We’re again seeing the value, and then we move on to production, and that’s great. You start to use it, and you might get some value or it might change over time. And to the point where, “Oh, our original thesis around this area, it’s actually not returning the value we’d expect.” So, you now can start to, like I said, rationalize that portfolio investments and continue the ones that are driving the value, but eliminate the ones that potentially are not. So, it’s all about establishing those enterprise discipline and standards and helping to move from that one-off ad-hoc approach to something that’s more consistent, industrialized, as we like to say.

Kelly: Right. Well, thank you, Dave, for sharing your insights with us on why AI ROI becomes guesswork once systems scale. And if a listener wants to learn more or contact you to discuss this topic further, how best can they do that?

Dave: Absolutely. At any time, please go to modelop.com, M-O-D-E-L-O-P.com, or reach out to me on LinkedIn. Always happy to have a discussion.

Kelly: Awesome. Thank you for providing that. And thank you all for joining us for this episode of The Hospital Finance Podcast. Until next time…

[music] This concludes today’s episode of The Hospital Finance Podcast. For show notes and additional resources to help you protect and enhance revenue at your hospital, visit besler.holdings/podcasts. The Hospital Finance Podcast is a production of Besler Holdings.

If you have a topic that you’d like us to discuss on The Hospital Finance Podcast or if you’d like to be a guest, drop us a line at contact@besler.holdings.

---

---