Health experts say harder evidence is needed as clinician-facing artificial intelligence rolls out.
Health sector executives are signing contracts for AI scribes, automated patient message replies and radiology-reading algorithms, but many concede they cannot yet say with confidence whether the technology will pay off.
That dilemma sits at the heart of a new viewpoint published this week in the journal JAMA Internal Medicine, which urges a fresh look at how health systems measure the return on investment for clinician-facing artificial intelligence tools.
“Early evaluations of these technologies have largely assessed the accuracy of their outputs and their impact on clinician time, experience, and decision-making, with mixed results,” the authors wrote.
“These results have left health care system leaders unmoored as they struggle to determine which tools are worth the cost – in money, time, and political capital. These trade-offs are often framed in terms of return on investment (ROI).”
Clinician-facing AI is spreading quickly through health systems, promising to cut administrative burden, streamline workflows and sharpen clinical decision-making.
From drafting responses in the electronic health record to summarising complex charts and flagging abnormalities on imaging, the tools are already reshaping day-to-day practice.
Yet as adoption accelerate, leaders are confronting a more complex challenge than accuracy or speed – how to measure ROI in a field where benefits are evolving, evidence is mixed and many gains are hard to quantify, the authors said.
Early evaluations have largely assessed output accuracy and time savings, with inconsistent results.
Some show modest reductions in documentation time or workload, while others reveal only limited measurable impact.
For decision-makers balancing financial cost, workforce pressures and institutional risk, those findings offered only partial guidance, they said.
“Quantitative evidence may take even longer to generate for the more difficult-to-quantify costs and benefits that are nevertheless critical in health care, such as enhanced patient experience, reduced risk of malpractice, or benefits to the reputation of an institution,” the authors wrote.
“Second, in contrast to many technologies for which ROI calculations are conducted, AI technologies are constantly evolving.
“As they do, their benefits and costs may change. Finally, while traditional ROI frameworks assume that benefits and costs accrue to the same party, in health care, there may be disconnects between payer and beneficiary.
“For example, while benefits may accrue to multiple parties (health care system, clinicians, and patients) due to implementation of a clinician-facing AI technology, the health care system will solely bear the costs.”
The authors said that health systems must take a broader, tailored approach that weighs both measurable and less tangible benefits from the purchaser’s perspective and that of key stakeholders.
Quantifiable gains might include increased revenue from clinician productivity, enhanced quality incentive payments or reduced turnover in a tight labour market.
Harder-to-measure benefits also deserve attention they said.
Faster response times to patient messages may boost satisfaction and retention, while improved diagnostic accuracy could reduce adverse events and legal risk.
Costs extend beyond licence fees to include one-off expenses such as EHR integration, training and change management, with ongoing costs tied to licensing, maintenance and monitoring.
Longer-term risks, including clinician deskilling, workflow disruption, governance burdens or privacy breaches, may emerge over time and erode anticipated gains, the authors warned.
They noted that payment models influence ROI priorities. For example, value-based organisations may favour tools that reduce unnecessary care, while fee-for-service providers might prioritise throughput and access.
Some health systems may also invest early in tools like AI scribes, not solely for immediate return but to build clinician confidence and infrastructure for more advanced applications.
“We believe that AI offers many opportunities to improve care delivery and enhance clinicians’ daily work,” the authors concluded.
“However, health systems will need to make difficult choices regarding which tools to invest in and adopt. In doing so, they should systematically consider benefits and costs and the time frame over which these can be realised.
“Vendors that can help health systems assess ROI with credible data and examples are likely to be advantaged in an increasingly competitive market.”
