A Framework For Deploying AI In Clinical Trials Without The Hype
By John Oncea, Chief Editor, Clinical Tech Leader
Liam Devenish describes himself as something short of an AI skeptic; closer to someone who thinks the field’s current use is “a little bit being hyped, not quite where people think we are yet.” That’s a fair characterization of someone who spent nine months running clinical trials at Wits RHI before turning to AI strategy and healthcare policy research at the London School of Economics. It’s also, in his account, an assessment built on a specific and fairly narrow test he’d apply before any AI tool touches a clinical trial.
The Question That Comes Before Every Other Question
Devenish’s starting point for any technology deployment in clinical research – not AI specifically – is risk. “The first question that needs to be asked, basically whenever you’re deploying a tool in clinical trials or in clinical research, is what is the risk that you’re introducing here and how does it relate to the protocol or the outcome that you’re trying to achieve,” he said. He applies that lens to AI with particular weight because of what he sees as AI’s specific risk profile: “It is to some degree could be unmonitored. It’s black box to some degree as well, depending on the deployment case. And so, it’s difficult to monitor; the quality is difficult to assure.”
That’s not a blanket objection to AI in clinical research. It’s a reason he thinks “general use cases for AI are quite difficult to imagine” and a reason his framework insists on narrowing the use case before anything else.
Gate One: Task-Specific, Not General
“I’m a firm believer in task-specific AI or outcome-focused AI,” Devenish said. The distinction he’s drawing is between a tool asked to do one defined thing well and a general-purpose system deployed loosely across a workflow. “If you have a specific process that you would like to expedite or improve the efficiency of and you develop an AI tool or deploy an AI tool within that sandbox, I think it’s really important.” A system built and evaluated against a narrow task is one whose performance can actually be assessed; a system asked to do everything a little bit is one nobody can meaningfully audit.
Gate Two: Sandboxed Before It’s Trusted
The second gate is process, not just scope. Devenish wants any AI tool run through a development sandbox before it touches live trial operations – a space to “iterate through the process of how this is going to interfere or add to your existing research process” before deployment. His bar for graduating out of the sandbox is specific: the tool has to demonstrate it can “produce the same answers or the same results every time within a sandbox” for a defined task, with transparency about the data it was trained on and the accuracy it achieves in that specific setting. “If it can show that applied to this specific situation, it’s able to achieve that, I’d be happy with that,” he said.
Gate Three: Demonstrated Value, Not Assumed Value
Even a tool that clears the first two gates has to clear a third: does it actually make anything better? “The world of AI is really struggling to demonstrate the business value of AI in a lot of settings,” Devenish said. “Sure, it makes our thinking easier in some settings, but it doesn’t necessarily produce good results.”
His test is concrete and metric-driven: “Is it making your employees more able to apply their time to human-to-human interaction? Is it reducing your error rate in record-keeping? Is it improving your external monitor’s evaluation of the quality of your data? Is it reducing the amount of time spent on integrating information or processes between the different electronic platforms that you’re using?”
Where He’d Actually Deploy It
Run through his own framework, Devenish names three specific use cases he considers genuinely ready today. The first is platform integration: using AI agents to bridge the fragmentation between systems that don’t natively talk to each other, addressing directly the “messy system” problem he describes elsewhere in the interview.
The second is record completion: an AI tool that takes a clinician’s notes or a recorded conversation and populates a case report form, provided the underlying information has already been vetted by a clinical researcher. The third is quality review: using AI to flag inconsistencies between data entries for human follow-up, a task he describes as important but “menial” and well suited to automation.
The Line He Won’t Cross
Devenish draws a firm boundary around clinical judgment itself. “Personally, I don’t think that there’s all that much of a role for the clinical decision-making aspect of clinical trials,” he said, “and where that is biggest is when you’re assessing adverse events or events of interest or primary or secondary outcomes.” Those calls, in his view, require a specific human skill in framing the right question, and one AI shouldn’t be trusted with “for now.”
The organizing principle behind all of it, he said, is that “the goal of AI in clinical research should be to avail more time for clinical researchers to do clinical research activities, as opposed to replacing any of the tasks that they are doing.” It’s a narrower ambition than most AI pitches make for themselves and, in Devenish’s framework, that narrowness is exactly what makes a deployment trustworthy rather than aspirational.
This is a companion piece to Why Younger Clinical Researchers Are More Wary of AI Than Their Elders, which explores a generational pattern in AI adoption that Devenish sees playing out well beyond clinical research.