KPIs To Consider When Evaluating Success With New Site Technology
By Aurea M. Flores, Ph.D., BS Pharm, CCRP, CHRC, CHC, CHPC, CCEP, PMP, PMI-ACP, RQAP-GCP
From a distance, it can be difficult for sponsors to determine how well a site is adapting to a newly introduced technology, particularly one that has been chosen by the sponsor. But first and foremost, the sponsor must determine whether the clinical trial site staff have embraced it. This can be challenging for site staff to do, as new technology can change how they perform their tasks. In some cases, staff will continue to maintain paper records or spreadsheets, duplicating their efforts and creating frustration and work dissatisfaction. To prevent this, site leaders must embrace the change, leading by example and supporting their staff. At the same time, leaders must not enable staff with negative behaviors to create a chaotic environment, while also rewarding positive employees who embrace change, even if there are mistakes or mishaps in the process. After all, there is a learning curve when adopting these technologies.
Once the trial staff begins to embrace the new technology, the sponsor can concentrate on the technology’s performance and how it positively impacts the organization's key performance indicators (KPIs). Before and during the period in which the new technology is implemented, the sponsor should develop and select specific KPIs for the technology. Since there is a learning curve, it is important not to start evaluating KPIs too early. A schedule of three months, six months, and 12 months post-implementation works well, understanding that staff may not be fully competent in or compliant with the technology, but trends can help determine whether the technology is working and the staff is embracing it.
Below is technology-specific advice for evaluating whether a new technology is embraced by site staff and fulfilling its purpose for the sponsor.
Clinical Trial Management System (CTMS)
When it comes to entering the trial design into the CTMS, the process should be straightforward. In particular, CTMS allows the creation of budgets, coverage analysis, and invoices. Part of determining whether this digitalized option is worthwhile is determining how long it takes to create each of these items versus the old process. However, keep in mind that CTMS interfacing with an organization’s EHR is helpful but can be problematic as EHRs are not designed to support clinical trials in the same way CTMS applications are. In my experience, this is one of the most challenging aspects of these projects and the organization needs to determine whether there is added value to pursue this interface.
Other areas to examine are whether the CTMS application improves communication among project stakeholders and reduces the time for trial implementation. A CTMS alone is not responsible for these timelines; it is the staff that carries most of the responsibility. Technology, however, can assist with improving team member job satisfaction by reducing duplication of effort.
eReg/eBinder
For eReg/eBinder, one will want to assess how user-friendly it is in organizing the hundreds, sometimes thousands, of essential documents associated with a trial. Additionally, the application should easily archive as well as retrieve older versions of documents. It should also easily create and maintain logs and similar documents that are 21 CFR Part 11 compliant, in particular, electronic delegation of authority logs.
Users may also decide to use eReg to create and maintain deviation, adverse events, concomitant medication, screening, and enrollment logs. These also need to be user-friendly to create and maintain and be Part 11 compliant. All should be easy to retrieve as needed. Lastly, the application should inform all trial personnel of new document amendments (e.g., protocol, investigator drug brochure, informed consent(s), etc.) once they are received and approved for use.
eSource
eSource is not supposed to replace the EHR or paper medical records. Instead, its purpose is to specifically capture initial clinical trial data. Since eSource applications are not subject to HIPAA regulation, there is more freedom to document because eSource documentation is not available to clinical trial participants to review and/or inspect, which allows for documentation of more sensitive information. Therefore, creating clinical trials in eSource should be user-friendly. Using eSource may add some time to the process. But if eSource is part of a technology suite, it may not take as much time as if it was a stand-alone application, because the CTMS build may migrate into the eSource or vice versa, from the visit build.
Evaluate how easily site staff can complete visit documentation and how easily sponsor staff and inspectors can review it. Current EHRs are not designed to segregate clinical trial documentation from non-clinical trial documentation, and it can be time-consuming for sponsor monitors/auditors and inspectors to locate the needed data. It also leads to unnecessary queries in the EDC. This is where eSource might become a time-saver. Therefore, timelines for completion and review of documentation should be included in the KPIs.
eConsent
The eConsent application selected must support the organization’s process to obtain consent. Since the COVID-19 pandemic, remote consenting has proven to comply with the regulatory intent of providing prospective participants with the necessary information needed to make an informed decision. However, remember that eConsent applications are not simply electronic versions of a paper document, and principal investigator-designated staff are still required to obtain consent. It is important to evaluate the application on how easy it is to write the document, as well as incorporate video, audio, graphics, a glossary of terms, and other features that inform prospective participants.
Time to completion of the informed consent process may be reduced if remote consenting is employed either by phone or video conferencing, but the actual process of informing the prospective individual should not be reduced, however, as it may negatively impact their understanding. Therefore, organization and participant convenience and understanding should be among the KPIs.
Electronic Quality Management System (eQMS)
eQMS has two main purposes. First, it’s a repository for policies, procedures, process maps, instructions, and similar documents that ensure operational efficiency and compliance with applicable laws and regulations. Similar to the eReg/eBinder, the eQMS should allow documents to be easily organized, archived, updated, and retrieved with specific responsibilities and access to these activities. Its second purpose is to support quality activities, i.e., assurance, control, and improvement. Because of this, eQMS should be user-friendly in building internal processes to support quality activities with timelines and milestones, while assigning tasks and responsibilities to the appropriate stakeholders. It is important to evaluate how staff utilizes the technology, how the technology communicates the completion of quality activities to stakeholders, how timelines for completion of assigned activities are achieved, and how the technology supports operational quality.
Electronic Data Capture (EDC)
EDC applications' main users are trial sponsors, but also academic centers, and other for-profit and not-for-profit entities. EDCs should be nimble to support different types of trial design (e.g., randomized, non-randomized, and observational). Furthermore, EDCs should be intuitive, easily build trial case report forms, and be straightforward in verifying the trial build. A top-tier EDC would be able to interface with EHRs to populate EHR-discreet data into the EDC, therefore saving time in data entry and data verification.
Investigational Medicinal Product (IMP) Management Systems
IMP management systems are designed specifically for sites to digitalize drug accountability. They are most valuable for busy sites conducting at least 100 trials. This technology supports the organization’s workflow for clinical trial drug receipt, as well as the dispensing, returning, and destruction of IMP. The technology should assist in ensuring that the site always has drugs on hand by tracking expiration dates and providing resupply alerts. The technology should be flexible to accommodate the site’s Kanban methodology. (Kanban is a scheduling system that uses visual cues to prompt an action, in this case, reordering of drug supplies.) The application should also be able to print and customize dispensing labels compliant with state and federal requirements. An added benefit is that the application can interface with the EHR or eSource to queue participants' order details. Additionally, the application can interface with facility equipment to obtain data from temperature and humidity recorders. Furthermore, the application should be able to generate ALCOA+-compliant documentation in a variety of formats. IMP management systems should be evaluated for paper burden reduction and compliance with regulatory requirements.
Evaluating A New Site-Facing Technology
In summary, the organization must have baseline data from the old process to compare with new processes for the selected KPIs. If the organization does not have baseline data, then it can be collected during the period the technology is implemented. One of the main reasons to explore new technology is to optimize clinical trial operations, quality, and compliance. Implementation of clinical trial technologies can take several months to complete. Integrating the technologies using application programming interfaces (APIs) from different vendors may add more time to the implementation and caution should be exercised, especially when interfacing with current EHR applications. Many of these interfaces can be complex and time-consuming and do not always lead to success.
About The Author:
Aurea Flores is a clinical research professional with over 25 years of experience. She is a licensed pharmacist and was awarded a Ph.D. in pharmacology & toxicology with an emphasis on drug metabolism and chemical carcinogenesis. She has conducted/supported clinical trials in a variety of areas (oncology, cardiology, neurology, gastroenterology, bariatrics, pediatrics, and others) with an emphasis on patient safety, operations, quality, and regulatory compliance. She is a Certified Clinical Research Professional (CCRP-SoCRA), Project Management Professional (PMP-PMI), and Agile Project Manager (PMI-ACP), and is Certified in Healthcare Research Compliance (CHRC-HCCA), Certified in Healthcare Compliance (CHC-HCCA), Certified in Health Privacy Compliance (CHPC-HCCA). She is also a Certified Compliance & Ethics Professional (CCEP-SCCE) and a Registered Quality Assurance Professional in Good Clinical Practice (RQAP-GCP/SQA). She is a speaker at national meetings and consults in areas of clinical research operations, project management, healthcare and research compliance, and quality assurance.