How To Choose Fit-For-Purpose Tech With The Modified Delphi Method
By Aurea M. Flores, Ph.D., BS Pharm, CCRP, CHRC, CHC, CHPC, CCEP, PMP, PMI-ACP, RQAP-GCP
Clinical research sites and sponsors do not need to adopt all the electronic applications available in the market. In particular, sites have limited budgets, so it is important to determine which applications will provide the most value.
United States clinical research laws and regulations state their intent but seldom state how to fulfill such intent. For example, informed consent regulations laid out by the FDA state that the investigator must obtain consent from the individual before involving them in research. It states what information is to be presented, but not how such information is to be presented. It is then left to the investigator to decide the best way to present such information. Therefore, the investigator site needs to have a clear understanding of its operational procedures designed to comply with all the applicable laws and regulations. My experience working with different sites over the last 20-plus years is that most do not have clear operating procedures.
An absence of SOPs is often attributed to a site’s lack of clear understanding of laws and regulations, in particular, the intent of such regulations. Once the site has a clear understanding of its processes, it can then ensure that technology supports the existing process. It is not in the best interest of the site to change/adapt its processes to those provided by an application. Change is never easy, and going from paper to an electronic application is a big change. Drastically changing existing processes, if they are in place, and especially if those processes are working, can lead to organizational chaos.
Therefore, a site should map all of the organizational procedures supporting its compliance with applicable laws and regulations. Drawing the process on a piece of paper is acceptable — no need for fancy applications. However, if the organization has access to such applications, then immortalize the process and verify that these are the processes that everyone in the organization is following before going through the painstaking process of selecting an application.
Then, when it comes time to dig in and select a tech to move forward with, I recommend teams employ the modified Delphi Method. The modified Delphi method is a forecasting process used to arrive at a group opinion or decision — a consensus. Additionally, it combines the benefits of expert analysis with elements of the wisdom of a group.
Researching, Identifying, And Vetting Potential Vendors
Once clinical trial operation processes are defined and verified, and it’s determined which general applications are to be pursued, the vetting process can begin. A list of all vendors providing the desired applications must be made and prescreened, while simultaneously selecting the vetting team. The vetting team membership may vary depending on the application, but it should always include end users. Organization executives should also be part of the team to ensure an understanding of the objectives and offerings, but the vetting team should not be the final decision-makers.
First, initiate a prescreening process that ensures the vendors are compliant with applicable laws and regulations, such as audit trails, electronic signatures, privacy/confidentiality, and documentation retention, among others.
Then, create a matrix listing all the must-have features for each vendor with a grading system (fig. 1). I recommend using a grading system of 0-5, with 5 being perfect for the organization’s needs and 0 being not at all supportive of the organization’s needs. I also recommend that teams add a final list of features the team was not looking for but may be nice to have.
Next, request demos from each of the prescreened vendors. Leave at least two weeks in between demos to allow time for vetting team members to play in the sandbox, which should be requested at the time of prescreening. During those two weeks, encourage members to explore the sandbox and use the matrix to grade each vendor’s application.
Once all applications have been reviewed, bring the vetting team members together to review the findings. The members should have completed the review matrix as best as they can. The next step is utilizing the modified Delphi method.
Conducting A Modified Delphi Method Session For Consensus
First, bring the vetting team members together in a room, with everyone seated around a table. Provide each member with a stack of sticky notes and a marker. Start at the top of the list of features in the matrix and ask each member to write their corresponding grade, from 0 to 5, on a sticky note without allowing their neighbor to see what they wrote.
In unison, ask each member to stick the note to their forehead. If the grading difference is less than or equal to one point among all the votes (e.g., some members voted 3, others voted 4), then take the grade with the most votes. However, if the differences are greater than two points, then open the feature for discussion. The person leading the discussion should be someone knowledgeable in clinical trial regulations (how to interpret and operationalize), operations, and whatever tool is being evaluated for use. The purpose is to come up with a consensus and then move on to the next feature. It is up to the project sponsor/facilitator to ensure these decisions are considered and made. Ultimately, the selected technology must work for all stakeholders, and compromises must be made between the must-have and nice-to-have features.
Once all features have been reviewed, tabulate the results. The vendor with the highest score is the one selected to move forward. Sometimes, there may be more than one finalist. In these cases, focus on the features that are creating the inability to select a clear winner. The team may go to the sandbox to explore these features or, if necessary, a demo targeted to these features may be requested from the vendor. The purpose of each exercise is to finalize the vendor selection.
Once a vendor is selected, it is important to share with the vendor the applicable process maps, a visual representation of a specific process, that the application(s) are to support to ensure alignment between client needs and vendor and request implementation timelines and client resources required to complete the project.
About The Author:
Aurea Flores is a clinical research professional with over 25 years of experience. She is a licensed pharmacist and was awarded a Ph.D. in pharmacology & toxicology with an emphasis on drug metabolism and chemical carcinogenesis. She has conducted/supported clinical trials in a variety of areas (oncology, cardiology, neurology, gastroenterology, bariatrics, pediatrics, and others) with an emphasis on patient safety, operations, quality, and regulatory compliance. She is a Certified Clinical Research Professional (CCRP-SoCRA), Project Management Professional (PMP-PMI), and Agile Project Manager (PMI-ACP), and is Certified in Healthcare Research Compliance (CHRC-HCCA), Certified in Healthcare Compliance (CHC-HCCA), Certified in Health Privacy Compliance (CHPC-HCCA). She is also a Certified Compliance & Ethics Professional (CCEP-SCCE) and a Registered Quality Assurance Professional in Good Clinical Practice (RQAP-GCP/SQA). She is a speaker at national meetings and consults in areas of clinical research operations, project management, healthcare and research compliance, and quality assurance.