What Is Shadow AI, And How Does It Impact Clinical Trial Tech?

By Carl Carpenter, Arrakis Consulting

Clinical trials are rapidly adopting wearables, sensors, eCOA platforms, connected medical devices, and digital biomarkers to capture real-world, high-frequency data. At the same time, teams are using AI to move faster summarizing device logs, drafting patient instructions, cleaning data sets, and triaging support tickets.

That combination creates a growing risk: shadow AI.

Shadow AI is the use of AI tools, features, plug-ins, or copilots that have not been approved, configured, or monitored by the organization. In clinical trial technology, shadow AI can quietly introduce:

• Data exfiltration: participant data leaving controlled environments
• Trial integrity risk: incorrect summaries, missed signals, biased decisions
• Regulatory and quality risk: weak audit trails, uncontrolled changes
• Device and vendor risk: unaudited AI features in platforms and firmware

This article provides an overview of where shadow AI shows up in clinical trial tech and a practical control plan that protects participants and data without slowing innovation.

Why Wearables And Connected Devices Raise The Stakes

Wearables and connected devices change the nature of clinical trial data in the following ways:

• Volume and granularity: Continuous streams of health data (heart rate, sleep, activity, ECG, glucose, SpO2, etc.) create large data sets that teams are tempted to summarize with AI.
• Sensitivity: Even when names are removed, high-frequency time-series data can increase re-identification risk when combined with dates, locations, or rare conditions.
• Operational complexity: Device provisioning, pairing, firmware updates, app versions, and data pipelines introduce many points where an unapproved AI tool can touch data.
• Multiparty ecosystems: Sponsors, CROs, sites, device manufacturers, app vendors, cloud hosting providers, and analytics partners all handle pieces of the workflow.

In other words: more data, more systems, more vendors, and more pressure to automate.

Where Shadow AI Shows Up In Clinical Trial Tech

Shadow AI rarely looks like a rogue model. Instead, it usually manifests as everyday productivity shortcuts. Consider the following AI-assisted improvements that can increase risks:

1. Device data triage and summarization

Teams often paste device logs, error codes, and participant-reported issues into public AI tools to identify root cause, draft troubleshooting steps, and summarize anomalies. The risk is that logs can contain participant IDs, timestamps, geolocation hints, device serial numbers, and app metadata — all sensitive in aggregate.

2. Digital biomarker analytics and AI-assisted data cleaning

Data managers and analysts may use unapproved AI tools to generate code (Python/R/SAS) for data cleaning and feature extraction, interpret outliers or missing patterns, and draft rationale for exclusions. The risk is that uploading data sets (even partial) can leak sensitive trial data and create integrity issues if AI-generated code is not validated.

3. Patient-facing communications and support

Support teams may use AI to draft device setup instructions, participant emails/texts, and call scripts for troubleshooting. The risk is that prompts may include PHI/PII, visit schedules, adverse event context, or other information that should never leave controlled systems.

4. Vendor platforms with embedded AI features

Many eCOA, ePRO, eConsent, device management, and analytics platforms are adding AI features (auto-summaries, smart search, anomaly detection, chatbot support). However, these features should first undergo a formal review, focusing on:

• data retention and training policies,
• model transparency and monitoring,
• access controls and logging, and
• cross-border data processing.

5. Connectors and plug-ins (the high-risk accelerant)

The highest-risk pattern is not a stand-alone chatbot but an AI tool connected to cloud storage, ticketing systems, email, device dashboards, and data lakes. The risk here is that one overly permissive OAuth grant or compromised account can become a high-speed exfiltration channel.

Additionally, key risk categories specific to wearables and devices include:

1. Re-identification risk from anonymized sensor data

Wearable data can be uniquely identifying when combined with timestamps and visit windows, rare disease patterns, location or activity signatures, and/or device identifiers. Shadow AI increases the chance that de-identified data is shared externally in a way that enables re-identification.

2. Trial integrity and safety signal risk

AI-generated summaries can be plausible but wrong. In device-heavy trials, that can lead to missed adherence issues (device not worn, syncing failures), incorrect interpretation of anomalies, and/or delayed escalation of potential safety signals.

If AI is used to triage or summarize, the organization must define clear human review requirements and escalation thresholds.

3. Quality system and audit trail gaps

Clinical trial tech must be defensible: who changed what, when, and why. Shadow AI creates gaps in which prompts and outputs may not be retained, decisions may be influenced by untraceable AI suggestions, AI-generated code may be used without validation, and/or device/app version changes may be explained with AI-generated text that lacks evidence. Even if AI is only used for drafting, regulated documentation still requires controlled review and approval.

4. Device life cycle and change control risk

Wearables and connected devices evolve quickly due to firmware updates, mobile app updates, and algorithm updates (signal processing, artifact removal). Shadow AI increases risk by introducing uncontrolled analysis methods or undocumented transformations that affect endpoints.

5. Vendor and supply chain risk

Device ecosystems rely on third parties. Shadow AI increases risk when vendors use participant data to improve their models without clear contractual limits, subcontractors handle support tickets containing sensitive data, and/or AI features are introduced mid-study without appropriate notification, testing, or validation.

Practical Control Plan (Without Slowing Device-Enabled Trials)

The goal is not to ban AI, because bans tend to push usage underground. Rather, the goal is to make approved AI the easiest option. To do so:

1. Define a “Never in AI” list for device-enabled trials. Keep it short and nonnegotiable:

• Participant identifiers and direct PHI/PII
• Device IDs/serial numbers tied to participants
• Raw device logs with timestamps and unique identifiers
• Data sets used for endpoint derivation or digital biomarker development
• Protocol/SAP language related to device endpoints (pre-publication)

2. Approve a small set of AI tools and configure them. Select 12 sanctioned tools and lock them down:

• SSO (Single Sign-On)/MFA (Multifactor Authentication) and role-based access
• Disable training/retention where possible
• Restrict connectors by default
• Centralize logging and usage monitoring

3. Provide safe workflows for common device tasks. Give teams templates that reduce the urge to improvise:

• Redaction templates for logs (remove IDs, timestamps, locations)
• Approved prompts (summarize without adding facts)
• A secure internal knowledge base for troubleshooting
• A defined escalation path for suspected safety signals

4. Treat AI-generated code and analytics as controlled changes. If AI is used to generate code for cleaning or feature extraction:

• Require peer review and testing
• Version control everything
• Document rationale for transformations
• Validate outputs against known test cases

5. Strengthen vendor governance for AI-enabled trial tech. Add AI-specific questions to vendor due diligence:

• Is trial data used for model training? Under what settings?
• Where is data processed and stored?
• What logs and audit trails are available?
• How are model changes communicated?
• What subcontractors have access?

6. Monitor behavior, not just tools. Detect patterns that indicate shadow AI:

• Uploads of log files, CSVs, and PDFs to unknown domains
• New browser extensions and unapproved AI sites
• OAuth grants to unfamiliar apps
• Unusual downloads from device dashboards or data lakes

Bottom Line

Wearables and other clinical trial technology can make trials faster, more patient-friendly, and more data-rich. But they also create a perfect environment for shadow AI: high-volume data, complex pipelines, and teams under pressure to troubleshoot and summarize quickly.

Organizations that act now can reduce risk without slowing delivery:

• Keep sensitive device data out of unapproved AI.
• Provide secure, approved AI workflows for troubleshooting and drafting.
• Validate AI-generated analytics and code.
• Enforce vendor governance and change control.
• Monitor usage patterns and train teams with real scenarios.

About The Author:

Carl Carpenter operates Arrakis Consulting, a full-service cybersecurity firm that specializes in helping clients of all sizes get, and stay, regulatory compliant.