Research Design Mistakes Even Experienced Analysts Still Make

By John Oncea, Chief Editor, Clinical Tech Leader

From 2016 to 2023, Natasa Rajicic, ScD, taught research design in Columbia University’s Applied Analytics master’s program, a part-time lecturer role she took alongside her full-time biostatistics career at Pfizer, then Cytel, then her own consulting practice. Her students weren’t training to become biostatisticians. “My most recent experience was research design not for biostatisticians or statisticians, but for applied analysts,” she told me, describing a classroom of people who came from varied fields and wanted to understand analytics, most without a formal statistical background.

Teaching that audience, for the better part of a decade, gave Rajicic a specific vantage point on where smart, capable people consistently go wrong before they’ve even touched a dataset. It’s a different lens than the one she uses when consulting with a fully staffed development team, and it surfaces a mistake that’s easy to overlook precisely because it doesn’t look like a technical error.

The Habit She Tried Hardest to Instill

Asked what she emphasized most, Rajicic didn’t point to a statistical technique. “I think what I tried to emphasize the most was to start with questions,” she said. “A good research question is really what sets you on the right path.”

That sounds almost too simple to be a teaching priority, until you consider how much analytical work goes wrong not because the modeling is flawed, but because the question being modeled was never clearly defined in the first place. A well-formed research question, in her framing, doesn’t need to be permanent. “Of course, you can revisit it,” she said, “but have some kind of a thought preceding the actual jumping into work.”

The second habit she pushed just as hard: staying skeptical of your own analysis as you go. “Just questioning everything along the way, knowing that things are so unknown,” she said. That instinct connects directly to a theme that ran through the rest of our conversation about her consulting work: the amount any single analyst, however capable, doesn’t know is larger than most people are comfortable admitting. Teaching students to sit with that discomfort, rather than resolving it prematurely, was a deliberate part of her course.

The Overconfidence Trap

Rajicic was candid about a specific failure mode she watched play out repeatedly and admits falling into herself earlier in her career. “It’s almost comical if a person has one course in statistics and learns really near regression and then thinks they know statistics,” she said. “And I fell into the same trap when I was younger. But as you get older, you realize you know nothing.”

For students who weren’t statisticians by training, one course in research design was never meant to make them experts. It was meant to give them enough grounding to ask good questions and recognize the edge of their own competence. “This is the start,” she’d tell her class, “but just please go to an expert once you need any help.” That instruction matters more in an industry where applied analysts, data scientists, and clinical operations staff are increasingly expected to interpret statistical output without formal statistical training. The goal isn’t to turn every team member into a biostatistician; it’s to make sure they know when they’ve reached the point where they need one.

What Doesn’t Change

I asked Rajicic, after nearly three decades in the field, what’s changed and what’s stayed the same. Her answer on what’s changed was structural: fewer sponsors employ large in-house biostatistics teams the way Pfizer once did, with outsourcing to CROs now common. But what stays constant, in her view, is more fundamental than any organizational trend. “You need good people, you need good teams,” she said. “Some teams include biostatisticians appropriately, some don’t. That hasn’t changed yet. Those that include them do better.”

She also pointed to a persistent structural challenge for the field itself: biostatistics remains a narrow, specialized discipline where formal education only gets a person partway there. “There’s so much that you need to learn by doing,” she said. “It’s almost like you need an apprenticeship to learn the business.” Combined with a talent pool she describes as consistently smaller than industry demand, that apprenticeship requirement is worth understanding for any organization trying to build internal analytical capability rather than relying entirely on outside expertise.

What This Means for Building Analytical Teams

For clinical technology leaders assembling or evaluating an internal analytics function, Rajicic’s classroom experience translates into a few concrete priorities. Train people to interrogate the research question before the dataset, not after. Build in explicit escalation paths so applied analysts and data scientists know when a question has crossed from something they can answer themselves into something that needs a trained biostatistician. And don’t mistake a single course, a certification, or a capable AI-assisted analytics tool for the years of applied experience that the field still requires to reliably know what you don’t know. That last point, more than any specific technique she taught, is the one Rajicic kept returning to her entire career, in the classroom and out of it.