When evidence met innovation

A love story in two parts

In 2003, there was an article published in the British Medical Journal (BMJ) that sought to answer the apparently unanswered question: Is there evidence that parachutes work when jumping from an aircraft? The authors pointed out that because there had not been a randomized controlled trial (RCT) for this question, we couldn’t conclusively say that parachutes actually worked for those falling from an aircraft. This call for more research was despite the significant amount of anecdotal evidence available indicating that people who used parachutes were more likely to live than those who jumped from an airplane without a parachute.

You see, there’s a dominant belief in most of health care that if there’s not an RCT for your given innovation then you shouldn’t be doing it. Period. This method, where you randomly assign participants to either an intervention group or a control group, is seen as the “gold standard” to test the effectiveness of a treatment or intervention. Purists of this approach believe that this method is the only way to minimize bias and ensure outcomes are solely attributable to the intervention being tested. Said differently: It’s much harder to argue with the outcomes of an RCT because of the rigorous design that reduces the likelihood of results being due to chance, providing strong evidence of causality, and making the findings more reliable and credible.

The BMJ article does a brilliant job of critiquing our rigid insistence on RCTs. We all know, even if we haven’t jumped out of an airplane, that parachutes work. However, despite widespread belief in their effectiveness, none of us have ever read a study showing us this fact. As I mentioned, our belief is largely based on anecdotal evidence and observational data, which show both benefits and associated risks.

What I love about this paper is that the authors write this up like a real paper. I mean it is a real paper, published in a highly reputable journal, but the seriousness they use to address something so common sense is still hilarious. They even go so far to lay out how they performed a systematic review of RCTs on parachutes and, surprise, surprise, found none. Their point, brilliantly made, is that there are a lot of things that work out there that have not gone through a rigorous RCT design but because we have not done the RCT, we can’t or won’t use them or include them.

Deaton and Cartwright in their article “Understanding and Misunderstanding Randomized Controlled Trials” published in Social Science & Medicine write:

“We argue that any special status for RCTs is unwarranted. Which method is most likely to yield a good causal inference depends on what we are trying to discover as well as on what is already known. When little prior knowledge is available, no method is likely to yield well-supported conclusions. This paper is not a criticism of RCTs in and of themselves, nor does it propose any hierarchy of evidence, nor attempt to identify good and bad studies. Instead, we will argue that, depending on what we want to discover, why we want to discover it, and what we already know, there will often be superior routes of investigation and, for a great many questions where RCTs can help, a great deal of other work—empirical, theoretical, and conceptual—needs to be done to make the results of an RCT serviceable.”

Basically, Deaton and Cartwright are arguing that RCTs don't always provide the best evidence and that the effectiveness of any method depends on the context and existing knowledge! Back to our parachute article for a second - the context of jumping out of a plane really matters as does our knowledge of a device that slows down our fall. Without these facts, an RCT does little to advance our knowledge or understanding of the issue.

In his wonderful book, The Half-Life of Facts, Samuel Arbesman writes about how our understanding of facts evolves over time; just as the Earth was once thought to be flat, many accepted facts eventually become outdated as new discoveries are made. Using pretty complex mathematical models, Arbesman shows how knowledge in various fields changes over time, likening this to the half-life of radioactive materials or medicines. Consider that half of today's physics books will need revision in 13 years, and history books in just seven! We learn, things change, we think differently. His ultimate point is that the sooner we are able to see how facts are transient and science evolves, the better we can adapt to the continuous evolution of understanding. This awareness not only encourages open-mindedness but also fosters a culture of lifelong learning, urging us to remain curious and skeptical as we navigate a world that’s always throwing new things at us.

Innovation, an often overused term, in its basic form is taking ideas and putting them into practice so there’s an impact. We talk about innovation all the time as the solution for all our problems.

“We just need an innovative way to increase access for mental health.”

“I bet that there is going to be some amazingly innovative tool that helps us better measure well-being.”

“Have you seen the latest innovative tools coming out of (fill in your favorite university here)?”

But how do we know what’s truly innovative? How can we be sure that this new idea will yield the outcomes we’re looking for? Well, my friends, we typically lean into the evidence to help us understand if the innovation works or not, so we don’t invest in ideas that won’t deliver. But what happens if the evidence we have isn’t good enough for someone to adopt our idea? What happens when we disagree on what evidence is and isn’t?

While it seems like a simple distinction, there’s actually a ton of debate in this space that’s extremely relevant for policymakers, investors, and anyone looking to help bring solutions to scale. Imagine the frustration of having a groundbreaking mental health intervention, only to see it stall because the existing evidence frameworks can’t fully capture its potential. Or consider the innovators who shy away from bold ideas because the rigid evidence requirements feel like insurmountable barriers.

I want to be clear: I am not against RCTs. In fact, there is a role for them to help strengthen the evidence base! However, similar to Deaton and Cartwright, I feel we have placed an outsized weight on their importance. It’s like trying to judge the value of a novel by looking at its cover or only reading its table of contents. We miss the nuances, the context, and the real-world applicability that other forms of evidence can provide.

We need both evidence and innovation, and can’t overlook one for the other. These two things must coexist, informing each other in a dynamic interplay. In 2024, it seems we could see innovative mental health solutions rapidly prototyped, tested in real-world settings, and then refined through a continuous feedback loop of diverse evidence. This is the balance we must strive for—where the rigor of evidence and the spirit of innovation dance together, leading us to breakthroughs that are not only effective but also transformative for our communities. Then let’s apply our learnings to the field in a way that lets people benefit from the innovation as quickly as possible.

In conclusion, much like the absurdity of questioning the efficacy of parachutes without an RCT, we must recognize the limitations of only rigidly adhering to one form of evidence. The BMJ article cleverly highlights how common sense and diverse forms of evidence can sometimes offer more practical insights than strictly controlled studies. As we innovate and seek to address complex issues like mental health, it seems we must embrace a more comprehensive approach to evidence—one that values real-world applicability and contextual understanding alongside rigorous research. By doing so, we can ensure that life-saving interventions, whether as straightforward as parachutes or as intricate as community initiated care, are given the opportunity to prove their worth and make a meaningful impact on our communities.