Stopping belief based policy

Creating a culture of policy that follows the science

Abhijit Banerjee, a Massachusetts Institute of Technology (MIT) economics professor, and co-founder and director of the Abdul Latif Jameel Poverty Action Lab (J-PAL) won the Nobel Prize for his work on poverty in large part by always seeking to answer the question, “How do you know?” As an academic and someone who has committed a lot of my life to learning how to ask the right question and then answer it, I was profoundly struck by the work of Dr. Banerjee on poverty. Unwilling to accept broad sweeping conclusions as to why things are the way they are, Dr. Banerjee broke things down and looked at how they were implemented, how they worked. He constantly challenged assumptions people had and wanted to see if what people thought was actually true was indeed true on the ground where the programs or policies were implemented.

In a brilliant interview on NPR’s Hidden Brain, Dr. Banerjee described a study that challenged the dominant assumption that more teachers would equal better test scores in Rajasthan, a state in northern India. He doubled the number of teachers in each school thinking, like most people, that more teachers would lead to better educational outcomes. But what he found was that it wasn’t true. Scores didn’t improve, educational outcomes weren't better. He did a similar experiment though this time decreasing the size of the class, decreasing the number of students in a classroom, and found the same result - no changes in outcomes or test scores. He saw this as a way to begin to challenge many assumptions in education and to start digging deeper in search for the solutions that did work.

How many times do we witness policy makers deciding on funding for a program that we don’t actually know if it works? I will tell you, it happens all the time. There are strong opinions and beliefs out there - call them assumptions - that simply aren’t grounded in the science of what works.

You don’t need to look around far to find examples. Consider work requirements for Medicaid beneficiaries. For some reason, there’s been a belief that people who are poor just need a job to get out of poverty. Medicaid, a health insurance program that’s a partnership between the states and the federal government, allows for subsidized health insurance coverage to be available for for individuals that fall beneath a certain income threshold. What some lawmakers have done is to require that people who are on Medicaid to work or show they are working to get their benefits. Here’s the funny thing - most of them already do work. But for some policy makers, they have a belief that they should put the burden of showing that people work on the back of people who are already struggling despite the evidence that: a) most of them do work; and, b) work requirements actually do more harm than good. But even in the face of irrefutable evidence, policy makers still continue to push for this idea.

One study looked specifically at how the evidence was used with policy makers and their decisions on abortion. Results revealed that legislators tend to not base their opinions on research or alter their views through it. Despite this, policy makers routinely cite selected research findings to give credibility to their arguments. The authors found that regardless of political party, both sides prioritized their political values over evidence, with personal narratives holding greater persuasive power than research evidence. Legislators also tend to place more trust in personal anecdotes than scientific evidence, underscoring the need for qualitative research to provide compelling stories alongside any quantitative data.

Studies like this affirm previous findings on evidence use in policy, highlighting motivated reasoning and the prevalent skepticism towards evidence. It also underscores the significance of trusted intermediaries and the consideration of evidence requirements within other aspects of government like the judicial branch. There’s a lot to learn from this type of work, including the need for advocates, researchers, and other stakeholders to effectively target their efforts in disseminating and educating policy makers.

Whatever the issue, it’s possible to dig deeper into the psychology of it to understand why it is so common. Confirmation bias, which is the inclination to favor information that aligns with preexisting beliefs while disregarding contradictory information, is a well-documented phenomenon in cognitive psychology. It’s one of the most extensively studied forms of flawed thinking, with numerous experiments dedicated to exploring its effects. This bias has been thoroughly examined and extensively discussed, even meriting entire textbooks dedicated to unraveling its intricacies and implications. But sadly, it exists and is pretty dominant even now, especially in our policy.

Another review of evidence-based policy making focused on how we can enhance our understanding of how research programs and researchers can work to improve the evidence design that inform policies and policymakers. Of course, most of us assume that if the data or the evidence is there and people use it; however, despite the seemingly simple concept of "evidence-based policymaking," both the term and the policymaking process itself are indeed complex and run counter to our assumptions. See the examples above! Definitions and interpretations of evidence, policies, and policymaking vary significantly among individuals. One person may see a story alone as sufficient data while another looks to a randomized controlled clinical trial (RCT). The point here is that we have to know how to take the best available evidence and put it into a format that works for policy makers.

Believe it or not, breaking the cycle of belief based policy begins with us. There’s a lot we can do to influence the direction of policy and infuse the best available evidence into the process.

• Listen: Sometimes in our fervor and passion on a given topic, we actually turn off our hearing and ignore what the other person is saying. Create a space for both sides to share their take on the issue, present the data, and then engage in a more informed dialogue. While it may not net out always the way you want it, beginning with a listening posture is far more effective.

• Summarize but don’t sell: It’s not uncommon for people to oversell their position based on one fact. It happens all the time, and we should be aware that just as we are making an argument with the evidence, someone else is using another set of data to make a counter argument. Being able to summarize and synthesize the data is an art that allows you to be more nimble in your dialogue. Our passion sometimes leads us to oversell, which for those informed policy makers, can turn them away from an issue.

• Be consistent: Become known as the person who gives the most up to date take on the evidence. Share articles as they come out, summarizing them for the policy makers. Even when a study that comes out works against your position, its important to share that too and explain why it matters. Chances are, if you are working from a place where the science is your foundation, you will have more studies supporting your position than not.

• Make yourself available: Policy makers love to have an expert that they can reach out to when they have questions. Offer up your email and phone (as you are comfortable) and just let them know you are there. Establishing trust is another great way to better infuse a lot of the science into the policy makes decision making.

If you want to learn more about policy, including the best ways to help us move away from belief based policy to evidenced informed policy, Inseparable is launching a new mental health policy fellowship. If you are passionate about mental health and making an impact in the policy space, check it out as they are accepting applications now for this coming year.

So, how do we know?

I ask myself this question a lot. It feels that in 2023, it’s our job to challenge assumptions about policy making, but alas, we don’t as much as we should. So many policies and programs we have inherited that we often assume that they have been grounded in some sound science. Policy makers also tend to get inspired, based on their own experience, and these leaders leverage their passion to really push for the change they think they need, and inspiring or convincing us along the way. Let’s do our part to help bring in as much as the evidence as we can to help inform our policy makers what a better future for mental health should look like.