The replicability crisis in behavioral sciences - A design perspective

For those, like me, who follow Behavioral Sciences regularly, the replicability crisis is no news to us. For years there have been reports on scientists not being able to replicate some of the fundamental results. Scandals on data manipulation and grandiose claims that just look too good to be true.

So this was not a surprise when some of you pointed me to Pete Judo’s latest video where he announces he would no longer be using his channel to communicate behavioral sciences findings because he no longer trust the science.

I’ve always regarded Pete’s channel as one of the best in explaining some of these effects, and the fact is I fundamentally agree with his conclusion. So why am I not reconsidering my whole career path?

My incursion to behavioral biases and frameworks is different from Pete’s (and from most) I’m not an academic, I’m a dropout who happened to be in the right place to learn this stuff from industry. I’ve seen casinos make a lot of money out of behavioral biases, I know that videogames are purposely designed to take advantage of these effects. And I had the opportunity to design some of these interventions and take them to production. I had the chance to learn by doing, and I know that some of these effects work.

But why, then, there is a replicability crisis?

The science of behavior is at best, at its infancy. Is difficult to define what is cultural, what behaviors are due to evolution, and what is just a fluke. To make it more complex, cultural and biological aspects interact with each other. And more importantly, context plays a huge role.

We often hear that there is a psychology to color, that fast food restaurants use red to get you moving. That red is the color for aggression and movement. Yet, as somebody who has shared some mates at sunset by a shoreline I can assure you that red is the most calming setting.

Behaviors are complex, and context is key in complex systems. When faced with strong currents or a heavy boat, even the best oars won’t be enough to change your path. But does this mean that oars are inherently unreliable or unpredictable tools?

Now it’s not the same to send a physical letter in Uruguay, in London or in the Amazonia. The fact of even receiving the mail is contextually different, who is the sender also matters. Is not the same for me to receive a negative framing message from my local tax authority than from the burger place down the street. Does this contradict loss aversion? My age, my cultural background, even if I’m feeling down, there are countless factors that can affect my decision to act. And, let’s face it, most studies are not broad enough (not just in number of participants) to make the claims they do. This is the primary reason why I view Applied Behavioral Sciences as more of a design or engineering endeavor than a purely scientific one. It is inherently iterative, where the inclusion of various effects and strategies in the design process becomes vital for intentionally generating results

Finally, Execution. How the strategies are actually implemented is one of the elements that papers and documents can rarely convey. We are dealing with a chaotic system and as such, small changes can have huge repercussions down the line. This is why I prefer to have product owners that understand these concepts execute the designs, and I follow the development and iterations processes closely of the projects I’m involved in.

In his video, Pete discusses his continued faith in his day job, highlighting the iterative process and the ability to form hypotheses based on data analysis. My own journey in applying these principles has led me to understand that success lies not in replicating the laboratory’s controlled environment. Instead, it revolves around systematizing, identifying effective strategies, and using heuristics to develop outstanding products. Even though Behavioral Sciences are valuable findings are valuable assets for the design of better products and services, I see Applied Behavioral Sciences as a discipline that still leans toward design principles rather than a hard science. Therefore, I stress the usage of the term behavioral (product) design that better encapsulates its essence.