6
Econometric Sense: The Value of Business Experiments Part 2: A Behavioral Econom...
source link: https://econometricsense.blogspot.com/2020/04/the-value-of-business-experiments-part.html
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
The Value of Business Experiments Part 2: A Behavioral Economic Perspective
In my previous post I discussed the value proposition of business experiments from a classical economic perspective. In this post I want to view this from a behavioral economic perspective. From this point of view business experiments can prove to be invaluable with respect to challenges related to overconfidence and decision making under uncertainty.
Heuristic Data Driven Decision Making and Data Story Telling
In a fast paced environment, decisions are often made quickly and often based on gut decisions. Progressive companies have tried as much as possible to leverage big data and analytics to be data driven organizations. Ideally, leveraging data would help to override biases and often gut instincts and ulterior motives that may stand behind a scientific hypothesis or business question. One of the many things we have learned from behavioral economics is that humans tend to over interpret data into unreliable patterns that lead to incorrect conclusions. Francis Bacon recognized this over 400 years ago:
"the human understanding is of its own nature prone to suppose the existence of more order and regularity in the world than it finds"
Decision makers can be easily duped by big data, ML, AI, and various BI tools into thinking that their data is speaking to them. As Jim Manzi and Stefan Thomke state in Harvard Business Review in the absence of formal randomized testing:
"executives end up misinterpreting statistical noise as causation—and making bad decisions"
Data seldom speaks, and when it does it is often lying. This is the impetus behind the introduction of what became the scientific method. The true art and science of data science is teasing out the truth, or what version of truth can be found in the story being told. I think this is where field experiments are most powerful and create the greatest value in the data science space.
"executives end up misinterpreting statistical noise as causation—and making bad decisions"
Data seldom speaks, and when it does it is often lying. This is the impetus behind the introduction of what became the scientific method. The true art and science of data science is teasing out the truth, or what version of truth can be found in the story being told. I think this is where field experiments are most powerful and create the greatest value in the data science space.
Decision Making Under Uncertainty, Risk Aversion, and The Dunning-Kruger Effect
Kahneman (in Thinking Fast and Slow) makes an interesting observation in relation to managerial decision making. Very often managers reward peddlers of even dangerously misleading information while disregarding or even punishing merchants of truth. Confidence in a decision is often based more on the coherence of a story than the quality of information that supports it. Those that take risks based on bad information, when it works out, are often rewarded. To quote Kahneman:
"a few lucky gambles can crown a reckless leader with a Halo of prescience and boldness"
As Kahneman discusses in Thinking Fast and Slow, those that often take the biggest risks are not necessarily any less risk averse, they simply are often less aware of the risks they are actually taking. This leads to overconfidence and lack of appreciation for uncertainty, and a culture where a solution based on pretended knowledge is often preferred and even rewarded. Its easy to see how the Dunning-Kruger effect would dominate. This feeds a viscous cycle that leads to collective blindness toward risk and uncertainty. It leads to taking risks that should be avoided in many cases, and prevents others from considering better but perhaps less audacious risks. Field experiments can help facilitate taking more educated gambles. Thinking through an experimental design (engaging Kahneman's system 2) provides a structured way of thinking about business problems and how to truly leverage data to solve them. And the data we get from experimental results can be interpreted causally. Identification of causal effects from an experiment helps us distinguish if outcomes are likely due to a business decision, as opposed to blindly trusting gut instincts, luck, or the noisy patterns we might find in the data.
Just as rapid cycles of experiments in a business setting can aid in the struggle with the knowledge problem, they also provide an objective and structured way of thinking about our data and the conclusions we can reach from it while avoiding as much as possible some of these behavioral pitfalls. A business culture that supports risk taking coupled with experimentation will come to value a preferred solution over pretended knowledge. That's valuable.
See also:
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK