Econometric Sense: The Value of Business Experiments and the Knowledge Problem

Why should firms leverage randomized business experiments? With recent advancements in computing power and machine learning, why can't they simply base all of their decisions on historical observational data? Perhaps statisticians and econometricians and others have a simple answer. Experiments may be the best (often the golden standard) way of answering causal questions. I certainly can't argue against answering causal questions (just read this blog). However, here I want to focus on a number of more fundamental reasons that experiments are necessary in business settings from the perspective of both classical and behavioral economics:

1) The Knowledge Problem
2) Behavioral Biases
3) Strategy and Tactics

In this post I want to discuss the value of business experiments from more of a neoclassical economic perspective. The fundamental problem of economics, society, and business is the knowledge problem. In his famous 1945 American Economic Review article The Use of Knowledge in Society, Hayek argues:

"the economic problem of society is not merely a problem of how to allocate 'given resources'....it is a problem of the utilization of knowledge which is not given to anyone in its totality."

A really good parable explaining the knowledge problem is the essay I, Pencil by Leonard E. Read. The fact that no one person possesses the necessary information to make something that seems so simple as a basic number 2 pencil captures the essence of the knowledge problem.

If you remember your principles of economics, you know that the knowledge problem is solved by a spontaneous order guided by prices which reflect tradeoffs based on the disaggregated incomplete and imperfect knowledge and preferences of millions (billions) of individuals. Prices serve both the function of providing information and the incentives to act on that information. It is through this information creation and coordinating process that prices help solve the knowledge problem.

Prices solve the problem of calculation that Hayek alluded to in his essay, and they are what coordinate all of the activities discussed in I, Pencil. The knowledge problem explains how market economies work, while at the same time, socially planned economies historically have failed to allocate resources in a manner that has not resulted in shortages, surpluses, and collapse.

In Living Economics: Yesterday, Today, and Tommorow by Peter J. Boettke, he discusses the knowledge problem in the context of firms and the work of economics Murray Rothbard:

"firms cannot vertically integrate without facing a calculation problem....vertical integration elminates the external market for producer goods."

In essence, and this seems consistent with Coase, as firms integrate to eliminate transactions costs they also eliminate the markets which generate the prices that solve the knowledge problem! In a way firms could be viewed as little islands with socially planned economies in a sea of market competition. As Luke Froeb masterfully illustrates in his text Managerial Economics: A Problem Solving Approach (3rd Ed), decisions within firms in effect create regulations, taxes, and subsidies that destroy wealth creating transactions. Managers should make decisions that consummate the most wealth creating transactions (or do their best not to destroy, discourage, or prohibit wealth creating transactions).

So how do we solve the knowledge problems in firms without the information creating and coordinating role of prices? Whenever mistakes are made, Luke Froeb provides this problem solving algorithm that asks:

1) Who is making the bad decision?
2) Do they have enough information to make a good decision?
3) Do they have the incentive to make a good decision?

In essence, in absence of prices, we must try to answer the same questions that prices often resolve. And we could leverage business experiments to address the second question above. Experiments can provide important causal decision making information. While I would never argue that data science, advanced analytics, artificial intelligence, or any field experiment could ever solve the knowledge problem, I will argue that business experiments become extremely valuable because of the knowledge problem within firms.

Going back to I, Pencil and Hayek's essay, the knowledge problem is solved through the spontaneous coordination of multitudes of individual plans via markets. Through a trial and error process where feedback is given through prices, the plans that do the best job coordinating peoples choices are adopted. Within firms there are often only a few plans compared to the market through various strategies and tactics. But as discussed in Jim Manzi's book Uncontrolled, firms can mimic this trial and error process through iterative experimentation interspersed with theory and subject matter expertise. Experiments help establish causal facts, but it takes theory and subject matter expertise to understand which facts are relevant.

In essence, while experiments don't perfectly emulate the same kind of evolutionary feedback mechanisms prices deliver in market competition, an iterative test and learn culture within a business may provide the best strategy for dealing with the knowledge problem. And that is one of many ways that business experiments are able to contribute value.

See also:
Statistics is a Way of Thinking, Not a Box of Tools

Post a Comment

Note: Only a member of this blog may post a comment.