Economics 101 taught generations of students learned about demand-supply curves and the supremacy and efficiency of the free market. The forgotten caveat of those microeconomic models was based on a number of assumptions that didn’t always hold, such as perfect information, frictionless costs, lack of entry and exit costs, etc.
Michael Porter, to take an example, has made a huge career and reputation on the idea that a firm can get more returns by capitalizing on advantages based on some of those market imperfections.
Market imperfections = Free market failure?
Recently, there has been some controversy over Goldman’s analyst huddles and how they affect the smooth functioning of the market. While this may turn out to be a tempest in a teapot and Goldman may not have done anything wrong in the technical sense, this is an example of a market imperfection.
Jeff Matthews also highlighted legal payoffs in the drug industry, which is another example of a market imperfection that results in a detriment to the public.
The Nature of Second Best
There is a very old economics paper dating from 1956, called The General Theory of Second Best. To summarize, if the current state of an economic system is one step away from the unconstrained optimum (the free market), the best single move is to towards the optimum. However, if the current state is two or more steps away from the optimum, or Second Best, then everything is up for grabs. The best single move may not be to move towards the unconstrained optimum. (If you didn't understand that, there is another explanation here).
The Theory of Second Best was a controversial paper when it came out. It was attacked by many theorists unsuccessfully for many years. Finally, it was ignored by economists, perhaps on the hopes that if you ignored it it would go away.
Greedy optimization algorithms
Quantitative analysts have performed constrained optimizations. We have all experienced optimization problems where too many constraints have yielded solutions that appear to be highly counterintuitive. The fact is, if you impose too many constraints on a system, a greedy algorithm that moves towards the unconstrained solution may give you a solution that is locally optimal, but suboptimal on a global basis.
Imperfect world, imperfection solutions
The lesson here is to understand the model, its assumptions and when the model may break down.
In the real world, there is no room for dogma and ideology. There is no doubt that an unconstrained free market solution (unconstrained as in assuming frictionless costs, participants having perfect information and equal bargaining power, etc.) is the most efficient economic system. However, we live in an imperfect world where many of those assumptions do not hold. Under these circumstances, moving towards a free market solution may not be the optimal solution. Each problem has to be studied on a case by case basis.