Critical Rationalism for Founders and Managers
Company building as the process of solving problems via knowledge creation. On decisions as explanations, diluted visions, error correction speed, and A/B testing Empiricism.
All problems—including an unmet customer need, a bad user experience, or a failing company—are caused by insufficient knowledge. (A lack of resources is another potential cause, but we can always find a solution that requires less resources given the right knowledge).
But how do we attain this knowledge?
Critical Rationalism is a theory of knowledge developed by Karl Popper and David Deutsch which illuminates many problems we encounter as founders, CEOs, or managers.
Decisions as explanations choice
Most people don't understand the true nature of decision-making. The Cambridge Dictionary defines a decision as "a choice you make about something after thinking about several possibilities". But what happens during this thinking process?
In his book, The Beginning of Infinity, David Deutsch explains the process behind decision-making:
One does not simply choose an option by weighting or justification but instead by choosing its associated explanation.
What does he mean by this? We decide between options that are each based on different explanations of the best choice. Instead of picking the option itself, we choose its underlying explanation.
The best option is the one that corresponds to our best explanation. But what makes a good explanation? The best explanation is the one that is the least easy-to-vary. All its details play a functional role that correspond closely to reality. In other words, it has to be very precise, and there has to be a good reason for the precision. Practicing “first principle thinking” is using explanations that correspond closely to an objective fact (or principle) in reality.
Let’s look at two competing explanations of what a startup should focus on:
We should build feature A because it completes our suite of tools nicely, solving additional problems some of our users might have.
We should not build feature A nor feature B but focus on increasing ease-of-use, simplifying our onboarding and the core of the product because they help increase activation and retention which are our main bottlenecks to growth and adding more features will not address this bottleneck.
The second explanation is hard-to-vary because it is based on the companies agreed upon goal of increasing activation thereby corresponding closely to reality.
Remember your last hard decision, which you pondered for a long time? At the end of it, a single explanation likely dominated your decision.
The bad options are not outweighed but out-argued, refuted, and abandoned.
Coming up with new solutions, hypotheses, options, etc. is a creative process. For most problems, the number of possible solutions is infinite. Bryan Magee in Philosophy and the Real World writes:
A task does not begin with the attempt to solve a problem .... It begins with the problem itself, and with the reasons for its being a problem. One learns to work hard and long at the formulation of problems before one switches one's main attention to the search for possible solutions; and one's degree of success in the latter is often determined by one's degree of success in the former
Decision-making is never about selecting from an existing or fixed number of options according to a fixed formula. (For example, The founders of Tesla avoided the decision of car dealership margins altogether by choosing a direct-to-consumer model). It is a creative problem-solving process that starts with the problem formulation itself. We often create entirely new options when the current possibilities fail to solve our problem.
Diluted visions
Another common misconception about decision-making is that we can mix options. However, combining two explanations to create a better explanation requires an additional act of creativity.
This phenomenon is vivid for design decisions. Each design solution (options) is based on different explanations of the user's problems, their behavior, and many other factors. One designer thinks the car should look like a horse, while the other one thinks of a stealth aircraft. Mixing their design will look awful.
This is also why companies with a single leader or founding CEO are often more successful and efficient than those with a committee of leaders that mix their options to find agreement.
Groups Search for Consensus, Individuals Search for Truth—Naval
Truth is never found in compromise. The mistake is to merge the solutions, designs, or proposals to please each other and avoid conflict instead of exchanging and clarifying the differing underlying explanations of the situation. Clear thinking often happen as part of clear (and augmented) writing and communication.
A thriving company culture is one based on a culture of criticism.
Error correction speed
Mutation and selection in biological evolution are analogous to trial and error in innovation.
Critical Rationalism, unlike any other philosophy, views criticism as indispensable and the suppression of criticism as a deterrent of progress.
Since we know that innovation and the growth of knowledge is an evolutionary process, it follows that we cannot predict what the perfect solution or product will be. Elon Musk said:
For pretty much any technology whatsoever, the progress is a function of how many iterations do you have, and how much progress do you make between each iteration.
We must rely on many iterations and user feedback, navigate the idea maze, ship fast, and break things. We must do things that don't scale, trying tentative prototypical solutions before investing large amounts of resources.
Fallible deadlines
Work complicates to fill the time available. More generally, the demand upon a resource tends to expand to match the supply of the resource (If the price is zero).
Evidence is everywhere. Private companies are more productive than government agencies. Employees with a deadline in mind are more productive than those without one.
However, plans, like all our knowledge, are fallible. We can underestimate a task or fail to predict a roadblock. This is the nature of planning. Rather than shying away to set deadlines or goals we should embrace them knowing that we’ll revise them as needed.
We continually error-correct our planning. For example, after we fail to account for a “bug fixing phase” in project 1, we incorporate it into our project estimate of project 2.
Deadlines and estimates itself are based on hard-to-vary explanation. “It will take the two engineers X days because they never interacted with the system but have experience with similar code base areas Y and Z” is less easy-to-vary than “It will take them ~X days”.
Deadlines and plans are useful but fallible theories.
A/B testing Empiricism
A/B testing is a popular tool for founders and product managers trying to improve their products. However, it is often mis- and overused, wasting precious resources, especially when one lacks a theory of knowledge or hypothesis generation. Peter Thiel explains:
People don't generally know what they want. If you ask them surveys if you do market studies you will not actually get the right answer. There is something about this inspired intuition about what the world as a whole wants. A/b testing does not work for products. The problem is the search space is too big.
What Thiel here called this inspired judgment is a good explanation of the user's demand and preferences. And ideally, these are more precise than the user's own explanation.
Empiricism is the widespread but refuted belief that knowledge comes from observations and experience. Some claim you can gain the “hard truth” from data alone. Why is this wrong? Observation data is not translated by magic into theories or hypotheses. Observations themselves are theory-laden. It requires additional explanations to interpret any observation.
We have to conjecture our hypotheses and ideas. They are products of our creativity.
The only use of tests and experiments is to decide between competing theories. For that, a test needs to be based on a well-formed hypothesis. In David Deutsch’s terms, a good explanation. E.g., if you have a theory that change X will improve your user’s activation, but you can't think of a good competing theory or reason why it would worsen the activation, it doesn't make sense to add a test for that.
A common mistake is to run multi-variable, non-isolated tests. You will not know what impacted the test outcome if you vary multiple variables. In most cases, an A/B test only works if it's sufficiently isolated.
When PMs don't understand the process of conjecture and criticism, we end up with a lot of confusing A/B test data. Without a clear understanding of how to use tests to decide between existing hypotheses, we're left with no useful or conclusive theories on how to improve the product.
Also check out this critique of A/B-testing by the CEO of Airbnb:
Summarizing, I recommend these rules:
do not run multi-variable tests
every test needs a clear articulated hypothesis
only run a test if we have two competing theories (e.g. for an onboarding step or UI)
enforce a max duration for a test (ex. 2 weeks)
Coda
The ‘idea maze’, popularized by Chris Dixon, is not so much a maze but an open-ended space that the founder or product leader needs to explore via iterative error correction, driven by an undiluted vision, by fallible deadlines, and guided by hard-to-vary explanations.
Thanks to Sven Schnieders, Logan Chipkin, Bart Vanderhaegen, and Jorg Doku for comments.