Decision science

Yes, decision science is a thing. It’s a skill and a set of tools. I've been learning and teaching it for ten years. The first two years were very intense - I studied about 20-30 hours a week. I'm advanced, but I know people who are several levels above me (they all work in hedge funds or teach economics). In my experience, it takes most people two years to learn the basic set of tools to become an evidence-based decisionmaker, and that's if you're under 30. It takes longer for older people, because they think they know everything.

You might think it comes down to values, but it doesn’t. Most decisions don’t turn out the way people think they will. Most “dilemmas” disappear when you look more at data and remove the beliefs and biases. What we call a decision is really a process.

Example: Apple watch

Suppose you’re Tim Cook. The engineers come to you and say “We’ve done it! We can integrate a heart monitor into the Apple Watch!” Of course, many people will want that feature. But to make the decision, you need to measure both costs and benefits. Aside from a small increase in price, what is the cost? Here's an example where I calculate how many people the Apple watch will kill each year. If you read that, you'll see it's very quantitative. Now, you have to calculate the benefits, and that is a similar exercise. Finally, you lay out the cost/benefit curve and try to find the right place on it. Then, you should track and see if it turns out the way you thought, because usually you have to adjust later.

This is not the way most decisions are made.

Example: The missing bullet holes

There is a famous story about Abraham Wald and the Missing Bullet Holes, which is excellent reading, but I’ll summarize:

In World War II, Wald, who worked for the top-secret Statistical Research Group for the US Armed Forces, was puzzled by the fact that bombers that returned had bullet holes in certain consistent places and no bullet holes in other places:

The navy decided they needed to add (heavy) armor plating to the areas with the bullet holes, but Wald realized this was an instance of survivorship bias. These planes were the ones coming back! The planes getting shot down were being hit in the other areas. The engines, not the wings, needed more protection.

Example: crimes and punishment

Danny Kahneman tells a story of positive vs negative reinforcement. In any given exercise, certain people will perform better, others average, and some poorly. When air-force flight instructors berated those who performed poorly, they noticed that those pilots did better next time, so they decided that punishment was effective. They failed to understand that this was mean reversion. Every time they ran the exercise, some people would underperform. Those who underperformed would improve next time, simply as a result of natural variance. Consistently punishing underperformers was totally counterproductive.

There are many examples where decision science can help us understand a situation. Most errors in decisionmaking come in the form of poor framing, bad data, or underestimating the options. The other thing we rarely do is check back on previous decisions to see how they turned out, given the information available at the time.

Decision Quality

The first thing that will practically double your decision quality is to walk through the process with someone else who is disinterested in the outcome. Someone who can objectively help you frame and set up the decision without tipping the scale. This person's objective is to arrange the solution space so that all choices have the same tradeoff ratio of costs and benefits, ensuring that the decision is on that curve somewhere.

The second thing is to use a decision process that comes from decision science. Here's an example, most others are similar. They have usually six or seven independent steps:

  1. Frame - background, deadline, objective

  2. Alternatives

  3. Information gathering

  4. Understanding costs, benefits, values, and tradeoffs

  5. Sound reasoning (applied rationality)

  6. Commitment to action

  7. Communication

Step five is the actual decision. There are many ways to make it, depending on the stakeholders, costs, benefits, and potential "unknown unknowns." Sometimes you can use a scoring system, other times you need to be guided by priorities. While this sounds straightforward, it's actually quite complex. For example, when making a medical decision, people are biased by relationships with doctors they trust, even though studies repeatedly show that doctors don't have the statistical training to understand the odds they provide to patients.

The third most important thing is to keep a record of your decisions, so you can go back later and check how you did against the outcome. Most of our decisions are made in the real world, which usually involves complex adaptive systems. A good decision can have a bad outcome, and a terrible decision can result in a good outcome. In addition, we have type-1 and type-2 decision errors:

  1. A Type-1 error is deciding to do something and it turns out wrong.

  2. A Type-2 error is deciding not to do something that later turns out very well.

Large companies deciding to drill for oil or or to buy a competitor or build a new factory go through a fairly formal decision process (see that document if you want an idea how thorough this can be). In general, the rule is that you should spend 3-6 percent of the value of a decision on the decision itself. Too many decisions are based on gut feel and personal experience and are, to use a gentle word, suboptimal. Companies that help with large decisions (usually $100m and up) include:

All of these organizations have excellent content on their web sites and many YouTube videos to watch. There's a lot to learn. Sometimes, it helps to build a spreadsheet model to see the alternatives. Sometimes, the model is the problem. Almost always, you're trying to quantify the unknowns. That's a skill that takes development. In most cases, you're trying to use Bayesian reasoning to arrive at the best alternative.

We have qualified modelers and decision scientists ready to help our clients with any decision - personal or business, large or small. And, they can help you build better decision systems for your team and your family.