Evidence-Based Consulting: A Manifesto

Management consultants are sometimes criticized for overuse of buzzwords, reliance on and propagation of management fads, and a failure to develop plans that are executable by the client. 
                         - Wikipedia entry on Management Consulting

In my white paper, I ask "What is the Role of Management?" In this paper, I ask the question: "What is the Role of Management Consulting?" The role of most consultants is to bring experience and methodology to a problem that a client may not be as familiar with. You bring your car to a mechanic and see a dentist to have your tooth fixed. But in complex areas like medicine and management, we're learning that most experts aren't as informed as they think they are. While some management consulting firms do add some value, it's not often that their goals are aligned with their clients'. Here are just a few of the web sites documenting the dramatic failures of the big management-consulting firms:

Management consulting is at a turning point. Clients are looking for better answers and measurable results. New models are emerging:

I want to build a new management consulting and coaching firm for the 21st century, where the goal is to help clients need less and less consulting, rather than more and more. 

A New Kind of Company 
The Infinite Game of Life offers an alternative model based on experiments, evidence, decision science, and lean start-up methods:

The goal of business agility is to be less predictive and discover what works through iteration and experiments. I'll start with an example, then I'll break down some of the key components.

Example: JC Penney
In 2012, JC Penney, a no-frills retailer with over 1,100 stores, was in trouble. Famous for weekly and monthly sales, the company was losing ground to more nimble competitors. They needed to bring in someone with vision. In 2012, it seemed that Ron Johnson had the swagger and the retail chops they were looking for. He had overseen the building of Apple’s retail store empire. He also had the benefit of two Steves: Steve Jobs, who built the iTunes platform, on which the company’s catalog of consumer products would iterate and scale; and Steve Balmer at Microsoft, who spent ten years trying to port his legacy enterprise mindset to the Web, in the process shoveling customers who didn’t want to buy a Windows phone at Apple by the handful. From there, the man who had made $400 million on Apple stock options commuted to Texas to head J. C. Penney, taking his hand at the helm. 

Just a few months after his arrival in late 2011, Johnson announced his team’s new vision for the chain — creating mini-stores dedicated to well-known brands (as Macy’s and Bloomingdales did 20 years ago). According to Johnson himself, he makes big decisions by gut feel:

All my ideas just sort of come. I don’t know how to explain it. It’s all intuitive, I think.
       - Ron Johnson

His plan, which I guess he worked out on the corporate jet on Mondays and Fridays, was to roll out the vision to 100 stores and then expand from there. He spent much of his time working on a secret high-tech concept store in Dallas. At an analyst meeting in August, 2012, Johnson said, “I’m completely convinced that our transformation is on track.” Unfortunately, Q4 2012 was a disaster for the chain. In April 2013, the board fired him. 

And who replaced him? Mike Ullman, the retailer’s former CEO, who immediately started putting up “sale” signs again. [Update, January, 2014: J C Penney closes 33 stores.]

This isn’t a management failure. This is a board failure. It's a result of confirmation bias, groupthink, the Halo Effect, and misunderstanding cause and effect. Not only did they bring in a guy who was going to spend a ton of money getting 100 stores ready for his big new vision, but they fired him before he had a chance to show whether his plan could even work. Why make such a huge bet in the first place? Rather than working on stealth concept stores with your smartest, most creative people, spend time listening to managers who listen to customers. 

Here is how business agility can help J. C. Penney and other retailers survive, if not win, in the 21st century: set up a training center where store managers can find an array of technologies, ideas, examples, and resources for trying things with their customers. Put them through a three-day experiment boot camp that shows them how to run controlled experiments, measure results, and set up the analysis properly. Help them brainstorm new but tryable approaches to online, advertising, word of mouth, the store experience, the after-purchase experience — every place where their brand touches consumers. Then, put all these experiments and results into an online system, so everyone (not just the C-suite executives) can see a comprehensive, evidence-based dashboard of all the experiments — what hasn’t worked, what’s working where, what needs improvement, what they have learned about increasing conversion rates for their stores. Some stores will find their customers receptive to new ideas, while others will continue the bargain-hunting tradition. This could all be set up and working within six months, and it doesn’t hurt existing revenues.

The key is convexity - experiments are cheap and failure costs little, while successful ideas spread out and multiply quickly and easily. 

In this new culture, a third of the store managers will likely eagerly embrace the new approach and start trying and sharing immediately, making plenty of mistakes but learning quickly, another third will be too "old school" to get on board and will eventually need to be replaced, while the middle third won't innovate much but will follow the first group, and mentoring will become a key part of the program. Within a few years, the company should start to see a new organization appear on its own, with leaders rising from the pack, some stores closing, digital innovation following trends on the ground, peer-to-peer education, and much more. You can characterize "J C Penney customers" as bargain-hunters, but you really don't know who wants what and how they might change until you try and learn.  

The astute reader will note that in this scenario the CEO and his direct reports matter far less to the success of the business. They are responsible for creating the environment in which the store managers learn, and they make sure everyone is supported. It lets their fantastic salaries and golden parachutes trickle down to the store-manager level, which coincidentally injects money into the local economies, rather than offshore hedge funds. 

Can Experiments Replace Planning?
In many cases, experiments can replace most of what traditional management consultants do. Recently, an article in the New York Times suggested that open-office plans were failing, but in the world of programmers, open offices are on the rise because they encourage collaboration. More companies are putting their work flow and ideas on the wall, for everyone to see, and that seems to be working quite well. Some companies are even starting to experiment with open financials - letting everyone in the company see the books at all times. Recently, Urban Outfitters gave students a chance to design products that will be placed in their stores, to see how they sell, rather than just being judged by experts at the end of a competition. Harrah's hotels are famous for doing carefully controlled experiments to keep up with consumer demand. Companies are starting to hear about the concept of running experiments, but they don't know how to do it. 

An experiment requires a control group and a single variable. Multivariate experiments often generate too much noise and no clear signal. Most drug companies do it poorly most of the time, and regulators don't understand the basics, so they approve studies that seem convincing. For large experiments, it takes competent, experienced designers and rigorous statistical analysis to find signals. Other times, companies can do many simple experiments and learn something every week if they are willing to try new things. 

The key here is a corporate culture that supports many small failures. Ask yourself: if you tried something for 2 months and it didn't work out, would your co-workers congratulate you on learning something? If not, your culture doesn't support experiments. And any corporate culture that doesn't support small failures unwittingly rewards big risk-taking: by promoting people whose projects happen to work and firing those who fail, many companies implicitly reward managers for taking big risks. It's very hard to make steady progress when you leap forward without a good idea of what drives success and what causes failure. 

In Joy, Inc, we learn the story of a company of 150 people who eliminated management, became much more productive, and had fun doing it. In Employees First, Customers Second, you'll learn about a company of 30,000 people that turned management upside down by doing repeated experiments with the goal of empowering employees in the "value zone" and managed to pass their competitors during the global economic crisis.  

Cognitive Diversity
While Johnson's kind of "genius" managers have long reigned supreme in the world of strategy, we're starting to learn that cognitive diversity plays an important role in predicting the future. Philip Tetlock's work has shown that a group of people chosen at random but with skin in the game will outpredict experts most of the time.

It's unusual for companies not to have big hero leaders, but that's starting to change. Thoughtworks, a 2,500-person IT services firm, now has co-CEOs and several people sharing the various top leadership roles, in 2s, 3s, and 4s across regions and verticals. It's not several people doing exactly the same job; it's more like a council approach to leadership that ensures more cognitive diversity and less halo effect. 

Measuring Success
According to Doug Hubbard, author of How to Measure Anything, most companies don’t get measurement right. They spend too much measuring the wrong things and don’t realize they could get away with far less data, if only it were good data. Measurement is at the heart of management, yet few people employ data scientists and statistics PhDs trained in rigorous data gathering and analysis. Most companies don’t have decision-science PhDs on staff. Many companies run ad-hoc experiments without controls and use the HIPPO method of decisionmaking (HIghest Paid Person’s Opinion). Most “analytics” groups do nothing more than validate decisions and positions already taken. Pseudoscience rules.

Hubbard uses his Applied Information Economics method to determine the value of information, acquire the right information at the right cost, use stochastic models to look intelligently at future scenarios, and use a portfolio approach to making decisions. Few companies approach a big project or decision with this kind of rigor and statistical understanding.

Data Science
These days, everyone seems to be "data-driven." As you'll learn if you watch my History of Information video, and as you've no doubt read, we are becoming exponentially surrounded by data. Soon, our shoes and wallets, our resumes and dog collars will be generating more data than we can use. There are many commercial applications of "big data" - we keep hearing about the "unlimited potential" of unlocking our genomes, proteomes, biomes, drug prescriptions, brain maps, wind maps, consumer data contrails, transaction data, and more. Before we begin using big data, we should recognize a few things: 

  • We're doing a poor job of analysis with the data we already have. Very few companies are making enough sense out of their data today. Until we can address our cognitive biases and systemic problems with science, and stop fooling ourselves, we would do well to work with less data, rather than more. 

  • We should not begin with the end in mind. The term for this is cherrypicking, and as Taleb says, "Big Data takes cherrypicking to an industrial level." We must ask open questions and try to disprove our findings before starting to believe them. 

  • The more data, the more noise. As Sherri Rose explains in her short, insightful paper, "With such large data sets, it is all too easy to find rare statistical anomalies and to confuse them with real phenomena."

  • Soft metrics don't firm up when you run them through various analytical tools. As an example, the hugely influential and stunningly flawed IBM CEO Study looks gorgeous and convincing, but it's statistical gibberish - nothing more than a cooked-up marketing piece that puts IBM right in the middle of the latest trends (coincidence?). IBM probably has some world-class data scientists, but they don’t work on that project. 

  • The error bars are still as or more important than the number. People like numbers. A single number is far easier to understand than confidence intervals, error bars, and statistical significance. By munging huge data sets and doing complicated calculations, we can deliver concrete answers that have little bearing on the real world. Worse - if the number happens to be right, that just increases our sense of confidence that we know what we're doing. 

  • Machine learning is going to come with risks. The more our "assistants" can do for us, the more we'll rely on them, but the more mistakes they will make. It's possible that the mistakes will take us into new territory. As an example, 60+% of all trades on the stock exchanges are now made by algorithms. Very occasionally, they have no idea what they are doing, as occurred in the Flash Crash of 2010, and we still haven't figured out why. Some day in the not-too-distant future, they could cause a lot of damage to an already fragile system. As Christopher Steiner shows, algorithms are going to play a role in or completely dominate almost all areas of our lives, and there are big questions to be answered. And, sometimes, the best tool for the job is a human being. 

  • Not using data comes with its own risks. Any critical look at our healthcare system will tell you that. The human mind is designed to overlook the details and skip the hard stuff. Business agility puts both human judgment and data-driven analytics on a level playing field, asking hard questions like "Is that really true?" and "How do we know it's true?" 

  • Our statistical capabilities and tools will need to improve. Most "data scientists" aren't really practicing science, the way Karl Popper and Richard Feynman defined it. In most cases, they aren't trained well enough to distinguish between prediction and inference. Researchers in statistics are scrambling to keep up. Most people still use Excel to do much of their analysis, while far better tools exist today. Those interested in forecasting should look into probability management. 

  • Our standards for data scientists will need to improve. Today, many people call themselves data scientists. They find patterns in huge clouds of data as easily as they find faces in french toast. A few people are starting to explore and teach data science in a more rigorous way - these people will lead us into the era of big-data analysis and stochastic forecasting. The rest is marketing. 

Workforce Science
The study of what actually works in human resources is really just beginning. For far too long, we've let tradition and "common sense" dictate how we hire, compensate, and work with people. Now, academics like Bob Sutton and Jeffrey Pfeffer, consultants like Patty McCord, companies like Google, and groups like the Mercer Workforce Sciences Institute are using data and evidence to find the signal in the noise. Much of what they are learning is that traditional interviews, 360-feedback, performance reviews, and performance pay do not contribute to employee satisfaction, productivity, or longevity. Longevity itself turns out to be a poor measure of happy productive workers. We have a long way to go, but we're learning to promote a more supportive, flexible environment and give people more autonomy. Smaller human-resources departments and fewer rules are likely to be big winners in the future. 

Workforce science looks at the evidence rather than tradition for better answers to human resource challenges. 

Organizational Democracy
More and more, we're seeing evidence that traditional top-down management structures aren't creating the value we want. Companies that put systems in place to develop self governance and democracy are speeding past slower competitors. According to a recent Gallup poll, 70% of employees are not engaged with their work. In fact, on average, 18% of employees are actively disengaged. Given that we spend the majority of our productive hours at work, wouldn't it benefit companies to measure and improve engagement? One of the best ways we've found is to reduce, reuse, and recycle managers into productive team members and let the systems manage the process. Organizations like Worldblu and The Great Game of Business are at the leading edge of workforce democracy. For serious inspiration, read the Valve Employee Handbook, Employees First, Customers Second, and Joy,  Inc.. 

Agile Methods
Startups are learning what works in the real world far faster than larger companies are. Not only are they doing experiments, they're doing away with traditional top-down management. Much of their new management philosophy comes from the Agile programming movement, which can be said to have three important characteristics (greatly simplified): 

  1. Work in pairs on almost everything. People work together, side by side, thinking and producing together, which reduces errors and improves quality dramatically. 

  2. Do your thinking in public. Agile shops have whiteboards everywhere and special software for sharing the ongoing development and creative processes. Their walls are covered in sticky notes, cards, and ideas. 

  3. Continuous delivery focuses on small, incremental work units and keeps delivering the next most important feature every week or two. Two of the leaders of the agile movement are Thoughtworks and Menlo Innovations. I believe both use 1-week sprints, though they may go longer if necessary. Tools like Atlassian Jira help large companies coordinate everything. 

  4. Continuous Improvement brings a Kanban approach to all areas of work. Using short work sprints and shipping often allows companies to keep improving their products every week. This gives everyone more confidence and ability to get things done. 

  5. You don't schedule your own time. You coordinate through a scrum manager or team coordinator, whose only job is to facilitate people working together to knock off the work backlog (work items to be done). They often do this through the use of planning games using sticky notes or cards on walls, showing what's happening in the next several days and who needs to work on what. 

Any company can apply this process to everything, from budgeting to marketing to product planning to human resources. There is a difference between production processes and projects, there are a few times when working alone is beneficial, and on rare occasions the situation requires a chief and many indians, but in general most layers of management can probably be replaced by scrum managers and project owners. There are many books and web sites dedicate to agile, but I recommend starting with The Scrum Reference Card and then reading The Leader's Guide to Radical Management, by Stephen Denning. 

Can we apply agile methodology to all businesses and situations? Probably not, but it's an experiment we can try and learn from - even some hospitals are starting to learn the value of continuous delivery of services. More and more companies are doing it successfully, but it seems better suited to collaborative, problem-solving or creative work than to line/production/routine tasks. W. L. Gore has 9,000 employees and no managers. Zappos recently repurposed all their managers and adopted Holacracy. Menlo Innovations produces software on time and on budget, and they do it with 40-hour workweeks and pets and kids all over the office. People go to their office tours and seminars from all around the world. Their new book, Joy, Inc, by Richard Sheridan, shows how they do it.

Statistics
Show me someone who claims to know the personal traits, habits, or characteristics of a good leader and I'll show you someone who flunked statistics. Most analysis is based on studies of success, and almost no one studies failure (exception: Jerker Denrell). This leads to the Halo Effect, the myth of investor alpha, the myth of the successful track record, being fooled by randomness, measuring with bias, and many other misconceptions. The world is complex, and systems are difficult to understand without simplification. As much as we want to connect cause and effect, we often do so without understanding the background rate - what would have happened by chance (or doing nothing) alone. There isn't much luck in chess, but there is a tremendous amount of luck in business. By understanding probability distributions and learning to speak a common language of risk management, we can close the gap between our distorted mental models and reality. 

A New Research Organization
Evidence-based medicine is now more than ten years old. There are several research institutions dedicated to this important work that is already changing health care. Outside of that, there are few research institutes working on evidence-based anything. One interesting group is MECLABS, and its sister company MarketingExperiments.com. They focus on the marketing-sales pipeline and converting prospects into customers. They have done thousands of studies and have developed data-driven methodologies that are far above their peers. This dual approach - a nonprofit research organization and a for-profit consulting company - is what I’m hoping to build in the long term. I hope to help governments, organizations, schools, villages, and companies of all sizes adopt the principles of business agility.

Nudge
All the research and all the evidence in the world won’t matter if you don’t change people’s behavior. People are generally unaware of their biases and are resistant to change. Once you understand what behavior you want, you have to use people’s view and values to influence them. In many cases, it’s better to try to change behavior indirectly than to try to change their minds. There are several successful examples of how to nudge people in a new direction; advertisers do it all the time. By combining the best of evidence-based decisionmaking with proven ways to modify behavior, we hope to be a force for positive change in the world.

Summary
In today's business environment, speed counts more than size. Many CEOs, when asked what their biggest fear is, say “two guys in a garage somewhere,” because they know that start-ups have agility they don't. Using new tools and methods, bigger companies can become agile too, if they can only change their cultures. It’s not impossible. By working actively with experiments and uncertainty, companies will learn to try things that don’t cost much but have potentially big payoffs. They will recognize the role luck plays in their business. They will encourage people to take risks without punishing them if they fail. 

Business Agility is just beginning. There are many challenges ahead. If any of this rings true to you, please see some of my YouTube videos to learn more. There's nothing like a small group of committed people to turn an idea into a movement. 

Business Agility Progress Meter: