I can think of no better day than Martin Luther King Day to focus on Scott Page’s new book, The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies. The book was officially published today and it provides the most penetrating and systematic exploration yet available of various forms of diversity and precisely how diversity enhances both problem-solving and predictions.
To be fair, Page is not directly concerned with exploring racial diversity (a form of “identity diversity” in Page’s taxonomy). Instead, he focuses on cognitive diversity and unpacks four “frameworks” of cognitive diversity:
- Diverse Perspectives: ways of representing situations and problems
- Diverse Interpretations: ways of categorizing or partitioning perspectives
- Diverse Heuristics: ways of generating solutions to problems
- Diverse Predictive Models: ways of inferring cause and effect
While sympathetic to the case for identity diversity, Page cautions that “. . . we can take the connection between identity and cognition too far. Identity diverse people can think alike . . . If well managed, identity diversity can create benefits, provided it correlates with cognitive differences and provided the task is one in which diversity matters.”
A blog posting can hardly do justice to the sophistication and nuance of Page’s argument. As Professor of Complex Systems, Political Science and Economics at the University of Michigan and an External Faculty Member of the Santa Fe Institute, Page is adept at crossing disciplinary boundaries to define points of view and muster evidence to support those perspectives.
Let me instead just highlight some of Page’s more provocative findings. First, in discussing problem-solving, he develops a perspective that “diversity trumps ability: random collections of intelligent problem solvers can outperform collections of the best individual problem-solvers.” To be clear, Page labels this a conditional claim that only holds under specific circumstances, but he helps us to understand why this assertion is often true.
Page then turns to predictive tasks and highlights two key conclusions from his analysis:
. . . that diversity and accuracy contribute equally to collective predictive performance, and that a crowd’s collective prediction must always be at least as good as the average prediction of a member of the crowd.
Page also looks at the role of preference diversity – “differences in what we value.” Here he makes a distinction between fundamental preferences (preferences about outcomes) and instrumental preferences (preferences about how we get what we want). In discussing the role of preference diversity, he observes:
Diverse perspectives, which we have touted as a panacea, have a dark side – they lead to the discovery of lots of possible alternatives. If people have diverse fundamental preferences, they less likely agree when they have more possible choices. On the flip side, diverse fundamental preferences, which cause so many problems when making choices, prove beneficial for problem-solving. What we desire influences how we look at problems, the perspectives we choose. Thus, collections of people with diverse preferences often prove better at problem solving than collections of people who agree. Difference of opinion not only makes a horse race, it also makes for effective, albeit sometimes contentious, teams.
This is an important point and, for my money, Page does not emphasize it enough. Diversity aids problem-solving enormously, but it also generates significant friction. We have come to believe in the business world that friction is bad but, in fact, certain forms of friction are essential to innovation. Even in the absence of diversity in fundamental preferences, people with diverse perspectives and tools and the best of intentions are naturally going to clash over potential solutions to problems that they believe are important before they converge around an answer. JSB and I have explored the growing importance of productive friction in driving innovation in both an HBR article (purchase unfortunately required) and our book, The Only Sustainable Edge.
Page’s book is enormously helpful in making the case for certain forms of diversity in enhancing both problem-solving and predictions. It is perhaps too much to ask for him to also explore the institutional implications of all of this. On the margin, Page makes some useful suggestions on this front in the final section of his book but, as he admits, it is only a preliminary start.
I left the book feeling that Page is much too sanguine about the capacity of existing institutions to embrace and reap the benefits of diversity. In stepping back from Page’s compelling analysis, I became even more convinced of the need to re-think at a fundamental level most of the institutional architectures that we have come to accept as givens – firms, schools, governments, NGOs, etc.
We are already seeing new institutional forms emerge on the periphery of existing institutions with the potential to embrace diversity in far more effective ways than most conventional institutions, whether it is in the many different forms of creation networks (e.g., open source software initiatives and process networks) that JSB and I explore here and here (registration required) or the X Prize initiative or the Creative Commons initiative or the Santa Fe Institute that Page is affiliated with, just to name a few.
Those who understand the power of cognitive diversity and even preference diversity need to explore the institutional mechanisms that are most powerful in mobilizing the appropriate forms of diversity, focusing diverse participants on appropriate problems, motivating them to engaging in collaborative problem-solving and creating the appropriate governance mechanisms to resolve disputes and facilitate convergence. On the one hand, the very meaning of institutional boundaries needs to be re-examined in a world where greater cognitive diversity yields superior insight. On the other hand, new ways to think about value appropriation become critical if new institutional forms are to become sustainable.
Page points us in the right direction, but there is much opportunity – and need - to innovate at the institutional level in order for the path to become clear. In many respects, the analogy with Martin Luther King holds up. MLK provided a compelling vision of a society that embraced the potential of all of its participants and celebrated the diversity that made American society so robust. The specific institutional mechanisms required to realize this vision may not even yet be clear, but we all had a much deeper sense of the destination as a result of his persuasive communication.
MLK had a knack for creating friction, but he also had a talent for turning it into productive friction rather than dysfunctional friction. As we confront the institutional implications of the value of cognitive diversity, we are likely to see much friction as existing institutions wrestle with the need to expand the scope of cognitive diversity. We can only hope that this will be productive friction, although it can rapidly turn into dysfunctional friction that will produce waves of creative destruction as new institutional forms replace the old.
Comments