If I had an hour to solve a problem, I’d spend 55 minutes thinking about the problem and 5 minutes thinking about solutions.
Successful problem solving requires finding the right solution to the right problem. We fail more often because we solve the wrong problem than because we get the wrong solution to the right problem.
The world we live in has been changing at an extremely fast rate. As Sir Charles P. Snow – the prominent British novelist and scientist who also held important positions in the British Civil Service and the UK government – described it when delivering his famous lecture on “The Two Cultures” at the University of Cambridge:
The rate of change has increased so much that our imagination can’t keep up.1
The increased rate of change in the world we live in has also increased the complexities of the problems we face. Some societal problems that were once relatively easy to comprehend – if not solve – have grown to be multidimensional and multistakeholder perplexing puzzles that we might even label as our world’s chaos.
But as these complexities have grown, so have our abilities to solve problems. An important branch of science – composed of many sub-branches as we will see – that we label analytics science in this book, has been particularly vital in solving our world’s problems. This science has given us an important superpower: the means to deal with our problems in an effective manner. This superpower, in turn, has revolutionized the focus and concentration of many other fields such as public policy, orienting them more than ever toward “public problem solving.”
The media is full of fiction-based arguments claiming that the use of this superpower – given to us via advances in analytics science – will eventually destroy humanity. Arguments like this became even more widespread after the public release of generative AI tools such as Open AI’s ChatGPT in 2022 (see Chapter 2, where we will learn more about the science behind these AI tools and how they work). But the reality is that this superpower has given us – and our future generations – an incredible amount of hope in shaping a much better world. When we realize this incredible amount of hope given to us by this superpower, we better understand the colossal importance of learning more about analytics science. This book introduces not only the main ideas and tools in analytics science that have collectively given us this superpower, but also showcases numerous examples of how these ideas and tools can be used to further improve the world we live in.
As we will see, analytics science has already been a major shaper of the world and will likely continue to be so. This by no means implies that we should be naïve and assume that all future uses of analytics science ideas will be altruistic, but rather that we should learn such fundamental ideas and appreciate that there is so much to hope for in using them to effectively solve our complex problems. As Phillip M. Hauser – the pioneer American demographer who was also a president of the American Sociological Association, the American Statistical Association, and the Population Association of America – put it:
We do have the means to destroy ourselves; it is naïve to assume that the use of these means is beyond the realm of possibility. We also have within our grasp the means to deal with our problems in an effective manner.2
We will also see that analytics science would not be a major shaper of the world if it were not for the important contributions of a variety of scientists from different fields and backgrounds. The important ideas in analytics discussed in this book have been shaped by an impressive scaffolding as scientists have stood on the shoulders of those before them. This scaffolding has yielded the pillars of what we today know as analytics science. Learning about these pillars, in turn, will enable you to stand “on the shoulders of giants,” allowing you “to see further” – as Sir Isaac Newton once described his position in his 1675 letter to Robert Hooke, the English polymath.
Interestingly, however, these pillars are not made of complex materials, but only a few simple though revolutionary main ideas that are used as inputs. What are these main ideas? And how have they been able to have a major impact on our lives?
In this book, you will see some important answers. My goal, however, is not purely to answer these questions. Good science books not only educate, but also enlighten and entertain. Thus, in this book, you will also be engaged in captivating stories that illuminate the impact of analytics science on the world around you. Through these stories, my goal is to teach you how the main ideas in analytics science can be applied to solving different problems both at the societal and personal decision-making levels. This, in turn, will avoid presenting you with what Samuel Johnson called knowledge that “will make no man wise:”
As gold which he cannot spend will make no man rich, so knowledge which he cannot apply will make no man wise.3
By reading this book, you will not only learn new ideas and tools, but also unlearn what you might have learned from some other sources, including the popular media that present such ideas and tools as a major force against humanity. The many students who have taken my course at Harvard on “Machine Learning and Big Data Analytics” over the years, have not only learned the main ideas and how they can be used to impact the world (many of whom have even created social impact organizations using such ideas), but have also learned to unlearn their prior beliefs. Learning to unlearn, I believe, is an important step toward growth. As Ronald D. Liang – the prominent British psychoanalyst – put it in The Politics of the Family and Other Essays:
We must continually learn to unlearn much that we have learnt, and learn to learn what we have not been taught. Only then do we and our subject grow.4
The Analytics Revolution in the Public and Private Sectors and the Age of Artificial Intelligence
Ancient Greek philosophers believed that the pursuit of man includes four aspects: scientific (the pursuit of truth), political–economic (the pursuit of power and plenty), ethical–moral (the pursuit of goodness and virtue), and aesthetic (the pursuit of beauty).5 You will see in this book that analytics science has contributed to all of these aspects (perhaps with the exception of aesthetics). It has contributed via an important revolution that has made its use more ubiquitous than ever. Analytics science was once limited to universities and a few private sector labs. But the analytics revolution changed the story, making it possible for both the private and public sector to make use of data, advanced algorithms, and often vast computational power to get engaged in problem solving. This revolution has also made it possible to live in an era that we might label as the age of artificial intelligence (AI).
However, as you will see in this book, AI is only one of the branches of analytics science. More importantly, various steps in advancing AI and its related fields such as machine learning (ML) would not have been possible if it was not for the developments in other branches of analytics science. Understanding the scaffolding of analytics scientists and their main ideas, thus, is important in gaining a better understanding of the impact of analytics science in our era – the age of AI.
Finally, while AI and ML have been major contributors to solving some societal problems and improving the world, they are not alone within the analytics realm. If we want to understand how analytics science can and has improved the world, we must also step outside the AI and ML fields. Without seeing the broader picture of analytics science, we cannot understand the range of possibilities for solving our societal problems.
Insight-Driven Problem Solving: Using Analytics Thinking in Solving Complex Problems
To better understand the role of analytics science in creating a better world by solving our societal problems, we must first grasp the roots of analytical thinking in problem solving. René Descartes (Figure 1) – the seventeenth-century French philosopher, scientist, and mathematician who revolutionized much of Western thought – is perhaps best known for declaring cogito ergo sum, meaning “I think, therefore I am.” Less known are his important contributions to problem solving. He advocated for a new paradigm in problem solving that emphasized the reliance on empirical evidence, logic, and analytical reasoning. His approach demanded a scientific resolution to solving problems which entailed diligent study and meticulous analysis. He was one of the proponents of what we know as reductionism. In his own words, to solve difficult problems, we must:
Divide each difficulty into as many parts as is feasible and necessary to resolve it.
Figure 1 René Descartes (1596–1650).
Analytical thinking is firmly tied to the doctrine of reductionism. As Russell L. Ackoff – an American scientist and a pioneer in the field of operations research, management science, and systems thinking – put it in his 1974 book Redesigning the Future:
Analytical thinking is a natural complement to the doctrine of reductionism. It is the mental process by which anything to be explained, hence understood, is broken down into its parts. Explanations of the behavior and properties of wholes were extracted from explanations of the behavior and properties of their parts. The temperature of a body, for example, was explained as a function of the velocity of the particles of matter of which it was composed. An automobile’s behavior was explained by identifying its parts and explaining the behavior of each and the relationship between them.6
Analytical thinking is also what arguably resulted in the first ideas for creating a machine capable of problem solving – what we know today as a “computer.” The analytical engine which is generally considered as “the first computer” was proposed in 1837 by Charles Babbage – the English polymath and inventor who is referred to as “the father of the computer” by some scholars.
However, the great mind behind making it useful for problem solving was Ada Lovelace (Figure 2, left), who worked with Babbage and was the first to understand that the analytical engine had capabilities far beyond pure calculations and number crunching. Modern analytics science would perhaps be far behind its current state had she not realized this prominent capability and not written in her notes that the analytical engine “might act upon other things besides number.” With her goal being “the illustration of the powers of the engine,” she also gifted us with a diagram (Figure 2, right) that is perhaps the first example of the potential use of a computer for an analytical purpose.
Figure 2 Ada Lovelace (1815–1852) and her diagram of the computations behind the analytical engine.
In this book, besides learning the main ideas in analytics science, you will see various real-world examples demonstrating how such ideas have enabled policymakers or other decision-making authorities to solve important problems and move the world to a much better state for its inhabitants. These examples collectively showcase what we might call insight-driven problem solving.
To better understand what insight-driven problem solving refers to, we should first note that solving societal problems often requires understanding various aspects of the complex phenomena at hand. Because not all such complexities can be included in the analyses, one needs to use the doctrines of reductionism and analytical thinking to divide the problem into as “many parts as is feasible and necessary to resolve it.” Reduction must be done at a suitable level – not so much as to compromise the integrity of the problem and not so little as to lose hindsight. Reductionism at the right level allows analytic scientists to obtain insights that are powerful in understanding the underlying phenomenon and, hence, offering useful solutions. This process of obtaining valuable insights through reductionism at the right level, and making use of them to solve real-world problems, is something that even most advanced AI systems currently lack. As we will see, such systems are developed primarily based on methods that rely entirely on learning from data (a process that we might call data-driven problem solving), which, as we will learn, is neither sufficient nor necessary for effectively addressing complex societal problems. Insight-driven problem solving, thus, is an ability that only belongs to human experts (at least at the time I am writing this passage). However, like many other uses of AI, benefiting from AI can significantly enhance humans’ ability in insight-driven problem solving.
The obtained insights from reductionism at the right level can also inform simple but useful regulations that are often required to manage various aspects of our complex world. We can, for example, think of the 2008 global financial crisis. While you might think more complex regulations would have helped, what the modern finance industry needs is, indeed, fewer complex regulations formed by powerful insights. Avoiding complexities can, and often does, go a long way. In a speech in 2012 at the Federal Reserve’s annual policy conference, Andy Haldane – the director of financial stability at the UK’s central bank (Bank of England) at the time – emphasized this, stating that what the complex finance world needs is simple but insightful regulations as opposed to complex rules:7
Modern finance is complex, perhaps too complex. Regulation of modern finance is complex, almost certainly too complex. That configuration spells trouble. As you do not fight fire with fire, you do not fight complexity with complexity.8
The influential analytics scientists you will meet throughout this book have all had a common ability: obtaining powerful insights capable of solving complex problems using insight-driven problem solving. In essence, they knew that they should not take refuge in complex solutions that lack useful insights. When finding a solution to a problem, they often chose the simplest and most insightful one. In Haldane’s terms, they knew not to “fight complexity with complexity.”
Fighting complexity with complexity is a common mistake that, unfortunately, we often see among many eager young individuals who genuinely strive to improve the world using analytics science. In deriving actionable insights that can improve governments’ economic policies and solve the underlying problems, for example, simple models such as the dynamic stochastic general equilibrium (DSGE) have been made significantly more complex over the years. As a result, they have become less insightful and actionable for policymakers and other users, losing their problem-solving purpose. Joseph Stiglitz – an American economist and recipient of the Nobel Prize in 2001 who also served as senior vice president and chief economist of the World Bank – highlights this, describing that the added complexity, or as he puts it the “Ptolemaic attempt to incorporate some feature or another that seems important that had previously been left out,” has been counter-productive:
The result is that the models lose whatever elegance they might have had and claims that they are based on solid microfoundations are weakened, as is confidence in the analyses of policies relying on them. The resulting complexity often makes it even more difficult to interpret what is really going on.9
A key idea here is leveraging the Occam’s razor principle. This principle in problem solving, attributed to William of Ockham – the fourteenth-century English philosopher and theologian – recommends searching for solutions obtained from the smallest possible set of elements. This principle, which in American English is more widely known as “KISS (Keep It Simple, Stupid!),” is vital for successful insight-driven problem solving.
For example, although several decades have passed since the civil rights period, the USA continues to be a residentially segregated society. According to a report by the Othering and Belonging Institute at the University of California, Berkeley, 81 percent of the metropolitan regions in the USA that have more than 200,000 residents were more segregated in 2019 than they were in 1990.10 Segregation, whether geographical, educational, or legal, can harm societies in various ways. Thus, understanding the reasons behind segregation and what needs to be done to address it is of utmost importance. As you will see later in this book, Thomas Schelling – an American economist who later won the Nobel Prize in 2005 – addressed this problem by following the Occam’s razor principle and using insight-driven problem solving: He proposed a simple analytics science model capable of generating important actionable insights. This is just one of the numerous examples you will see in this book where insight-driven problem solving powered by analytic thinking has enabled scientists to address complex societal problems. This has also enabled analytics scientists to inform policymakers about actionable insights to curb COVID-1911 and future pandemics, address the climate change crisis, and make healthcare more accessible for everyone, all of which are complex societal problems that our world faces.
This method of solving real-world complex problems is different from other methods of problem solving. For example, for a plethora of reasons, methods used for solving puzzles are often not suitable for solving real-world complex problems. For starters, unlike real-world problems, a puzzle is often defined as “a problem that one cannot solve because of a self-imposed constraint.”12 More importantly, in solving what is typically called a puzzle one does not need to generate insights. What is needed is a clear solution. As an example, when in 200 BC, Zenodorus tried to solve Dido’s Problem – a problem based on a passage in Virgil’s Aeneid in which the goal is to find the figure bounded by a line that has the maximum area for a given perimeter (or the maximum “land as they could enclose with a bull’s hide” as appeared in Virgil’s Aeneid)13 – he did not need to derive actionable insights. He had to come up with a clear solution.
Similarly, when the great John von Neumann – a true genius and one of my own role models in analytics science, whom you will meet several times in this book – was challenged by a questioner to solve a puzzle, he did not need to derive insights. The questioner wanted him to figure out the total distance that a fly would cover in a specific scenario: Two bicyclists start twenty miles apart and head toward each other, each going at a steady rate of 10 miles per hour. At the same time, a fly that travels at a steady 15 miles per hour starts from the front wheel of the southbound bicycle and flies to the front wheel of the northbound one, then turns around and flies to the front wheel of the southbound one again, and continues in this manner till it is crushed between the two front wheels.14 Von Neumann solved it instantly and replied, “15 miles” (see Chapter 1 for details of his solution).
In addressing complex societal problems, unlike the case with puzzles, we need to generate actionable insights and look at how they can improve the situation. The main role of an analytics scientist is to generate such insights. Insight-driven problem solving in this vein also has another important advantage: Since the actionable insights are driven by a simplified version of the problem using the doctrine of reductionism – employing what we will define as a model in Chapter 1 – they can be easily communicated to the decision-makers, improving the chances that they will be implemented (more on this in Chapter 1).
The difference between insight-driven problem solving and other methods of problem solving can also be understood by noting that what we typically comprehend as a societal problem relates to experiences of a problematic situation, not problems per se. The American philosopher, psychologist, and educational reformer John Dewey emphasized this important role of analyzing a phenomenon. As Russell L. Ackoff put it:
[John Dewey] argued that decision makers have to extract problems from the situations in which they find themselves. They do so, he said, by analyzing the situation […] What we experience, therefore, are problematic situations, not problems, which like atoms and cells, are conceptual constructs.15
Analyzing a phenomenon using insight-driven problem solving, however, can be detrimental if an important prerequisite is not met. This prerequisite is avoiding solving the wrong problem. The quotes that begin this introduction highlight the importance of this crucial step. They also remind us that avoiding solving the wrong problem, which can occur in various ways, is often the most difficult and time-consuming step. This is mainly because one needs to spend the time and effort required to fully comprehend the problem, thereby avoiding making wrong assumptions before attempting to solve it.
The importance of spending time on understanding the problem and avoiding making wrong assumptions was perhaps best described by Kenneth E. Boulding – an English-born American economist, educator, and interdisciplinary philosopher. In his book Economics as a Science, Boulding discussed a joke – often labeled as “assume a can opener” – which President Ronald Reagan rephrased and used in his remarks to the students and faculty at Purdue University in 1987:
[An] economist, a chemist, and an engineer were stranded on a desert island. And between them they had only a single can of beans, but no can opener. The engineer suggested that he climb a palm tree to a precise height, then throw the beans at a precise distance, at a precise angle. “And when the can hits,” he said, “it will split open.” “No,” said the chemist. “We’ll leave the can in the sun until the heat causes the beans to expand so much the can will explode.” “Nonsense,” said the economist. “Using either method we’d lose too many beans. According to my plan, there will be no mess or fuss and not a single bean will be lost.” Well, the engineer and the chemist said, “Well, we’re certainly willing to consider it. What’s your plan?” And the economist answered, “Well, first assume we have a can opener.”16
This “assume a can opener” joke, though at the surface about an economist, is much broader. In solving problems using analytics we quite often see two related sets of issues: Either a wrong assumption is made about the problem, or a wrong tool is used to solve it. This book will help you avoid both of these inimical pitfalls.
The Impact of Analytics-Based Insight-Driven Problem Solving on Shaping Our World
When the assumptions made about the problem are not wrong, and the right analytics tools are used to solve it, insight-driven problem solving can go a long way. In fact, as you will see via various examples in this book, it might even end up shaping the world we live in.
Consider, for example, the work of Mario Molina and Frank Sherwood Rowland – two chemists who won the Nobel Prize in Chemistry in 1995. Starting with a paper that they published in 1974 in Nature,17 they discussed their insights from their analysis: The more chlorofluorocarbon (CFC) gases we emit into the atmosphere, the higher the threat of damaging our planet’s ozone layer.
This simple, yet important, insight was based on complex chemical analysis. But it resulted in policymakers taking action to solve an important problem, namely, finding ways to protect the Earth’s ozone layer. In 1978, the US Environmental Protection Agency (EPA) issued the first ban on CFC-based aerosols, stating that:
Starting October 15, 1978, manufacturers of bulk fluorocarbons can no longer make them for use in most aerosol products.18
International efforts also followed. The Montreal Protocol, signed on September 16, 1987, formed an important international agreement to limit the use of CFCs. The International Day for Preservation of the Ozone Layer, also called “Ozone Day,” is celebrated annually on September 16. It was designated by the United Nations to commemorate the day that the Montreal Protocol was signed.
Finally, while the focus of the book is on large-scale, public, and complex problem solving using analytics science, you should note that learning the basic ideas discussed in this book can also help your personal growth. One main reason is that analytics science is an undeniable part of the information economy we live in, which mainly rewards labor with analytically oriented skills. Thus, if you possess the analytics science tools that enable you to execute insight-driven problem solving, you will be highly attractive to many giant employers of our era both in the public and private sectors. You can even become an entrepreneur and start your own business; after all, most successful entrepreneurial ideas are about problem solving. If you doubt this, look more carefully into the most successful entrepreneurs and you will realize that their main idea has been about solving a specific problem. Larry Page and Sergey Brin founded Google to solve the problem of efficiently finding relevant results in an ever-growing universe of online data using a specific method that we will learn later in this book. Amazon was founded by Jeff Bezos to solve the problem of limited access and selection in traditional brick-and-mortar bookstores. Uber was founded by Travis Kalanick and Garrett Camp to solve the problem of inaccessibility and inefficiency in traditional taxi services. And Microsoft was founded by Bill Gates and Paul Allen to solve the problem that personal computers were facing, namely the lack of user-friendly operating systems and productivity tools.
Summary: What This Book Offers You
In a nutshell, your chance of becoming a successful entrepreneur, a highly sought-after individual in our information economy, or someone who will be remembered for many generations for finding a solution to a lingering societal problem (like many of the heroes we will meet) will significantly increase if you learn the main ideas of analytics science discussed in this book.
If you are interested in learning ways to make an impact on the world and are curious to know the range of possibilities that analytics science offers, you have picked the right book. So let’s begin the journey. Welcome aboard!
