Book of Why

The Book of Why is a popular introduction to Judea Pearl’s branch of causal inference. But it is also so much more.Pearl has written many other textbooks introducing his graphical approach. But in this book, Pearl provides an engaging narrative of the history of causal inference, the important distinctions he sees in his branch and its importance for the future of Artificial Intelligence.Briefly, Pearl views classical statistics as seriously flawed in not having developed a meaningful theory of causality. While able to demonstrate correlation, Pearl asserts that in classical statistics all relationships are two-way: that is 2x=3y+6 can also be written 3y=2x-6. We are left in doubt as to whether x causes y or y causes x.Fundamentally, Pearl sees this problem as still plaguing all artificial intelligence and statistics. In its place, Pearl argues that the exact causal relationship between all variables should be explicitly symbolized in graphical form and only then can mathematical operations tease out the precise causal effect.To be transparent, I am trained in the Rubin approach to causal inference and disagree with some of Pearl’s history and characterization of statistics. But that is not the point. The history is well-written, engaging and understandable by the lay reader. Similarly, his account of graphical causal inference theory is followable even for someone like myself who did not learn these techniques in graduate school.The last part of the book, where Pearl opines on the future of AI, is the most sensational. Pearl believes that if computers were programmed to understand his symbolization of causal inference theory they would be empowered to realize counterfactuals and thus engage in moral decision making. Furthermore, since Pearl himself was a pioneer in deep learning, his characterization of contemporary AI as hopelessly doomed in the quest to replicate human cognition because of a lack of understanding in causal inference will be sure to garner attention.But one would be misguided to think that speculations about AI or mischaracterizations of other kinds of causal inference make this book any less of a classic. For the first time, Pearl has written a popular, interesting and provocative book describing his branch of causal inference theory—past, present and future.This book is a must read then, not only for causal inference theorists, but more widely for those with any interest in contemporary developments in computer science, statistics or Artificial Intelligence. A book that, like Kahneman’s Thinking Fast and Slow, is a triumphant summary of a lifetime of work in scientific topics that have ramifications, not only for fellow scientists, but for all of humanity.

Someone recommended this book to me based on my interest in causality, specifically Goldratt's thinking processes and his change question sequence (why change, what to change, what to change to, how to cause the change and how to measure and sustain the change). I had long thought AI was snake oil, with its use in areas where it shouldn't be used. Pearle describes the problem with AI in the title, The Book of Why. AI tries to predict upward from correlations. One must down determining why a problem exists and the assumptions around that cause, not predict blindly from a lack of understanding of the underlying system's interrelationships. I attended an academic conference, and as you might imagine, the hot topics were AI and data analytics. There was no research question, no propositions and conclusions, no working hypotheses, just very large data sets and the application of numerous statistical models to determine what might be correlated. DATA DREDGE! Much of Pearle's book was above my head, but I suggest that he study Goldratt's thinking processes and his categories of legitimate reservation (rules of logic) and teach readers to build a system model of the environment before applying AI. I believe AI is a powerful new methodology, BUT I fear we will have decades of misapplication and wasted brainpower studying the wrong problems where simple logic would be a better alternative when applied using the question of WHY to dive deeper and deeper into our understanding of causalities before blindly applying AI. Great read for me. I applaud any author who wants to ask WHY like a scientist does. Please check out TOC for Education, where kids learn simple logic tools to improve their lives.

It is doubtful that Professor Pearl is at all surprised by the polarity in the reviews of this book. I imagine, in fact, he has a slight smile on his face. This is a man that clearly does not cower from a debate.To me this is not so much a book about science but a book about statistics, which are used almost universally. Many of his examples involve science – hard or social is irrelevant – because that is the world he knows. My world is business, and I can tell you from experience that everything he says about the disregard for causality and the limitations of linear statistics using data alone is spot on.The book covers many fronts but the overarching theme is causality. Why? When we investigate cause and effect how do we know that we have reached the right conclusion without challenging that conclusion, both intuitively and using the tools of mathematics?One of the great myths of science today is that we have conquered the causality problem. We haven’t. Most scientific discoveries are ultimately proven wrong, or at least incomplete. Major drug studies cannot be replicated. And peer review alone – the gold standard of proper science – is not, by itself, any guarantee of truth. In a recent study of scientific papers published on COVID-19, all of which were peer reviewed before being published in prestigious journals, the researchers found that a surprising number ultimately had to be retracted.In my world, the world of business, the results are both staggering and a bit horrifying. A large percentage of students graduating from university today with an interest in business have degrees in something to do with data: data mining, data analysis, Big Data. Data is the new marketing.If you want to launch a program or make an investment you must first make the “business case.” That means you must create a statistical case, almost always based on data. Unfortunately, these cases are often wrong and businesses continue to make bad investments.The preoccupation with data is based on the belief that “data are facts.” But that’s only partially true. Data are facts only in a specific context. And there can be an infinite number of contexts in the real world, a world that is constantly changing.With data in hand, people are no longer asking why. They are no longer even bothering to access their intuition to ask what they might be missing. Intuition, in fact, has become a dirty word, something akin to voodoo or folklore.When it comes to AI, Professor Pearl notes that we are not as far along as many people assume. We are decades away from AI that is even remotely humanlike. Because, as Pearl notes, machines cannot imagine what isn’t. They cannot ask why at even the simplest level.Yet humans, even young children, do it all the time. At least we used to. Which is why I don’t believe we will ever create AI that is humanlike. We don’t yet understand how or why humans think intuitively and what prompts us, or allows us, to imagine alternate realities. How can we teach machines to do it?We can only use algorithms, piled one on top of the other, to calculate a probable answer. And while machine learning can make these algorithmic machines incrementally more accurate, I do not wish to defer to an incrementally more accurate answer when it comes to the big issues of life and society. Or my health.It has been widely reported, for example, that the engineers of Google are no longer entirely sure how their search engine works. It’s too complex. Which is why modifications are not just calculated and applied. They are tested first, on a large test database, to see what results they get. Those results are then reviewed intuitively to see both if they make sense and are what the engineers expected.And that is how we should treat all statistics. Why? Why? Why? Professor Pearl has given us some tools to help in the process. But he has not given us a final solution, as even he admits. Nonetheless, he has moved us down the path. His methods still require assumptions and work largely in the world of probabilities.This book will be a tough read if you are uncomfortable with mathematics. And there are a lot of models and formulas that will be impossible to decipher if you don’t speak the language of mathematics. In every case where he offers a formula, however, he explains what it says, so that while he admits a personal fondness for formulas you can really just ignore them and still get a lot from this book.He is a little harsh, however, regarding other people in the scientific world, past and present, some of whom have obviously offended him in the past. I found that a little off-putting, which is the only reason I didn’t rate the book a 5. Nonetheless, this is an insightful book by a passionate man and I believe I invested my time wisely in reading it.