Kindle. The premise of this book is that mathematical models not only can be, but are, very damaging to society. O'Neill aims for a Al Roth-style enumeration of their key flaws, which I think are:
- people being unaware of the model or the uses to which their data are put;
- feedback, in the sense that the model may reinforce its own assumptions; and
- scaling out, the capacity to grow exponentially.
Unfortunately there is no mathematics in the main section of this book, and moreover most of O'Neill's complaints hold even of non-mathematical models; such models only intrinsically make things more efficient, not better or worse. Given that flaws in general systems have been canvassed at length already (see, for instance, the venerable comp.risks), her only scope for novelty is to hammer the vacuum of values in current-day U.S.A. But perhaps, as usual, I am not her target audience, or the "mathematics is morally neutral" meme has taken on Nuremberg overtones, or even more likely, I'm an outlier.
I'd go further and claim that one is better off contemplating the pathologies of general systems, however realised: simply marry John Gall's Systemantics with the McNamara fallacy and you have a whimsical but soundly provocative and fecund account. (See Matt Levine for one such synthesis.) For the more technical, perhaps Mirowski's Machine Dreams and thereabouts is more persuasive. For Generation Y, try Kobek: is there any reason to think that libertarian geeks would aim for anything other than what we now have?
O'Neill is not careful to separate out the data modelling from the control aspects, nor the various kinds of feedback in systems. On the former, consider a lone researcher cooking up the perfect machine learning system. In many ways this is innocuous as they have no power to influence the world; it is almost a purely descriptive activity (up to the researcher's own biases, of course; as with all science, there is always the question of what to observe, and more generally, choice of ontology, logic, etc.). Conversely, consider exactly the same hooked up to the systems of government, or Facebook: it may now do immense damage, or perhaps even something worthwhile. The difference is in how much and what kind of control is exerted, not (just) the model built. This is a gap many data scientists can fit their morals into.
As for feedback, she finds it offensive that some systems sometimes become self-justifying in pernicious ways, as they can exert pressure on their inputs to optimize their outputs with respect to the control criterion (see, for example, the just-mentioned post by Matt Levine on the recent Wells Fargo fiasco). For instance: poor people tend to have poor credit scores, which makes it harder for them to finance things that might them get out of poverty, thereby reinforcing their poor credit rating. That the finance outfit therefore potentially misprices risk is beyond the scope of the model. Conversely feedback is used to train the models in the first place, which we might call "evidence based policy" in another setting. This leads to a point she doesn't quite make: modelling is an essentially reactionary activity, an attempt to make the future conform to the past (for otherwise the model is in error, or the control too weak, which leads to another round of optimization; witness Matt Levine on index funds).
So, is there anything more to this book? Well, maybe. She was apparently horrified that outfits like DE Shaw gouge their profits out of "dumb money" pension funds and so forth. I'm more sanguine about that: market access is cheaper than ever for institutional investors (according to institutional investors), and really, this is simply the markets teaching dumb money the expensive lesson of needing to be either less dumb or not there. I have more sympathy for the argument that (small groups of) individuals cannot manage risk adequately over the long term (say lifetimes) and that the government should take an active role there, as it has in generations past. O'Neill (Chapter 10) observes that by showing different ads to different constituencies, common knowledge about political candidates decreases, which splinters democracy. I agree with her, but really, this happens with or without mathematical models simply because of people's priors (selective hearing). Sure, exacerbation, I get it.
In Chapter 5, O'Neill takes the "broken windows" fallacy to task, just as the Freakonomics boys did a decade ago. I got a little excited to see her propose a platinum-rule style of policing: roughly, "treat others as they wish to be treated", and specifically have the police maintain the standards of each community, not getting too far ahead or behind those. (Sounds like ... England! If you're sufficiently English.) The multifacted identity she pushes in the conclusion is old hat to, for instance, greybearded econo-moralists like Amartya Sen, who would probably have been accused by the O'Neill of 1975 of cybernizing society, what with all his mathematics and all.
Ultimately I didn't learn much here. I already thought that modelling merely promotes the normative, and is extremely illiberal therefore. She doesn't take models to task for being opaque and lacking explanatory (and not just predictive) force. There are far richer accounts of the history of operations research out there. She mostly argues from authority. Perhaps there's more meat in the endnotes. I would have been less disappointed if I'd read more of her blog; for instance this post makes it seem she has a narrow experience of the world. David Runciman writes at length on why this might be, despite O'Neil's extensive education.