I spent too many years using computers in my old profession. They have wonderful uses, and they helped hang on as the economy receded from Anderson. I also saw Microsoft Word change its format so that all prior versions were rendered nullities. Symantec never translated a very capable DOS database to Windows, and one that I still doubt its full potential was tapped. From this background, some of my friends call me a Luddite for my doubts in all the wonders of tech.
A 1976 Buick LeSabre with its distribution cap directly behind the fan blades left me with a general distrust of engineers. They get bright ideas, implement them, and never have to deal with their practical impact.
I prefer empiricism to idealism.
All this leads me to recommend The efficiency trap: why optimising everything can backfire, badly (The Guardian).
The hospital bed story illustrated the first one: weakening a system’s robustness by squeezing out too much slack. An empty bed can be redundant on a quiet Tuesday night, but it’s somewhere precious to put the next patient when flu cases surge. If you judge a system by its performance under ordinary conditions, you can lose the ability to cope with extraordinarily challenging ones.
A second trap is having a too-narrow definition of “best”. Take the city of Milton Keynes. Designed by urban planners in the late 1960s, it has an American-style grid system. As the city began to be built out, roundabouts rather than traffic lights became a defining feature of its road junctions; today there are about 130 of them. The roundabouts were great for traffic flow, but they came with a now-notorious cost. Have you ever taken the coach from Oxford to Cambridge? An hour in, you get great views of Milton Keynes, but a busload of queasy passengers, as the coach negotiates one roundabout after another. Slow down, swing left, curve right, accelerate. It’s hard to appreciate the seamless merging of vehicles when your stomach is churning.
Traffic flow is important, to be sure. Engineers find it convenient to optimise because it’s relatively easy to measure. Passenger comfort, on the other hand, is harder to quantify. In a case like this, optimisation gives you the best answer to the question you asked, but can deliver unwelcome answers to questions you failed to consider.
A third trap appears when attempts to optimise collide. Even if every part of a system is acting optimally, the system as a whole may not. On 6 May 2010, US financial markets gave a dramatic demonstration of the effect. In the middle of an already falling market, the S&P 500 suddenly plunged another 5% in just five minutes. The whipsaw on the upside was even wilder; within 10 minutes, the market had recovered. Hundreds of individual securities briefly traded at absurdly low prices.
Details are in the original.
I never heard of Ivan Illich, wish I had and I think What Is a ‘Responsibly Limited Tool’ in the Age of AI? (Los Angeles Review of Books) goes well with the preceding.
Illich’s stature as a public intellectual was established by a sequence of pamphlet-sized books that he published in the 1970s while he was running the Centro Intercultural de Documentación (CIDOC) in Cuernavaca, outside Mexico City. CIDOC had become a magnet for students of a countercultural stripe, as well as a crossroads for thinkers from Paulo Freire to Susan Sontag. The first of these books, Deschooling Society (1971), argued that compulsory education had become institutionalized to the point of being counterproductive, a machinic system for churning out credentialed future employees rather than independent thinkers. It remains perhaps his most provocative and influential book. But it is Tools for Conviviality (1973) that offers the more wholesale and devastating critique of life in an industrialized capitalist society.
***
What is most radical in Tools for Conviviality—and also most frustrating, because it is so difficult to envisage—is the proposal that we should limit our tools. Illich rightly calls this a “taboo.” Western society, loosely called, has harnessed its concept of progress so tightly to technology that it is almost unthinkable to call for limits to be imposed on technological development without being dubbed a Luddite. Or, at least, that was the case until recently. With today’s growing nervousness about the power of AI and its rogue capacities, that taboo is being broken. Both Anthropic’s Dario Amodei and DeepMind’s Demis Hassabis have confessed that they think the AI arms race needs to be stopped. To be clear, they are not proposing that we limit AI tout court but rather acknowledging that the frenetic zero-sum game they’re playing is forcing them and their competitors to move at a pace that abandons all checks and balances for fear of falling behind. They have admitted that they’ve lost control and—incredibly, given Silicon Valley’s libertarian ideals—are begging the government to step in and slow things down. Tech commentators, meanwhile, are giving odds on the risk of human extinction.
It might be worth pausing here for a moment to consider the challenge Illich sets us when he talks about limits to tools. Even with more prosaic technologies than neural networks, achieving a static state is potentially impracticable. Digital devices are not like mechanical machines; they are mutable by definition, designed to evolve in our hands. A smartphone may feel as tangible as a slab of basalt, but this belies a shape-shifting operating system that is hell-bent on either self-improvement or what Cory Doctorow calls “enshittification.” There is no point at which you can freeze this system at some platonic ideal. At the very least, Illich’s notion that we should put a “roof” on our technical prowess would require a degree of consensus on what is enough that is difficult to imagine reaching.
The question is, in a world of billions of autonomous agents running the show, what is the value of being human? Another way of asking this is, what is the human purpose that we are being freed up for? Manual work is still very much on offer, of course, but creativity? Aren’t the tech bros at pains to prove that AI can do that too—make illustrations, compose music, write novels? Here, I think, Illich’s concept of limits, grounded as it is in the human scale and in human agency, is exposed as somewhat technocratic. It suggests that there is a technical solution through our tools, if we could muster the collective and political will. Interestingly, Pope Leo XIV’s first encyclical, Magnifica Humanitas, published earlier this year, takes a more ontological approach, arguing that it is the sanctity of human dignity that must be protected, not human capability.
... Tools were meant to be a practical place to start. Retooling may sound like a modest proposal, but in Illich’s mind, it was potentially revolutionary and involved inverting the politics of tools in our favor. As one of his most insightful interpreters, David Cayley, puts it, “The question essentially is who’s on top, the user or the tool.”
Can we not have some common sense, please?
sch 11/10


