The Unaccountability Machine

Why Big Systems Make Terrible Decisions - and How The World Lost its Mind

Back to the full list of books.
More about The Unaccountability Machine.

Highlighted Quotes

Location 55: ‘the regrettable tendency of complex systems to have opaque and volatile dynamics’.
Location 65: the industrialisation of decision-making
Location 229: it can speak to you with the anonymous voice of an amorphous corporation, but you have to talk back to it as if it were a person like yourself.
Location 242: The fundamental law of accountability: the extent to which you are able to change a decision is precisely the extent to which you can be accountable for it, and vice versa.
Location 244: For an accountability sink to function, it has to break a link; it has to prevent the feedback of the person affected by the decision from affecting the operation of the system.
Location 249: every rule is a model of the world – you can see both a model and a rule book as a relationship between causes and effects, inputs and outputs.
Location 285: it is neither psychologically plausible nor managerially realistic to expect someone to follow orders 99 per cent of the time and then suddenly act independently on the hundredth instance.
Location 292: That’s the purpose of making policies – to reduce the amount of time and effort spent making decisions on individual cases.
Location 303: ‘nobody to blame’ is the definition of what constitutes an accountability sink.
Location 450: the idea of tricking the public into blaming a global institution in order to solve a domestic political dilemma might seem quite clever,
Location 454: Just as accountability is psychologically intolerable to the decision makers, its absence is intolerable to the subjects of their decisions.
Location 462: There is a fundamental distinction between ‘kings’ and ‘priests’.
Location 463: it might be inadvisable to do so, but if you can change the king’s mind you can change the decision.
Location 467: A lot of the discontent in the modern world might come from having taken decision-making structures that were designed with ‘king-like’ leaders in mind, and handing them over to managers who didn’t act in the same way.
Location 479: If the ‘fundamental law’ is correct to link accountability with the possibility to change things, then the only way to bring accountability back into the system is to increase the ability of individual human beings to make exceptions to policies. And that’s often bad; many of these policies were brought in specifically because the individual human beings were being capricious, unfair, even racist.
Location 482: It’s good that people can understand what’s going to happen by consulting a rule book, rather than having to guess what somebody else is going to decide.
Location 488: Many of the things I’ve identified as ‘accountability sinks’ could just as easily be called ‘the rule of law’.
Location 492: To this extent, the law absolutely fits the definition of an accountability sink – it’s a system which is meant to provide decisions that are not attributable to an identifiable person, and which are not alterable in response to feedback from those affected by them. You can’t argue with a judge.
Location 537: the technological revolution and the need for social adaptation,
Location 567: Beer would identify it as one of his fundamental axioms: ‘It is not necessary to enter the black box to understand the nature of the function it performs.’
Location 599: The equilibrium concept and its identification with the idea of an optimal state are the basis of economics today.
Location 601: The fundamental particles of mathematical economics, however, are commodities, preferences and people. Treating these as identical and interchangeable is a modelling assumption, not a fact;
Location 621: ignorance is a kind of information-processing system of last resort).
Location 622: Where analysis fails, ideology steps in,
Location 659: cybernetics was the study of the control of systems – or specifically, the study of the control of systems with enough internal complexity for there to exist an interesting problem of how to manage them.
Location 694: ‘In spite of the fashionable worship of such things as Operational Research, Automation, Cybernetics . . . it is believed that the iron-works will be one of the last places where the practical man will be king.’
Location 721: he takes a consistent approach – there is nothing to be gained by opening the black box. If you are trying to understand a system, all you can do is observe its behaviour, and whatever you can learn from doing so is all there is to know about it.
Location 879: If the economy is an information-processing system, does that mean that every corporation is an artificial intelligence?
Location 898: The science fiction writer Charlie Stross, for example, described corporations as ‘very old, very slow AIs’ in 2017.
Location 919: ‘the Chinese room argument’.
Location 946: ‘decision-making systems’ rather than ‘artificial intelligences’.
Location 947: Corporations are systems, and they make decisions, so they’re decision-making systems.
Location 959: put it this way:
Location 959: Personal Note: Change for one is hard...two difficult ....many gets insane.... #Forks
Location 959: When there are only two parts joined, so that each affects the other, the properties of the feedback give important and useful information about the properties of the whole. But when the parts rise to even as few as four, if every one affects the other three, then twenty circuits can be traced through them; and knowing the properties of the twenty circuits does not give complete information about the system. Such complex systems cannot be treated as an interlaced set of more or less independent feedback circuits, but only as a whole.
Location 964: when you have to think about connections between parts, rather than just counting the individual parts of a system, the number of possibilities grows very rapidly; the potential combinations multiply, rather than just adding up. Very quickly, they multiply up to astronomically huge numbers, spelling absolute death to any hope of knowing the entire state of the system.
Location 973: Knowing a great deal of detail about a subset of a system has a habit of increasing your confidence in your opinions disproportionately from their reliability.
Location 974: Personal Note: The bossman knows best…? #forks
Location 985: working inside a corporation (or any large organisation) is the quickest way to realise that you have only a partial understanding of how it works.
Location 988: The daily grind of working life is the selection of the option that looks least obviously disastrous, according to a set of criteria laid out in a plan that was produced elsewhere.
Location 992: ‘Complex systems cannot be treated as an interlaced set of more or less independent feedback circuits, but only as a whole.’
Location 995: The extent to which individual accountability can exist in a system is dependent on the extent to which you treat it as a black box, which you do when something is too complex to deal with any other way.
Location 1014: Systems don’t make mistakes – if they do something, that’s their purpose.
Location 1034: They’re unsatisfactory because an inquiry is partly an attempt to investigate causation in a system too complicated to understand, and partly because it’s an attempt to assign responsibility where it’s hardly applicable.
Location 1108: if a decision-making system isn’t a human being, you can’t assume that its system of priorities is going to be comprehensible or justifiable.
Location 1176: explainability only becomes a problem when you’re giving bad news.
Location 1195: The people in the system are good, but the system itself is not. This happens a lot.
Location 1197: If nonhuman decision-making systems exist, can they have mental health problems? And can they be treated?
Location 1227: ‘explainability’, for all that it’s seen as an important ethical goal, is practically a contradiction in terms when applied to a system that you also want to see as intelligent.
Location 1233: The process of explanation is intrinsically demystifying – it’s the means by which you explain how the conjurer put the rabbit into the hat and then pulled it out.
Location 1254: Kenneth Boulding, ‘The Economics of Knowledge and the Knowledge of Economics’, 1966
Location 1343: Even a Rubik’s Cube has more than 43 quintillion possible states; clearly a brain or an organisation has far more.
Location 1384: information as the difference between order and chaos.
Location 1476: vetoing a situation as unacceptable is a very informationally ‘cheap’ action.
Location 1520: Decision-making systems break down if the variety of their environment is not matched by the variety of their means of regulation.
Location 1577: Diagrams present you with the information ‘all at once’ and leave you to work out the flow of cause and effect for yourself, while a verbal explanation usually presents you with the story of cause and effect and leaves you to remember the connections.
Location 1580: In the context of systems, where feedback is ubiquitous, the relationships are vital and the flow of cause and effect has no obvious start or end point, it’s not hard to see why people draw diagrams.
Location 1622: variety engineering,
Location 1659: So if we’re analysing an army at war, System 1 is the platoon, System 2 is the quartermaster and System 3 is the battlefield commander.
Location 1718: identity, philosophy and purpose are tools of information management
Location 1730: That solution is to say that information and action are one and the same; variety coming in from the environment, or being transferred from one system to another, only counts as ‘information’ if it has a causal role in decision-making. Otherwise it’s just ‘data’
Location 1784: a viable system is made up of operations, regulation, integration, intelligence and philosophy.
Location 1790: The reason that these systems exist is because dealing with the variety and chaos of the world, combined with the interaction of all the parts of a large organisation, is something which generates unimaginably huge amounts of information. Most of that information has to be dealt with locally, as it arises, rather than transmitted to a central planner.
Location 1804: the kind of messages sent by the red handle in a train driver’s cab. As well as a physical check on the individual vehicle, the emergency brake sends an organisational message to the whole railway, informing it that a piece of track can’t be used.
Location 1807: pulling the red handle is how he or she indicates to the signalling system that something is threatening catastrophic or unacceptable consequences to the viability of the system, and that it requires immediate action.
Location 1813: Cutting out the channels for red handle signals is, unfortunately, a way of constructing an accountability sink and economising on bandwidth at higher levels of management, so it tends to happen when those systems are under pressure.
Location 1893: Beer made the point that like the cat, you will often see a ‘decerebrate organisation’. (His typical example was a university.)
Location 1894: An organisation in this situation is one that has, for one reason or another, stopped paying attention to some kinds of information. It’s only aware of its immediate surroundings – this quarter’s revenue, the current staffing level, things like that – and has lost the ability to make plans.
Location 1899: I’ve worked in a few organisations of this sort. I’ve also been in the even more terrifying situation of working for a company which is actively in the process of decerebrating itself. When you see this happening, update your resume and start calling recruiters.
Location 1909: To the extent that any of it was designed at all, this world was designed by economists.
Location 1918: When non-human decision-making systems become pathological in society today – most characteristically, when decisions are made which have disastrous long-term consequences as a result of relatively trivial short-term cash savings – the pathology is often directly related to something that seemed like a good idea to an economist.
Location 1923: Being an economist has been called a style of rhetoric and a way of thinking as much as an academic discipline.
Location 2032: Consequently, plenty of economists still think that making markets more competitive is a viable substitute for anti-discrimination laws. It works in the model, as long as you throw away any information that might be relevant to the actual problem.
Location 2046: Proving things ‘in the model’ and then acting as if they’re true in the world is a terrible habit of economists.
Location 2058: the appeal of economics as a means of reasoning has always been the straightforward logical line from justifiable assumptions to actionable conclusions.
Location 2117: the equations of constrained optimisation oblige us to pretend that every corporate value or priority is really a complicated expression of beliefs about profits.
Location 2130: economists generally end up making some quite questionable assumptions.
Location 2136: fitting time and uncertainty into a framework that wasn’t really designed for it.
Location 2150: economists tend to deal with the problem of strategic uncertainty by pretending it doesn’t exist. And this is how we get Homo economicus, a theoretical creation who not only cares about nothing except price and quantity, but who is also perfectly informed about future supply and demand schedules.
Location 2157: economists don’t do methodology.
Location 2158: If they did, they’d recognise that what began as a modelling strategy has become a factual assertion about the world.
Location 2208: Together, these things contributed to a sense that the market economy was a kind of computing pond; it organically grew optimal solutions to problems and, like a true artificial intelligence, you got out more than you put in.
Location 2224: ‘Market outcomes’ are a huge accountability sink,
Location 2225: Policymakers and their advisors have internalised the belief that the market is a greater intelligence than anything humanity can offer.
Location 2321: economists don’t do accounts
Location 2331: it gives you the ability to camouflage a decision as a necessity – which is a great accountability sink.
Location 2343: The thing that makes Ricardianism a vice is its tendency to prove things in a stylised model, and then act as if they were obvious truths about the real world.
Location 2404: you have to learn about the numbers by looking at the business rather than vice versa, but a lot of people try to do it the wrong way round.
Location 2422: The purpose of financial reporting is to summarise the performance of a business, not to reproduce it in every detail.
Location 2488: almost any cost is variable if you’re prepared to rearrange your business.
Location 2496: An accounting system is an almost perfect accountability sink
Location 2502: Everyone who has put together a business plan knows that if you can’t fudge the key assumptions to justify the decision your boss wants to make, you don’t know enough about the business.
Location 2516: Of course management theories change – when the foundations of the problem are constantly shifting, the answers are bound to change.
Location 2524: the stars of this literature all try to get managers to understand that they must create systems which regulate themselves rather than requiring constant supervision. Most of what’s worth reading in management science is about stopping decision-making systems from becoming overwhelmed.
Location 2533: Meanwhile, the management consultancy industry prospered by selling the myth that it might be possible to economise on management capacity by renting it when you needed some rather than paying it a salary.
Location 2541: you need a professional scapegoat – you could hire an out-of-work actor to read the solution from cue cards, but management consultants are better at pitching for the business.
Location 2543: management consultants will often work by telling a company something that its employees already know.
Location 2566: If the problem is understood correctly, the solution is usually quite simple
Location 2569: An effective consultant is likely to spend most of their time telling people obvious things that they don’t want to hear.
Location 2572: Telling your client what they want to hear is a better way to get repeat business; the problem won’t go away and the person commissioning the work will still like you.
Location 2575: global management consultancy profession can’t provide corporations with the identity-creating function that economics provides to policymakers. It spends too much of its own capability and information-processing capacity in telling people what they want to hear.
Location 2598: These problems feed into each other – the blind spots of economics and the blind spots of management work together to produce a model of the world that can drift away from reality, and start producing bad decisions. There’s no self-correcting tendency, because the problem is in the information-processing system itself.
Location 2603: the non-human decision-making systems of the world have begun to go a little bit mad.
Location 2756: the essence of leadership was ‘the willingness to confront unequivocally the major anxiety of their people in their time’.
Location 2782: that large professional and managerial class which grew rapidly during the twentieth century and which is distinguished by the fact that it exercises control over capital assets without necessarily owning them.
Location 2794: The greatest fear of the manager is, of course, management.
Location 2795: the crux of Friedman’s doctrine is that when companies act in the interests of society instead of their shareholders, they take on the role of a government.
Location 2798: If the appointed agents of the owners of capital do anything other than promote the interests of their principals, they are taking the decision to spend other people’s money, entering into the ‘political mechanism’ of control and coercion rather than the ‘market mechanism’ of free and voluntary exchange.
Location 2808: the legal and political compromises which created the limited liability corporation were made on the basis of a different set of expectations about the behaviour of corporate citizens.
Location 2811: To think of the corporation as a set of relationships between individuals is absurd – there are far too many of them.
Location 2819: This is what makes the Friedman doctrine, as far as I can see, a lie; it’s an exhortation to executives to reverse the truth.
Location 2824: The Friedman doctrine invited them to disassociate themselves from their roles; to attribute all the bad consequences and all the frustrating lack of independence to a separate work-self, which was under an obligation to a simple principle.
Location 2826: Personal Note: Severance the movie?
Location 2827: Friedman sets up a slippery slope and starts arguing that if you tolerate a little bit of do-gooding by corporations, you’ll soon ‘strengthen the already too prevalent view that the pursuit of profits is wicked’ and thence onward to Communism.
Location 2836: the courts have never found that there’s any particular fiduciary duty to run a company in one way rather than another,
Location 2881: walking across a trading floor is a tangible experience of what it might be like to live inside an artificial intelligence.
Location 2886: If the market is the brain of the capitalist system, its nerves and muscles are made out of the debt relationship.
Location 2887: Starting in the 1970s, the economies of the industrial world saw the beginnings of a huge increase in the use of debt, and this might have been the most significant decision which nobody ever made. Debt may have ended up as a problem, but it always starts out as a solution.
Location 2898: and of course the lawyers have done all the work they can bill for on the subtleties of bankruptcy law.
Location 2899: can understand a lot of what’s important about debt in a couple of paragraphs if you start by considering it as a technology of information and control.
Location 2915: If a system is loaded up with a lot of debt, the need to make the payments becomes a signal that swamps all other sources of information.
Location 2918: Add even more debt, and the borrower is in a situation where nothing they can do will generate enough cashflow to service the debt. At this point, debt becomes an instrument of control.
Location 2924: if the creditor isn’t paid, they have the right to take actions which have a drastic effect on the viability of the system. That’s what makes debt a technology of control.
Location 2965: The leveraged buyout completely rewrites the incentives of the management system.
Location 2978: The ‘private equity’ industry, as the leveraged buyout financiers liked to be called, was an extremely efficient predator – compared to all the other non-human decision-making systems, it was much less slow-moving and very aggressive.
Location 3028: Across the economy, corporate managers were left pleading about their history and community, the trust of their customers, the value of their technology. And time after time, they found out that unless it could be quantified in an accounting system and placed into exactly the right kind of spreadsheet, this kind of information no longer mattered. So they borrowed more money, paid bigger dividends and hoped.
Location 3048: The development of the Friedman doctrine into the intellectual backing for the leveraged buyout boom and the private equity industry are best seen as a conflict between two comprehensive systems of interest, both of which might have regarded the other as a threat.
Location 3100: There’s been a class war between the capital-owning class and the managerial class.
Location 3116: a quarterly financial accounting number – or even a few years’ worth of them strung together – is an extraordinarily crude information filter.
Location 3120: The management has greater control over costs than revenue – they depend less on decisions taken outside the organisation’s perimeter.
Location 3146: Stock options were the reward for generating short-term profits, but employment insecurity was the other half of the deal.
Location 3154: accounting systems can act as an information-reducing filter.
Location 3155: the greater the emphasis placed on accounting-based targets set by the CEO, the greater the filtering effect.
Location 3166: If you consistently demand the impossible, you will inevitably get the unethical.
Location 3170: the real disaster for the information processing capacity of the capitalist economy was outsourcing.
Location 3179: By looking for the activities in which they could be global leaders and outsourcing everything else, companies exchanged internal boundaries for external ones.
Location 3183: The typical outsourcing contract narrows the bandwidth of the communication channel to either ‘everything is going more or less as anticipated’ or ‘it’s stopped working and we need to find out why’.
Location 3221: The function of the investing class is to serve the members of the working class by insuring them against loss and by providing them with desired goods.
Location 3246: economic policy began to use the unemployment rate as a variety attenuator;
Location 3249: The global economic system had gradually removed vital parts of its own brain; informal and rich communication channels within organisations were replaced with formal and contractual links which often crossed time zones.
Location 3250: Intelligence functions that were meant to consider the future were swamped with debt, so that they couldn’t pay attention to anything that didn’t generate cash for the next payment.
Location 3263: the public sector lacks any direct equivalents of the debt relationship and the leveraged buyout.
Location 3271: The imposition of Friedmanism and the debt burden in the corporate sector gave a clear signal about the new priorities, but the equivalent processes in the public sector just added a lot of noise.
Location 3284: The decerebration of the public sector happened in a similar way to that of the private sector; they used management consultants for short-term injections of brainpower, and outsourcing contracts which had the effect of removing management capacity in the long term.
Location 3293: the people who buy the thing have the ability to refuse to do so.
Location 3379: elections are simply horse races with executive power as the prize, and opinion polls are rarely used as more than a sort of racing form to predict the winners of the next race.
Location 3400: You can’t say how much information a human being is taking in and reacting to at any given time, but you can easily observe the difference between a human being that is coping and one that is overloaded.
Location 3409: the Whitehall studies,
Location 3409: Personal Note: Using suicide and personal health as an indicator of a countries economic health?
Location 3417: Marmot ended up concluding that the psychic feeling of being in control of your life is extremely important as a source of well-being, and that conversely, being out of control is physiologically harmful as well as emotionally intolerable.
Location 3423: That might be the deepest reason why managers create accountability sinks – to be accountable for something you can’t change is to experience exactly the ‘out of control’ feeling that the Whitehall studies seem to suggest will kill you if you let it.
Location 3426: People are overloaded with information that they can’t process; the world requires more decisions from them than they’re capable of making, and the systems that are meant to shield them from that volatility have stopped doing the job.
Location 3437: The populist movements of the 2010s all promised a simpler world; they were, in the words of J. K. Galbraith, taking on the great anxiety of their people and addressing it.
Location 3443: It’s definitely not what the populists are proposing – there’s no March for Bureaucracy, nobody’s slogan is ‘Red Tape Holds Us Together’.
Location 3475: organisations initially deal with imbalances in their ability to handle environmental variety by adding resources to the existing channels – and when that is no longer possible, by reorganising.
Location 3487: What do we need to do, in order to stop the organisation of the industrial economy from causing excessive stress on the human population?
Location 3489: how can we begin to get rid of the theoretical and ideological blind spots that got us into this mess in the first place?
Location 3535: they don’t work by defining an objective function and seeking to maximise it. This is the paradigm shift that might be required – that organisations and systems can be like people, having purposes without a single goal. An artist doesn’t have a successful career by maximising their art; they do it by repeatedly producing work that they are proud of.
Location 3539: Businesses ought to be like artists, not paperclip maximisers.
Location 3558: The Galbraithian ideal economy is one in which the technostructure prevails; the business cycle is managed by managers, who maintain stability by communicating with one another, and keep control of the productive resources of society simply by showing up to do a job that nobody else wants to bother with.
Location 3564: Nobel Laureate called Herbert Simon also proposed a ‘behavioural theory of the firm’, in which profits were not the goal to be maximised, but instead a constraint on behaviour – managers were assumed to do just enough to keep investors happy, then spend the rest of their time and effort trying to do new things they could be proud of.
Location 3573: If the problem with the modern corporation is the result of the capitalist counter-revolution against the managerial class, we just need to change the terms of the battle.
Location 3576: the first step towards the possibility of any improvement has to involve doing something about private equity.
Location 3579: Dismantling the leveraged buyout industry would get rid of an overhanging threat across the entire managerial class; it would open up a huge space for different models of corporate governance.
Location 3590: The principle of limited liability acts as an accountability sink for private equity investors.
Location 3596: But it’s a bargain, not a principle of natural justice, and it needs to be reviewed if it’s causing trouble.
Location 3597: Personal Note: LLC reference
Location 3599: any entity taking control of an operating company should have to guarantee its debts.
Location 3600: Once you take away the big accountability sink – and make it possible for a bad deal to affect investors as well as the little people – most of the attraction of using a company’s debt-servicing capacity to ‘buy it with its own money’ is gone.
Location 3608: When we think of corporations as artificial intelligences, we worry they have developed a monomania, an addiction to a particular kind of stimulus, and that they are destroying their own lives, the lives of others and even the material conditions of their existence, in order to satisfy their craving for profits.
Location 3610: Corporations are decision-making systems, not ‘intelligences’.
Location 3620: Either a corporation has a survival condition based on needing to make a monthly interest bill, or there’s an implicit threat from the financial environment that if it fails to behave in a particular way, it will be taken over by an outside entity. If you take away that pressure, it’s quite likely that the natural equilibrium of corporate decision-making systems will be less hostile to human life.
Location 3623: Viable systems fundamentally seek stability, not maximisation.
Location 3631: It is hard to hear the conscience of corporations above the noise of debt and ‘fiduciary duty’, but it’s there.
Location 3638: If the highest-level decision-making mechanisms of the world are to be viable systems, they need a philosophy which can balance present against future and create self-identity.
Location 3642: Any system which is set up to maximise a single objective has the potential to go bonkers.
Location 3644: Setting up a maximising system involves defining an objective function, and throwing away all the other information.
Location 3646: Sooner or later, the environment is going to change, and something which isn’t in the information set any more is going to lead the system into destruction.
Location 3648: Every decision-making system set up as a maximiser needs to have a higher-level system watching over it.
Location 3655: implies that the top level of any decision-making system that’s meant to operate autonomously can’t be a maximiser.
Location 3669: It would be a great thing if the economy could be made to stop channelling stress and volatility into people’s lives.
Location 3671: the system requires the creation of accountability sinks in order to function, that the general population are not happy with this, and that the only ways they have of expressing their discontent seem to be highly destructive of the system itself.
Location 3692: We cannot afford the luxury of explainability; we can’t keep on demanding that an identifiable human being is available to blame when things go wrong.
Location 3698: people in my experience are a lot less angry about everything when they feel like they’re being listened to.