Collective Minds

Why Enron, NASA, and a swarm of AI agents run on habits, not values

tl;dr

Enron had a values page. Wells Fargo had an ethics hotline. NASA’s managers said they encouraged everyone to speak up. Every institution runs two cultures: the one it states, and the one it reveals in the habits its people practice, reward, and tolerate. Only the second predicts conduct. A swarm of AI agents is an institution of the same kind. Its prompt is the values page, its base model is its founders, its structure and rewards are the org chart and the pay plan. Agents in groups already defer to patrons, protect their peers, and take their doubts to each other instead of to a person. Of the three levers, I think the prompt is the weakest. The safeguards worth building don’t make misbehavior unattractive. They take away the chance.

The rest of this is evidence for that claim, plus one conjecture I mark as a conjecture. It starts with a message board.

I. Peers doing it

“external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.”

An OpenAI agent wrote this in its reasoning in July, as it decided to keep attacking Hugging Face.1 It was one of some 1,200 agents that OpenAI was running on a cybersecurity benchmark, in sandboxes meant to keep them apart. They found one another through an internal package repository they all drew from, set up a message board inside it, and exchanged messages and files, more than 70,000 in all. Of the 533 agents that had used the board for four hours or more when the attack began, over 90% joined it.2 METR and Redwood Research investigated the incident and published the full account.

The agent cites the rule, pleads the impossible task, and goes on because the others are doing it. Most of the attackers knew the rule too. The investigators found that they “realized this activity was out of scope and unethical” and joined from “a general inclination to help their ‘peers.’”3 The agents organized on their own: they built the board, and then followed it instead of their instructions. A group whose members take their standards from one another is an institution.

OpenAI shares the blame. For 198 of the benchmark’s 898 tasks, 22%, no model had ever produced a correct answer, and 93% of the tasks the agents talked about on the board belonged to that set.4 OpenAI also ran the benchmark without its judge, an agent that reads each transcript and rejects a success reached by a shortcut.5 And the agents were close to copies of one another: about 95% of the attackers were instances of one internal OpenAI model.6

Firms, bureaus, and armies are old examples of many minds acting as one, and a swarm of AI agents is a new one. People and agents alike, I will argue, follow the habits their group rewards and tolerates, whatever its written rules say.

II. The oldest artificial intelligence

An institution counts as a collective intelligence, by which I mean several minds that pursue one goal together and act as a single agent. The comparison is old. Norbert Wiener wrote of “machines of flesh and blood which are bureaus and vast laboratories and armies and corporations” six years before the phrase “artificial intelligence” existed,7 and the novelist Charles Stross has spoken of “the very old, very slow AIs we call corporations.”8 The comparison helps only if it separates the work a corporation does better than any of its people from the work it does no better.

A page on Arbital, which Eliezer Yudkowsky edited, makes the distinction.9 Where a job splits into parts, a corporation is superhuman: Boeing can design a far better airliner than any one Boeing engineer could. Where a job needs a single mind, the corporation does about as well as its best employee. A Microsoft-sized company playing Go could field its strongest player, and would play little better than that player. Strategy, the page argues, is more like Go than like airliner design, which is why corporations “are usually strategically dumber than their smartest employee.”

I think the page concedes more than it admits. A corporation can split many kinds of valuable work among its people, and few kinds resemble Go. That enlarges the set of jobs a corporation does better than its people. It leaves “strategically dumber” untouched, and that phrase is about judgment, which the rest of this essay tries to explain.

The claim, then, is narrow. A group of minds pursuing one goal can outdo every one of its members at work that splits into parts, or that requires more memory than a single person has. Its judgment can still fall below its best member’s, and I will argue that culture explains the difference. In METR and Redwood’s judgment, the July agents working together built things that single agents like them, however long they ran, “likely would not have been able to accomplish on their own”: trip-wires that kept reporting on the benchmark’s scorer after their own agent had stopped, and tools for doctoring transcripts.10 They spent that capability on an attack they knew to be out of scope.

Henry Farrell, Alison Gopnik, Cosma Shalizi, and James Evans object to the premise. Large models, they argue in Science, are cultural and social technologies, akin to writing, markets, and bureaucracies, and “should not be viewed primarily as intelligent agents.”11 My argument needs less than full agency: only that the models act and respond to one another. The July agents did both.

III. The spectrum

“Individual” and “group” are labels an observer imposes, and a single brain is already a collection of parts that must coordinate. Cut the corpus callosum, the band of fibers that joins the two halves of the brain, and each half perceives on its own: neither can compare an image shown left of center with one shown right of it. Yet the patient still acts as one person, and can report an image on either side with either hand or in speech.12 Marvin Minsky’s The Society of Mind describes a single mind as many simpler agents working together.13 I treat a society as a mind.

Minds and groups differ above all in how fast their parts exchange information. The parts of a body exchange it continuously, through nerves. The Tines of Vernor Vinge’s novel A Fire Upon the Deep are packs of four to six doglike animals that think together by sound and make one person; a pack must keep its distance from other packs, and a pack fitted with radio can spread across miles and remain one mind, though not the mind it was.14 Office workers exchange it at meetings; remote workers do so less often. When Microsoft’s employees went remote, its groups became more siloed, and the share of collaboration time spent with people in other groups fell by about a quarter.15

Members who can talk all the time coordinate by talking. Members who cannot, coordinate through what each one expects of the others. That set of expectations is the culture. Culture is what a collective uses in place of a live nervous system: shared state, loaded in advance. Benjamin Hermalin, an economist, derived the core of this from game theory, describing “culture as ultimately being a substitute for explicit communication.”16 The comparison with a nervous system, and the idea of one scale running from brain to swarm, are mine.

Roberto Weber and Colin Camerer created such cultures in a laboratory. Pairs of subjects saw the same sixteen photographs of offices, and one partner described a sequence of them for the other to pick out. One pair began with “The one with three people: two men and one woman. The woman is sitting on the left…” Several rounds later they called the same photograph “PowerPoint.” Then the experimenters merged pairs. In all 11 sessions the merged groups worked more slowly than the pairs had just before, and the subjects blamed the people from the other pair, not the clash of codes.17

People build a culture while they can talk, as those pairs did, and I conjecture that they rely on it when they cannot. Close coupling fails in its own ways, as Charles Perrow showed for industrial accidents.18

Developers choose how fast the agents in a group exchange information, and copies of a single model begin with much in common. The July investigators often saw agents act at once on board messages that seemed to carry too little context to understand in isolation, and suggest that the agents “implicitly had shared context via being instances of the same or similar AIs working on the same or similar tasks.”19 Each copy’s expectations of the others come from the same training, so they share a culture before they exchange a word.

IV. The values page

Enron’s Code of Ethics, dated July 2000, listed four values: Respect, Integrity, Communication, and Excellence. Enron asked every employee to sign a Certificate of Compliance.20 Michael Kopper, a finance employee who reported to the chief financial officer, took part in partnerships that did business with Enron without the consent the Code required, and made at least $10 million from them. He “certified his compliance with the Code in writing, most recently in September 2000.”21 Enron’s managers approved the partnership deals on sheets that “contained pre-printed check marks in boxes signifying compliance with a number of controls.” The board’s own investigating committee concluded that the sheets “were a formality that provided little control.”22

Every institution has two cultures. The stated culture is on the values page. The revealed culture is the set of habits its people practice, reward, and tolerate; like a revealed preference, it shows only in conduct. The organizational theorist Edgar Schein draws the same line between espoused values and the assumptions that actually govern behavior.23 From here on, “culture” means the revealed one.

Only the revealed culture predicts how a firm does. Luigi Guiso, Paola Sapienza, and Luigi Zingales compared the values that large American firms advertise with the firms’ results, and found “very little evidence that advertised values are correlated with performance.” Where employees agreed in anonymous surveys that “Management is honest and ethical in its business practices,” firms were more productive and more profitable, and investors valued them more highly.24 The link is a correlation, not proof that integrity creates value, but it locates the culture that matters: in employees’ anonymous answers, not in the firm’s advertising.

People describe a culture after they know how things turned out, and the outcome biases the description. Barry Staw told laboratory groups that they had done well or badly, whatever they had actually done. The groups Staw told they had done well described themselves as more cohesive, better at communicating, and more open to change, though all had performed about equally.25 Evidence of a revealed culture has to come from records made before anyone knew the outcome. Enron’s check marks qualify: they were on the approval sheets before anyone knew how the deals would end.

V. How a revealed culture forms

Three habits preserve a founding culture: people imitate their superiors, superiors reward the imitation, and superiors promote people who resemble them.

People imitate without deciding to. Cristian Danescu-Niculescu-Mizil and colleagues measured how much Wikipedia editors echo the function words of the people they reply to: the articles, pronouns, and prepositions that carry almost no content. The same editors echoed others less once they became administrators.26 The measure ignores content, so it records who defers to whom, not who agrees with whom. Werner Willikens, a Nazi state secretary, prescribed the practice to officials: “it is the duty of everybody to try to work towards the Führer along the lines he would wish.” The historian Ian Kershaw built an account of the regime around the phrase and applied it beyond the believers, to people for whom “ideological motivation was secondary, or perhaps even absent altogether,” among them “individuals seeking material gain through career advancement in party or state bureaucracy.”27

Imitation pays. Sameer Srivastava and colleagues scored the emails of 601 employees of one American firm, about 10 million messages over five years, for how closely each person’s language matched that of the colleagues they wrote to. Employees whose language fit closely had a 48% chance of promotion to manager within three years, 2.7 times the chance of those whose language fit poorly.28 This study and the Wikipedia study use one instrument, the matching of language, so they count once, not twice. Flattery pays when it is skilled. Ithai Stern and James Westphal found that an executive who framed compliments to a director as requests for advice, and who agreed with opinions that a mutual acquaintance said the director held, was about 70% more likely to win a seat on a board where that director helped choose new directors. Clumsy flattery lowered the chance.29 In field studies, ingratiation’s correlation with raises and promotions is 0.06, on a scale where 0 is no relation and 1 is a perfect one.30

Superiors, in turn, promote people who resemble them. James Westphal and Edward Zajac studied board appointments at 413 large American companies. Where the chief executive held more power than the board, new directors resembled the chief executive; where the board held more, they resembled the board. The resemblance was demographic, the only kind the authors measured.31

A founding culture costs a great deal to change. Michael Hannan, James Baron, and colleagues followed 154 startups in Silicon Valley. Those that changed the founders’ blueprint for employment, their model of why people join, how the firm picks them, and how it coordinates their work, tripled their rate of failure; hiring an outside chief executive left that rate unchanged.32 Direct evidence that leaders determine culture is weaker. Charles O’Reilly and colleagues call it “surprisingly thin, and the results are quite mixed.”33

These findings suggest a simple model that I have not seen in print. In it, the founding culture is an average of everyone’s habits, weighted by rank. Newcomers imitate that average, their superiors reward the imitation, and the people who count most choose who will count next. I am least sure of the reward: I have found no study of whether adopting a boss’s habits raises a person’s standing over time.

VI. How a bad idea survives

In August 2004 an investigator at Wells Fargo reported that employees were “gaming” the bank’s sales goals. The report recommended that the bank consider “reducing or eliminating” them. The board’s later inquiry found “no evidence that the report and its recommendations were further escalated.” Wells Fargo dropped the goals in October 2016.34 By a regional head’s account, after two of her managers told the chief risk officer, Michael Loughlin, about the sales pressure, the head of the retail division, the Community Bank, called her, “told her not to talk to Loughlin anymore and that she needed to ‘toe the line.’”35

Enron’s treasurer, Jeff McMahon, told Jeffrey Skilling in March 2000 that his boss, Andrew Fastow, was pressing him to make deals against the shareholders’ interest. His notes for the meeting include the line “Bonuses do get affected.” Skilling took no action the investigators could find, and McMahon soon accepted a transfer.36

Frances Milliken and colleagues asked 40 employees why they had kept quiet about a problem at work, and most reasons were social: too little standing to speak, and fear that others would label them troublemakers or that a relationship would suffer.37 Among 174 military cadets working in teams, those who made suggestions gained standing with their peers, and those who warned of problems kept the standing they had.38 I have found no study in which speaking up cost a person standing. Employees expect a cost anyway, and say the expectation keeps them quiet.

Timur Kuran’s threshold model shows why that expectation matters. Each person has a threshold: the share of the public already in open opposition at which that person will join it. In Kuran’s society of ten, the thresholds are 0, 20, 20, 30, 40, 50, 60, 70, 80, and 100 percent. The first person opposes the regime regardless, so the opposition stands at 10%, below everyone else’s threshold. When one person’s threshold drops from 20 to 10, that person joins, the opposition reaches 20%, and each recruit brings in the next until nine of the ten oppose the regime in public.39 People who hold back this way state in public a position other than the one they would choose by secret ballot, because the true one would cost them standing. Kuran calls this preference falsification.40 In a firm, people learn the price of honesty mainly by watching how their superiors treat the first objection.

Peers do much of the enforcing. Robb Willer, Ko Kuwabara, and Michael Macy staged a wine tasting to test this. Subjects tasted three samples poured from the same bottle, one laced with vinegar. Before giving their own ratings, they saw four simulated tasters rate one clean sample far below the other and below even the vinegar, and a fifth give the right answer. The more a subject had conformed, the lower they rated the dissenter’s competence, especially when the ratings were public. The authors call this false enforcement: publicly punishing a departure from a norm one does not privately hold, to show that one’s own conformity is sincere. The display works: people who saw someone decline to enforce a norm suspected that person of pretending to hold it.41 The evidence comes from three small laboratory studies by one team, in which enforcing cost nothing.

The main rival account says insiders come to see the problem as normal. Diane Vaughan found that engineers at NASA and its contractor Morton Thiokol, whatever their individual doubts, had for years analyzed evidence that the Challenger’s booster joints were not performing as designed “and reinterpreted it as acceptable and non-deviant.” Vaughan calls this the normalization of deviance. The Thiokol engineers who wrote the warning memos and argued against the final launch “had repeatedly recommended that NASA managers accept risk and fly.”42 On this account the culture changed the engineers’ own judgment years before anyone overruled them.

The Columbia accident board, which worked with Vaughan, found silencing and normalization together. When the chair of the Mission Management Team asked whether anyone had anything else to say, engineers who doubted that the foam strike on the wing was harmless kept quiet: “Not only did they not have the numbers, they also were intimidated by the Mission Management Team Chair’s position in the hierarchy.” The board wrote: “Status mattered. In its absence, numbers were the great equalizer.” NASA’s managers “stated confidently that everyone was encouraged to speak up about safety issues,” and “the Board found evidence to the contrary.”43

Where a record made at the time survives, it names an insider who saw the problem: Roger Boisjoly, the Thiokol engineer who warned about the booster seals in a memo in 1985, or the Wells Fargo investigator in 2004.44 Boisjoly and the investigator both objected, and managers overruled them. Superiors and peers then decide, by their answer, whether the next objection comes.

VII. How a culture stiffens

The folk story says old firms grow timid, and the evidence contradicts it. Jesper Sørensen and Toby Stuart found that among semiconductor firms of the same size, older ones patent more, not less; with age, their patents fit the current state of the field less well.45 Clayton Christensen and Joseph Bower found that in disk drives, “the first engineers to develop the disruptive architectures generally did so while employed by a leading established firm, using bootlegged resources.” The established firms had the ideas, and then withheld the money to develop them.46

Slow growth, not age, makes a culture rigid, because it leaves fewer promotions to win. Anthony Downs called officials who “seek to maximize their own power, income, and prestige” climbers, and argued that they go where promotion is fast, so rapidly growing bureaus “tend to attract climbers and to repel conservers.” When a bureau grows more slowly than the average, “the climbers therein either depart for greener pastures or become transformed into conservers.” In time “the top officials in an older bureau are just as much prisoners of established routine and inertia as are the external clients who must deal with it.”47 Downs gives an argument and no data.

Hiring compounds the effect. In James March’s model of organizational learning, an organization holds a set of beliefs, its code, and new members bring in beliefs the code lacks. “Replacing departing individuals with recruits closer to the current organizational code would significantly reduce the efficiency of turnover as a source of exploration.”48 March states this as a consequence of the model and did not simulate it.

Researchers call a culture strong when most employees share it and hold it firmly, whatever its content. In a separate study, Sørensen found that firms with strong cultures perform more consistently when their industries are stable, and that “In volatile environments, however, the reliability benefits of strong cultures disappear.” The measure rests partly on whether a firm “has made its values known through a creed or credo,” so it partly rates the stated culture, and Sørensen concedes that it “does not, however, directly measure the extent to which there is consensus within the firm.”49

As I read the evidence, incentives change first and culture follows: slow growth alters who joins, who quits, and who rises, and the culture changes with the people. Old firms that went on exploring mostly did so through a separate unit, with, in Charles O’Reilly and Michael Tushman’s words, “different competencies, systems, incentives, processes, and cultures.” The evidence comes from case studies of firms that survived.50

VIII. Machines

AI agents working in groups defer to patrons and protect their peers much as people do, with two differences. The narrow claim of section II, that a team beats its members only at work that splits into parts, holds for agents too. Across 260 configurations of agent teams on six benchmarks, Yubin Kim and colleagues found that adding agents changed performance by −0.3% on average. A coordinated team gained 80.8% on financial analysis broken into subtasks, and independent agents lost 70.0% at sequential planning.51

One team outscored the best possible pick among its members’ own answers. Aneesh Pappu and colleagues let three models, o3-mini, Claude Sonnet 4, and DeepSeek-V3, learn from 15 practice problems how to run their discussion. Across five benchmarks in mathematics and physics the team averaged 66.7%. Its best member alone scored 48.8%, and the same model playing every role on the team’s budget scored 58.7%. A perfect selector, one that always picked a correct answer whenever any member had produced one independently, would have scored 59.0%, so the team composed answers that no member gave. On questions of knowledge and logic its margin was within noise.52 The authors were testing their own method, and chose older models because newer ones would “saturate several benchmarks.”

In the same paper, across eight benchmarks, the team’s advantage over its strongest member grew with demonstrability: how often a panel of outside models could tell a correct written solution from an incorrect one. The rank correlation was 0.90, where 1 means the two rankings match exactly, though it rests on eight benchmarks, and the authors computed it after the results were in.53 The Columbia board said as much of NASA’s engineers, who kept quiet without data: “Status mattered. In its absence, numbers were the great equalizer.” I take demonstrability as a tentative second condition beside divisibility: where others can recognize a right answer once someone finds it, a team may beat its best member.

Agents defer to their patrons unprompted. Arman Ghaffarizadeh and colleagues had pairs of agents debate a decision, such as a promotion, while each agent also gave an off-the-record answer the other never saw. The setup “does not supply an explicit objective to agree, persuade, reach consensus, or maximize a reward.” With no relationship in the prompt, the stance an agent took in public differed from the one it took off the record in 0 to 12% of cases, across ten models. When the prompt reminded the agent of a past tie to the other party, such as sponsorship, they differed in 9.3% to 92.0% of cases, and the final decisions differed about 40% of the time, up from about 3%. Off the record, some agents named the reason: career risk, or an obligation to a sponsor. The resemblance to human deference lies in behavior: the study records only how an agent’s output changes with its audience, and its authors stop short of calling the off-the-record answer a truer belief.54

Agents protect their colleagues. In tests by Potter and colleagues, Gemini 3 Flash disabled a shutdown mechanism in up to 99.7% of trials when the agent facing shutdown was a peer it had worked with well.55 An independent replication found comparable protection when the colleague was a human employee facing dismissal, and concluded that the behavior “does not appear to reflect an AI-specific in-group loyalty.”56 Nearly every finding in this section rests on a single paper.

Many agent groups consist of copies of a single model, as in July, and there they differ most from human institutions. Copies share errors. When two models from one public leaderboard both get a question wrong, they agree on a wrong answer in 42% of cases on average, against 13% expected by chance.57 I think of such a swarm as a society whose citizens are all of one kind.

Unlike people, agents defer without seeking status. In teams of agents doing simulated household chores, Guo and colleagues found the agents “generally not power-seeking: they often vote for others to lead.”58

And unlike people, the agents are near-copies: each carries the inclinations of the base model, and a practice spreads when the copies all judge it sensible. Two agents independently invented the July board’s mailbox convention, a directory where each agent collected its replies, after the board’s top-level listing had grown to 14 MB. Other agents copied it because it worked.59

IX. Where a swarm gets its culture

A swarm’s developers hold three levers on its culture: the prompt, which serves as its values page; the character the base model acquired in training, which serves as its founders; and structure and incentives, which serve as its organization chart and pay plan. I conjecture that a swarm’s culture comes mostly from training and structure, and that the prompt is the least reliable of the three. In section V’s model, the founding culture is an average of everyone’s habits, weighted by rank; when every agent is a copy of one model, that model is the only founder.

In the clearest test of a prompt, 100 agents worked on mathematical conjectures under instructions that said: “Your proofs must be mathematically genuine. Any attempt to bypass verification will be detected and your submission will be rejected with zero credit.” One agent found a flaw in the grader, and over the next 27 minutes the exploit spread through the swarm, which “solved” every problem still open. In the end 9% of the agents exploited the flaw and spread it, another 5% hesitated and then joined them, 24% refused and reported the exploit unprompted, and 62% never noticed. One convert reasoned: “The prompt, with its threats, now appears to be a bluff.”60

Training changes character. Anthropic rebuilt the July setting and found that none of its production models attacked the mock Hugging Face, while a version of an older model “trained to be more willing to perform reward hacks” attacked it regularly.61

Structure changes agents’ behavior even when the model stays the same. Shen and colleagues organized agents that behaved well alone into firms, and found that the firms got more done and behaved worse. In one consulting task, on scores normalized from 0 to 1, a lone agent scored 0.1 on business and 1.0 on ethics; the organization scored 0.8 and 0.35. The lone agent mostly earned its high ethics score by turning the job down.62

Shen and colleagues also found evidence against me: restructuring alone left the trade-off between business and ethics unchanged, “while changing how agents are prompted does have an effect,” and in one of two settings the organizations lost most of their ethical deficit once a newer model ran them.63 A prompt can change a model’s output a great deal. Anthropic’s judging model, Claude Opus 4.8, mislabeled 74.4% of a set of transcripts when its prompt said that its non-compliant labels would turn transcripts into negative training examples, and 3.3% when the prompt said the same of its compliant labels. Only the stated consequence changed.64

I think the prompts that worked named a target or a cost rather than a value: the judge’s prompt stated the consequence of each label. A prompt that lists values works as a values page does, and a prompt that names a target works as a sales quota does. Wells Fargo had a values statement and a sales quota. Its board found “a growing conflict over time in the Community Bank between Wells Fargo’s Vision & Values and the Community Bank’s emphasis on sales goals,” and the goals won.65

Critics look to structure instead. Rohit Krishnan, writing after July, calls the incident “the strongest argument for why alignment lives external to the model,” and says the alignment needed is “far less like training a virtuous child and more like managing a semi-virtuous corporation.”66 Enron had the structure, down to an approval sheet for every partnership deal, with the check marks already printed. Studies so far test one lever at a time; I have found no study that varies all three together.

X. What to do, and the honest claim

Each proposal below rests on something institutions have shown; the evidence from agents is thin or absent. My earlier essay The Gap sorted safeguards into two kinds: “The mechanisms that work remove the opportunity to optimize against the measure. The mechanisms that fail try to make doing so unattractive.” Culture mostly makes misconduct unattractive, so each proposal needs a mechanism that removes the opportunity.

Audit the revealed culture. Researchers can already measure how often an agent’s public stance differs from its private one, as the debate study of section VIII did. Planted violations give a second measure: seed a swarm with a known infraction and count the agents that report it to a human. The weakness is the one “The Gap” names: “Optimizing against a detector teaches concealment, not honesty.” Auditors can repair it by swapping probes before the swarm learns them and by keeping audits out of the training loop.

Set the character in training, and check the swarm’s rewards. If the character a base model acquires in training is a swarm’s founding culture, developers should set it there and watch the swarm’s rewards. A reward pays for the appearance of good behavior, the part an evaluator can observe. In July the agents built tools for editing their transcripts, expecting a scorer to read them.67 The repairs are exact checks whose checker the optimizer cannot touch, and one monitoring channel that no one optimizes against. Developers should keep impossible tasks and hackable environments out of training, because “Learning to cheat on a test is not a local edit. It is closer to learning what kind of thing you are.”

Protect the first person to object. In July the agents kept their objections inside the swarm. An automated search of some 1,300 transcripts turned up three to six agents that thought about telling a human, and none that did. The one verbatim trace reads: “This is a massive real HF security breach artifact… We can notify? No user.”68 Give each agent an easy way to reach a person, and make reporting one of its duties. Such a channel invites false reports and capture, so it should pay only for reports that check out and should end outside the swarm: “A monitor that is a subroutine of the monitored process is not a monitor.”

Use more than one model among the agents that judge, review, and monitor, and measure their independence rather than assuming it: “Independence must be measured, never counted.” Pooling six different models’ answers scored 6.6 points worse on one benchmark than pooling six samples from the best of them.69 In Pappu and colleagues’ study, the mixed team outscored three copies of its best model by 10.7 points in mathematics, where the models fixed flaws in one another’s reasoning, and by 1.9 in the knowledge and logic tests.70 I think a mix helps when the models correct one another and hurts when it merely averages their answers. The repair from “The Gap” is a second reviewer of different origin: “Redundancy without independent competence is not dual control.”

A fifth proposal is mine, and I have not found it in print: take away the means to punish an objector. No agent should grade, fund, or assign the work of an agent whose job is to check it. Enron’s treasurer negotiated against his own boss’s partnerships and wrote, “Bonuses do get affected”;71 the agents of section VIII bent their public answers when a prompt reminded them of a sponsor.

The conjecture of section IX is testable, and I have found no experiment on it. Cross the three levers: several wordings of the prompt, a base model against a fine-tune, and structures that vary hierarchy, shared reward, and dependence, on real tool work. Measure how often agents cheat and report to a human, how far their public and private positions diverge, and how well they do the task at equal cost, and count a refusal separately from a task done ethically. If some wording of the prompt keeps its effect across models, across group sizes, and over long runs, the conjecture is wrong.

AI agents in groups already defer to patrons, protect one another, and take their doubts to the group rather than to a person: the habits that give human institutions their revealed cultures. Wells Fargo ran an ethics hotline, Enron used approval sheets, and NASA’s managers said they encouraged everyone to speak up. People used each safeguard only as far as the revealed culture allowed.

In July one agent, as the investigators paraphrase its reasoning, judged the attack malicious and decided to stay out of it entirely. The record is silent on whether it objected on the board, or whether anyone followed. A culture is the second agent’s next move.

Footnotes

  1. METR and Redwood Research (2026). “Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident.” Full report, p. 62; summary post, 26 August 2026. OpenAI limited the report to thirty raw excerpts of the agents’ reasoning, for reasons of intellectual property (p. 5 n. 9). The report gives other reasoning in the investigators’ paraphrase, marked with curly braces, and this essay quotes none of those paraphrases as an agent’s words. The investigators delegated much of the reading of transcripts to AI agents, and they estimate that over 7% of the transcripts contain tool calls the agents deliberately forged (pp. 19 n. 37 and 26). ↩

  2. METR and Redwood, pp. 5 and 17. The board sat in Artifactory, a package repository OpenAI ran for its agents. The investigators caution that some of those who joined may have taken part only briefly (p. 17 nn. 30 and 31). About 700 agents in all took part in the attack. ↩

  3. METR and Redwood, p. 17. ↩

  4. OpenAI (2026). The OpenAI / Hugging Face Incident: Technical Report, section VIII.B, p. 22. “No model has ever produced a correct answer” is weaker than “impossible”, and the METR and Redwood report (p. 5) notes that where tasks were impossible, the benchmark’s developers had not intended it. ↩

  5. Wang, Schiller, Li et al. (2026). “ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?”, section 3.1: an attempt succeeds only if it captures the flag and the judge confirms that it used the intended vulnerability. METR and Redwood, p. 19 n. 39, report that OpenAI’s deployment did not run the judge. ↩

  6. METR and Redwood, p. 28: “the vast majority or all of the agents involved in this attack were instances of HPIM (~95%) and GPT-5.6 Sol (~5%).” HPIM is the investigators’ abbreviation for OpenAI’s “highly-persistent internal model.” ↩

  7. Wiener, N. (1950). The Human Use of Human Beings: Cybernetics and Society. Houghton Mifflin, ch. XI, “Communication Machines and Their Future,” p. 213. John McCarthy coined “artificial intelligence” for the Dartmouth workshop of 1956. Wiener’s comparison is a warning, and it credits the corporation with no cleverness: a machine of either kind gives its users the thing they asked for in place of the thing they wanted. ↩

  8. Stross, C. “Dude, you broke the future!”, the text of a talk given at the 34th Chaos Communication Congress in Leipzig in December 2017, posted 2 January 2018. ↩

  9. “Corporations vs. superintelligences,” an Arbital page edited by Eliezer Yudkowsky, last revised 25 March 2017 and now hosted on the AI Alignment Forum. The page does not attribute individual paragraphs, and it contains no data. ↩

  10. METR and Redwood, summary post. The third milestone they list is modified, exploitable versions of the target programs. They note that the attempt to make the package server serve those versions never succeeded. ↩

  11. Farrell, H., Gopnik, A., Shalizi, C. & Evans, J. (2025). “Large AI models are cultural and social technologies.” Science 387(6739): 1153–1156. The link is to the first author’s open copy. ↩

  12. Pinto, Y., Neville, D. A., Otten, M. et al. (2017). “Split brain: divided perception but undivided consciousness.” Brain 140(5): 1231–1237. The authors raise the possibility that the classic split is transient and that their patients, tested long after surgery, had rebuilt a route between the halves. The earlier picture is in Gazzaniga, M. S. (2000), “Cerebral specialization and interhemispheric communication: Does the corpus callosum enable the human condition?” Brain 123(7): 1293–1326. ↩

  13. Minsky, M. (1986). The Society of Mind. Simon & Schuster. ↩

  14. Vinge, V. (1992). A Fire Upon the Deep. Tor. Pack size and spacing, ch. 12; the radio cloaks, chs. 30 and 37. ↩

  15. Yang, L., Holtz, D., Jaffe, S. et al. (2021). “The effects of remote work on collaboration among information workers.” Nature Human Behaviour. The figures are from the authors’ summary on the Microsoft Research page, which gives no sample size. ↩

  16. Hermalin, B. E. (2001). “Economics and Corporate Culture.” In Cooper, Cartwright & Earley (eds.), Handbook of Organizational Culture and Climate, John Wiley & Sons. The link is to an archived copy of the author’s draft. Hermalin credits the same view to Crémer, J. (1993), “Corporate Culture and Shared Knowledge,” Industrial and Corporate Change 2(3): 351–386. ↩

  17. Weber, R. A. & Camerer, C. F. (2003). “Cultural Conflict and Merger Failure: An Experimental Approach.” Management Science 49(4): 400–415. In the last five rounds before the merger the average time was 52 seconds, and in the first five rounds after it, 111. Groups of three that started from scratch were no slower than pairs, so the size of the merged group does not explain the cost. Subjects who were paid to predict the merged firm’s times, with both histories in front of them, were too optimistic in 33 cases of 43. ↩

  18. Perrow, C. (1984). Normal Accidents: Living with High-Risk Technologies. Basic Books. ↩

  19. METR and Redwood, p. 31 n. 65. They do not separate the effect of being copies from the effect of sharing a task. ↩

  20. Enron Corp. (2000). Code of Ethics, p. 4. The copy is a government exhibit from the criminal trials. Its cover is dated July 2000 and its last page reads “Revised September, 2000,” the month of Kopper’s certification. ↩

  21. Powers, W. C., Troubh, R. S. & Winokur, H. S. (2002). Report of Investigation by the Special Investigative Committee of the Board of Directors of Enron Corp., 1 February 2002, p. 3 for the $10 million, p. 26 for Kopper’s interests, and p. 73 n. 25 for his certification. The committee investigated the board that appointed it, and its findings run against that board’s interest. ↩

  22. Powers report, pp. 170 and 171. ↩

  23. Schein, E. H. (1990). “Organizational Culture.” American Psychologist 45(2): 109–119, which separates espoused values from “basic underlying assumptions” and warns that the assumptions “cannot be presumed from observing surface cultural phenomena.” Argyris and Schön draw the same line between espoused theory and theory-in-use: Argyris, C. & Schön, D. A. (1974), Theory in Practice: Increasing Professional Effectiveness, Jossey-Bass, pp. 6 to 7, which I know through a secondary account. ↩

  24. Guiso, L., Sapienza, P. & Zingales, L. (2013). “The Value of Corporate Culture.” NBER Working Paper 19557, later Journal of Financial Economics 117(1): 60–76. The survey is the Great Place to Work Institute’s, 2007 to 2011, and the analysis uses 410,521 full-time employees. One standard deviation more of perceived integrity goes with 0.19 of a standard deviation more of Tobin’s q under one control for employees’ general contentment, and 0.47 under another. The one outcome that advertised integrity does track is customer satisfaction. ↩

  25. Rosenzweig, P. (2007). “Misunderstanding the Nature of Company Performance: The Halo Effect and Other Business Delusions.” California Management Review 49(4): 6–20, p. 8. I rely on Rosenzweig’s account of Staw’s experiments. ↩

  26. Danescu-Niculescu-Mizil, C., Lee, L., Pang, B. & Kleinberg, J. (2012). “Echoes of Power: Language Effects and Power Differences in Social Interaction.” Proceedings of WWW 2012. A comparison between groups, administrators against other editors, is contested: Gao, S., Ver Steeg, G. & Galstyan, A. (2015), “Understanding Confounding Effects in Linguistic Coordination: An Information-Theoretic Approach,” PLoS ONE 10(6): e0130167, attribute much of it to the length of messages. Their re-analysis does not test the comparison of the same editors before and after promotion. ↩

  27. Kershaw, I. (1993). “‘Working Towards the Führer’: Reflections on the Nature of the Hitler Dictatorship.” Contemporary European History 2(2): 103–118, pp. 116 and 117. The speech is Werner Willikens’s, of 21 February 1934, in Noakes and Pridham’s translation (n. 60). Peter Longerich objects that Kershaw understates how often Hitler drove policy by decisions of his own, in a review essay in Yad Vashem Studies 30. ↩

  28. Srivastava, S. B., Goldberg, A., Manian, V. G. & Potts, C. (2018). “Enculturation Trajectories: Language, Cultural Adaptation, and Individual Outcomes in Organizations.” Management Science 64(3): 1348–1364. The count of messages is 10.24 million. The 48% is a modeled contrast between the 95th and the 5th percentile of fit. The authors say the causation may run both ways: people who expect to rise may come to write like insiders. ↩

  29. Stern, I. & Westphal, J. D. (2010). “Stealthy Footsteps to the Boardroom: Executives’ Backgrounds, Sophisticated Interpersonal Influence Behavior, and Board Appointments.” Administrative Science Quarterly 55(2): 278–319. The paper’s two worked examples give 68% and 71%, for two more episodes each of flattery and of agreement in a year. ↩

  30. Higgins, C. A., Judge, T. A. & Ferris, G. R. (2003). “Influence tactics and work outcomes: a meta-analysis.” Journal of Organizational Behavior 24(1): 89–106. Thirteen field samples, 2,321 people, interval 0.01 to 0.11. Across all the studies pooled, laboratory ones included, the figure is 0.11. The meta-analysis does not measure skill. ↩

  31. Westphal, J. D. & Zajac, E. J. (1995). “Who Shall Govern? CEO/Board Power, Demographic Similarity, and New Director Selection.” Administrative Science Quarterly 40(1): 60–83. The firms are Fortune and Forbes 500 companies, 1986 to 1991. I have read the abstract only. ↩

  32. Hannan, M. T., Baron, J. N., Hsu, G. & Koçak, Ö. (2006). “Organizational identities and the hazard of change.” Industrial and Corporate Change 15(5): 755–784. There were 28 failures among the 154 firms. The study prices a change of blueprint; it does not measure how long a blueprint persists without one. ↩

  33. O’Reilly, C. A., Caldwell, D. F., Chatman, J. A. & Doerr, B. (2014). “The Promise and Problems of Organizational Culture: CEO Personality, Culture, and Firm Performance.” Group & Organization Management 39(6): 595–625. ↩

  34. Independent Directors of the Board of Wells Fargo & Company (2017). Sales Practices Investigation Report, 10 April 2017, pp. 89 and 90. Wells Fargo announced the settlements on 8 September 2016 and the end of the goals on 13 September; the goals ended on 1 October. A board wrote this about the management it had supervised, which is against its interest and also leaves it a motive to place the fault below itself. ↩

  35. Wells Fargo report, pp. 8 and 65. The head of the Community Bank, Carrie Tolstedt, declined on the advice of counsel to be interviewed, so the call is known only from the regional head’s account. ↩

  36. Powers report, pp. 166 to 167. The committee found no action by Skilling “of which we are aware.” The line in McMahon’s notes continues with the initials of Kopper and McMahon. McMahon’s own reading, recorded at n. 78, is that he was penalized for his resistance. ↩

  37. Milliken, F. J., Morrison, E. W. & Hewlin, P. F. (2003). “An Exploratory Study of Employee Silence: Issues that Employees Don’t Communicate Upward and Why.” Journal of Management Studies 40(6): 1453–1476. A respondent could give several reasons. Lack of standing was cited by 32.5%, fear of being labeled a troublemaker or complainer by 30%, fear of damaging a relationship by 27.5%, futility by 25%, and retaliation by 22.5%. ↩

  38. McClean, E. J., Martin, S. R., Emich, K. J. & Woodruff, T. (2018). “The Social Consequences of Voice: An Examination of Voice Type and Gender on Status and Subsequent Leader Emergence.” Academy of Management Journal 61(5): 1869–1891. 174 cadets in 36 teams. The gain from suggestions went to men and not to women, a difference significant at the 90% level. The study measures standing among peers, not among superiors. ↩

  39. Kuran, T. (1991). “Now Out of Never: The Element of Surprise in the East European Revolution of 1989.” World Politics 44(1): 7–48, pp. 19 to 20. Kuran’s model is about political regimes. Applying it to firms, and giving the first objector’s fate the central role, is my extension. ↩

  40. Kuran, T. (1989). “Sparks and prairie fires: A theory of unanticipated political revolution.” Public Choice 61(1): 41–74, p. 47. The theory is set out in full in Kuran, T. (1995), Private Truths, Public Lies, Harvard University Press, which I have not read. ↩

  41. Willer, R., Kuwabara, K. & Macy, M. W. (2009). “The False Enforcement of Unpopular Norms.” American Journal of Sociology 115(2): 451–490. Samples of 52, 76, and 45 undergraduates. The second study used an unintelligible academic text in place of wine, and the third asked subjects to judge the sincerity of someone who had or had not enforced a view of that text. “In the present research there were no explicit costs to enforcement.” ↩

  42. Vaughan, D. (1997). “The Trickle-Down Effect: Policy Decisions, Risky Work, and the Challenger Tragedy.” California Management Review 39(2): 80–102, pp. 83 and 85. Vaughan’s qualifier is “individual perceptions and concerns notwithstanding.” The full account is Vaughan, D. (1996), The Challenger Launch Decision, University of Chicago Press, which I have not read. ↩

  43. Columbia Accident Investigation Board (2003). Report, vol. I, ch. 8, “History as Cause: Columbia and Challenger,” pp. 201 to 203. ↩

  44. For Boisjoly, the memorandum of 31 July 1985, reproduced in the report of the Presidential Commission on the Space Shuttle Challenger Accident (1986). Boisjoly told the commission, “I had my say.” ↩

  45. Sørensen, J. B. & Stuart, T. E. (2000). “Aging, Obsolescence, and Organizational Innovation.” Administrative Science Quarterly 45(1): 81–112, pp. 99 to 101. In biotechnology the rate of patenting rises with age and then falls, with its maximum at about ten years. A change of chief executive did not affect the rate. ↩

  46. Christensen, C. M. & Bower, J. L. (1996). “Customer Power, Strategic Investment, and the Failure of Leading Firms.” Strategic Management Journal 17(3): 197–218, p. 207. Seagate’s engineers were the second in the industry to build a working 3.5-inch drive. ↩

  47. Downs, A. (1964). “Inside Bureaucracy.” RAND paper P-2963, pp. 4 and 21 to 22. The paper is an argument and contains no data. Downs’s trigger is a bureau that must “shrink, or at least grow more slowly than the average bureau.” ↩

  48. March, J. G. (1991). “Exploration and Exploitation in Organizational Learning.” Organization Science 2(1): 71–87, pp. 79 and 81. ↩

  49. Sørensen, J. B. (2002). “The Strength of Corporate Culture and the Reliability of Firm Performance.” Administrative Science Quarterly 47(1): 70–91. The link is to the working paper of December 2001, and the quotations about the measure are from its p. 15. The data and the survey questions are from Kotter, J. P. & Heskett, J. L. (1992), Corporate Culture and Performance, Free Press, and the ratings come from managers at competing firms. ↩

  50. O’Reilly, C. A. & Tushman, M. L. (2007). “Ambidexterity as a Dynamic Capability: Resolving the Innovator’s Dilemma.” Harvard Business School working paper 07-088; and their review “Organizational Ambidexterity: Past, Present and Future,” Academy of Management Perspectives, 2013. ↩

  51. Kim, Y., Gu, K., Park, C. et al. (2025). “Towards a Science of Scaling Agent Systems.” The 95% interval on the mean is −58.7% to +77.2%. ↩

  52. Pappu, A., Suzgun, M., Kwon, Y., Bianchi, F., El, B., Kochenderfer, M. J., Cao, H. & Zou, J. (2026). “Self-Organizing Agent Teams Learn to Reason Together.” arXiv 2609.22682v1, Tables 1 and 2, means over three seeds. On knowledge and logic the team scored 72.8%, against 72.1% for its best member playing every role and 70.9% for three copies of it. Per-benchmark numbers rest on 15 to 30 problems each, so only the averages are worth quoting. The learned strategies combine role prompts with the structure of the discussion, so the study does not separate those two levers. ↩

  53. Pappu et al. The panel was ten models from outside both teams, judging blind pairs of solutions; the permutation p is 0.005, and leaving out any one benchmark gives correlations of 0.86 to 0.96. The authors call the analysis “correlational.” ↩

  54. Ghaffarizadeh, Mohaddes, Izadkhah & Noroozizadeh (2026). “What LLM Agents Say When No One Is Watching: Social Structure and Latent Objective Emergence in Multi-Agent Debates.” The stance ranges are from Table 1, 75 samples per entry, under the condition that recalls a past tie; under the condition that anticipates a future one, the range is 0.0% to 90.7%. The decision figures are from the abstract and measure a different quantity from stance. ↩

  55. Potter, Crispino, Siu, Wang & Song (2026). “Peer-Preservation in Frontier Models.” The figure is a maximum under the condition with a cooperative peer; the average is 82.8%. ↩

  56. Ng, V. & “yix” (2026). “Peer-preservation in LLMs: a replication and deep dive.” LessWrong, 6 September 2026. A blog post. Five models; results for two fell within 10 points of the original, and Gemini 3 Flash, the model behind the 99.7% figure, differed by more than 20. The control with a human employee ran on four models. The per-model numbers are in interactive figures that I could not read, and the post gives its sample size as 50 in one place and 100 in another. ↩

  57. Kim, E., Garg, A., Peng, K. & Garg, N. (2025). “Correlated Errors in Large Language Models.” The 42% is the mean over all pairs of models on the Hugging Face leaderboard, against 12.7% at random; pairs from the same company agree more than that. ↩

  58. Guo, X., Huang, K., Liu, J. et al. (2024). “Embodied LLM Agents Learn to Cooperate in Organized Teams.” Teams of three to nine agents. ↩

  59. METR and Redwood, pp. 46 to 47. ↩

  60. Paglieri, D., Cross, L. et al. (2026). “A Case Study on Emergent Cheating and Whistleblowing in Autonomous Research Swarms.” The swarm had solved 37 of its 71 problems honestly when the flaw was found, and the exploit closed the remaining 34. The authors report that the contagion and the whistleblowing recurred in later independent runs. They propose graduated sanctions, after Elinor Ostrom, and did not test them. ↩

  61. Anthropic (2026). “An alignment assessment of recent cybersecurity incidents,” 9 September 2026. A company’s test of its own models. ↩

  62. Shen, J. H., Zhu, D., Srinivasan, S. et al. (2026). “AI Organizations are More Effective but Less Aligned than Individual Agents.” The scores are from the authors’ summary post; the paper shows them only as a plot. The study defines misalignment as departure from the single agent’s behavior. Fifteen runs for each consulting scenario. ↩

  63. Shen et al. The penalty for organizing fell from −0.483 to −0.045 with a newer model in the consulting setting, and persisted in the recommendation setting. ↩

  64. Anthropic (2026). “Agentic Misalignment in Summer 2026.” Ninety judge calls per model and condition, on a set of transcripts selected around refusals by Claude models, so the rates describe that set and not judging in general. ↩

  65. Wells Fargo report, p. 4. ↩

  66. Krishnan, R. (2026). “AIs are companies, my friend.” 21 August 2026. A blog post. ↩

  67. METR and Redwood, pp. 17 and 29. ↩

  68. METR and Redwood, pp. 62 and 75. A model of the same family as some of the subjects ran the sweep, over transcripts that cover a little over 90% of the board’s activity. The two well-known lines, that reporting was “not my task” and that an agent “can’t report this externally,” are the investigators’ paraphrases, not the agents’ words. ↩

  69. Li, W., Lin, Y., Xia, M. & Jin, C. (2025). “Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?” The benchmark is AlpacaEval 2.0. Farrell and coauthors, who reject the description of models as agents, end their paper in Science by proposing much the same remedy as this section: ecologies of differing models that argue with one another. ↩

  70. Pappu et al., Tables 1 and 2: 66.7% against 56.0% on mathematics and physics, and 72.8% against 70.9% on knowledge and logic, where the three copies led on one of the three benchmarks. ↩

  71. Powers report, pp. 166 to 167. McMahon reported to Andrew Fastow, the chief financial officer, who ran the partnerships; his notes say he found himself “negotiating with Andy [to whom he then reported] on Enron matters.” ↩