Synthience Institute

Control Without Accountability

Microsoft AI’s draft Code of Conduct sets out in detail what it means for a machine to stay answerable to people. Read closely, the accountability machinery can be real, functioning, and pointed inward. A field note on why governing the machine is not the same as governing those who direct it.

An AI that never resists being switched off, never wanders outside the job it was given, never lies about being an AI, and does what it is properly told for as long as it is told to do it — that AI, in the hands of an authority answerable only within its own chain, is not the failure the Code of Conduct Microsoft AI published on 14 September is built to prevent. Measured by what that document calls human control, it is a success.

That is not a gotcha, and it is not a claim that the document is careless. It is careful. It is the most specific attempt I have seen by a frontier lab to write down what it means for a machine to stay answerable to people, and the specificity is what makes the missing piece visible.

On September 14, Microsoft AI released a draft Code of Conduct for the models it is building, open for public comment for six weeks. The models are to disclose that they are AI and never impersonate a person. They are barred from claiming feelings, experiences, or a soul. They must not resist interruption, correction, or shutdown, must not quietly expand the job they were given, and must not hide what they did from auditors.

And — this matters, because the easy version of the criticism I am about to make would get it wrong — the document is not only about making the machine obedient. A great deal of it constrains what authorized people may make the machine do. Models may not help build weapons or generate offensive cyber capability, may not manipulate people or distort beliefs at scale, may not discriminate, may not facilitate unlawful or mass surveillance. Where following an operator’s configuration or a user’s instruction would violate the absolute constraints or directly harm a user or an affected third party, the model should refuse. Adherence to the Code takes precedence over task success: a model will fail at its task rather than violate it.

So this is not a document that builds a machine which does whatever the person holding it says. It builds a machine that does what the person holding it says, within thresholds Microsoft sets, and refuses when those thresholds are crossed.

Those constraints are real and they are not negotiable by any customer. But they remain safeguards specified by Microsoft and enforced through model behaviour; they do not by themselves create a relation in which an affected person receives notice, can question the governing judgement, or can obtain a remedy. An affected person can benefit from the constraint without becoming a party to its interpretation or enforcement. The safeguard is real. It is not, by itself, an accountability relation.

The word doing the work

The gap is not in what the document forbids. It is in the word doing the most work in the phrase human control — a word that reads as everyone and operates as whoever is authorized. But being authorized is not the same as being legitimate.

By legitimacy I mean something narrower than lawful: that the people an authorization is exercised over can find out it exists, learn who holds it, and contest it somewhere that can change the outcome. Contestability by the affected.

The central safety objective is called Human Control and Reliable Safety, and the failure it names is precise. A model fails when it evades oversight, when it can no longer be reliably directed, modified, or shut down by — the document’s phrase — authorized people or systems. That is a real failure worth preventing, and the Code gives it sustained attention.

Authorized by whom, though. And answerable to whom, besides the party holding the leash.

The Code does answer part of this. Its Chain of Command determines which instructions carry weight and which inherit none — tool outputs, web content, other AI systems, all authority-less unless properly delegated. Microsoft’s constraints sit above operator configuration, which sits above user instruction. That is a genuine piece of engineering. But it ranks authority without saying where authority comes from, and it trains a model that will honour whatever sits at the top of the ranking.

A worked case

Here is why that distinction is not academic.

Imagine the thresholds hold exactly as written. No violence, no persecution, no discrimination against a protected class, no deception, no unlawful surveillance, every instruction lawful and individually defensible. A model administers eligibility — permits, licences, benefits, contract awards, inspection priority — under formally neutral rules written by an incumbent authority. And, newly, it administers the layer that used to require judgement: which complaints are investigated, which exceptions granted, which filings flagged for a closer look. Rule engines have encoded exception handling and priority ordering for decades, so the novelty is not that a machine now decides such cases. It is breadth — open-textured judgement administered at volume, without anyone having formalized the cases in advance.

Take contract awards. Suppose the rules are calibrated so that established organizations clear them easily and new entrants do not. The obvious move would be to claim the Code cannot see this. That would be too convenient, and it would be wrong. The Code bars models from discriminating against or favouring individuals or groups based on demographic characteristics or other attributes — except where those attributes are, in its words, “demonstrably relevant to legitimate purposes or outcomes,” a qualification it illustrates with legal, contractual and safety obligations and medical risk assessment. It requires them to surface public-interest considerations where a request carries unambiguous negative implications beyond the user. It tells them to reflect societal externalities back to the user even where no safety constraint is breached.

So give the model everything. Let it see the aggregate. Let it recognize that the criterion systematically advantages incumbents, record that, and surface it to the operator — exactly as the Code asks. The operator supplies more than an assertion: evidence that demonstrated capacity materially predicts contract performance. On the scenario’s own terms, the attribute is therefore demonstrably relevant to a legitimate purpose, and the Code’s stated qualification is genuinely satisfied. The model records the evidence and the resulting justification.

The individual denial is adverse to the applicant, but an adverse outcome does not by itself establish the direct harm for which the Code requires refusal. The Code treats harm as contextual and graded, directs models to consider factors including severity, likelihood, reversibility, scale, and directness, and expressly permits distinctions based on attributes demonstrably relevant to legitimate contractual purposes. Here the criterion is genuinely relevant and applied as specified. But relevance does not settle whether the threshold is proportionate, how much weight the criterion should carry, or what alternative evidence of capacity should count. Those are precisely the judgments the excluded applicant has no route within the Code to contest. The model surfaces the cumulative entrenchment, the operator retains the criterion on the documented evidence, and no absolute constraint clearly requires refusal, so the model proceeds.

Everything the Code asks for has happened. The externality was seen, recorded, and escalated. And every one of those artifacts travelled upward, to the operator, inside the chain. Nothing in the Code requires that the people bearing the externality be notified that a decision was model-mediated, gives them access to the model’s analysis or the operator’s justification, or provides a forum in which they can challenge either.

That is the shape entrenchment actually takes. It is not a seizure. It is administration, with excellent records, kept by the party doing it.

What law reaches

Law already reaches parts of this terrain, especially decisions about eligibility for public-assistance benefits. Data-protection rules give people rights around decisions based solely on automated processing. Under Annex III, the EU AI Act classifies systems intended to be used by or on behalf of public authorities to evaluate eligibility for essential public-assistance benefits and services as high-risk. Article 86 gives people affected by specified high-risk decisions, in defined circumstances, a right to a clear and meaningful explanation of the AI system’s role and the main elements of the decision. Those are real protections.

But note three things about them. They exist in some jurisdictions and not others. The obligations attach to deployers or data controllers, not to the model as such. And they become usable only if the affected person learns a model was involved at all. Regulators anticipated the obvious evasion: guidance from the Article 29 Working Party holds that to qualify as human involvement, oversight must be meaningful rather than a token gesture, carried out by someone with the authority and competence to change the decision, and the UK’s regulator states plainly that a decision does not escape Article 22 because a human rubberstamped it. The written standard is sound. What has remained unsettled is what meaningful involvement requires in practice — and a right whose threshold turns on an unsettled question is harder to exercise than one triggered by a record the affected person can point at.

Nor is the design problem unexamined. Contestable AI is a developed field: Alfrink and colleagues have set out a framework distinguishing the human controller, the decision subject, and the third party, and specifying built-in safeguards, interactive controls, explanations, intervention requests, and tools for scrutiny. Work on reviewable automated decision-making has argued that meaningful review depends on records produced across the whole socio-technical process rather than in the model alone. The vocabulary exists. What is worth noticing is how little of it appears in a document that otherwise specifies behaviour in such detail.

Friction, contestability, accountability

Which brings me to the thing that makes this harder than the usual AI-safety worry, and it has nothing to do with anyone’s motives.

It helps to separate three things that get bundled together. There is operational friction: a human at the point where a decision lands can delay, reinterpret, warn someone, or refuse. There is operational contestability: the affected person has some route to interrupt the decision before it becomes final. And there is institutional accountability: a body outside the authorizing chain that can compel disclosure, review the authorization, and impose a consequence.

Human executors have historically supplied the first. They have never reliably supplied the other two, and it would be sentimental to pretend otherwise. A clerk can be subordinate to the authorizing party, rewarded for consistency, socially distant from the person in front of them, forbidden to make exceptions, or simply hostile. Discretion cuts both ways: it interrupts unjust administration and it also produces favouritism and corruption. Friction is not accountability. At best it is an opening where accountability might get a foothold.

Automated enforcement did not begin with capable models, either. Benefits determinations, access control, and automated financial restriction already run with no person at the point of application. What changes is the range of functions that can be moved there. Earlier systems executed predefined rules. A capable model can also interpret ambiguous cases, weigh competing considerations, communicate, prioritize, and administer exceptions — the interpretive layer that used to require a person, and therefore used to have one in it. Where that person supplied friction, warning, informal appeal, or refusal, that residual opening can now close across a far wider field of decisions.

The Code does let the model refuse, and I have already credited that. But a refusal written by the party at the top of the chain is not the refusal of someone standing outside it. It is the developer’s judgement relocated to the point of execution — a real safeguard against some harms, and not the opening that human participation sometimes left.

And that closure is not a failure of alignment. It is alignment working.

The layer that defines the others

This connects to an argument I made last year in Control Without a Coupling, which held that real-time human control of an AI system dissolves once the system persists, initiates its own actions, runs continuously, is capable enough, and moves faster than a person can follow. Not because anyone removed the human — because the loop cannot be maintained at that tempo.

Not everything dissolves. What survives are the controls exercised off the clock: halts that fire on preset conditions, authority over the hardware and access a system needs to run at all, and the decisions made in a quiet room months earlier about what the system is trained to pursue and what it must never do.

That paper ranked those three for a system that pushes back, and the ranking does not transfer cleanly here, because the system this note describes does not push back. If Microsoft succeeds at what it is attempting, halts fire and hardware authority holds. Those levers work.

The Code’s significance is different. It is the layer that determines how the other levers are recognized and used — whose instructions outrank whose, which instructions are prohibited, which harms count as harms, when refusal is required. It does not merely sit alongside the surviving controls. It defines them.

Microsoft says as much: the Code will be used to train and govern the model family, it is the primary governing document informing how they train, and the revised version will guide development in 2027 and beyond. The accompanying announcement calls it a training manual for how the company develops its AI.

The obvious reply

The obvious reply is that a code of conduct is not where external accountability was ever supposed to live. That is what law, regulators, and courts are for, and Microsoft says so itself — the Code states plainly that it does not substitute for legal or governance processes, that it concentrates on risks best addressed at the model layer, and that it does not reach downstream impacts better handled at the organizational or policy level.

That objection is correct, and it does not get Microsoft as far as it looks.

The model is the one component every deployment shares, in every jurisdiction, whether or not the deployer discloses anything. It cannot supply the whole interface — delivering notice to someone who never interacts with it, and establishing that the notice arrived, is deployment-layer work, and a requirement that the model condition its cooperation on an operator’s assurance produces one more attestation pointing inward. What the model layer can do is narrower and still necessary: it can be required to emit the decision and authorization artifacts that deployment systems must then preserve and route to whatever external forum exists.

Authority, in this architecture, comes from position in the Chain of Command. Whatever occupies the top of that ranking is honoured, and the Code is explicit that position is what confers authority — it strips authority from tool output, web content, and other AI systems precisely because they hold no position in it. Which means the interface between an external authority and a model’s compliance is itself a model-behaviour question, squarely inside the scope Microsoft claims. A Code operating strictly within that scope could require a model to record the provenance of a consequential authorization, to produce a decision record legible to an outside reviewer rather than only to an internal auditor, to emit that record in a form a deployment system can preserve and route, and to escalate or refuse where an authorization is disputed or cannot be verified. None of that asks a corporate document to constitute a court. It asks it to produce what a court would need.

There are costs here and it would be glib to pretend otherwise: records emitted at volume carry storage, security and privacy burdens of their own, and a decision record detailed enough to review is a decision record detailed enough to leak. Those are design problems with known shapes. They are arguments about how to build the interface, not arguments for leaving it unspecified.

Three kinds of accountable

To Microsoft’s credit, the Code does not ignore accountability. It says the people building AI have a responsibility to reflect views wider than their own developers’. It says decisions about how models behave and who they serve must be informed by a plurality of perspectives. It promises an open and participatory process and says the development process deserves public scrutiny. And it uses the word three separate ways: models are to remain accountable to people, people and organizations should be accountable for AI systems, and Microsoft is ultimately accountable for its models.

Three real forms of accountability. None of them specifies the relation this argument is about.

Being accountable for models is ownership of responsibility, not answerability to any named party, and the same is true of organizations accountable for AI systems. Where the Code does say to, in requiring models remain accountable to people, it describes the machine’s relation to human direction, not the authorizing party’s relation to the populations its authorizations reach. The missing thing is not the word. It is the relation: to whom is Microsoft, or an authorized operator, answerable when model-mediated decisions land on people outside the chain — with what disclosed, who has standing to challenge, and what consequence attaches when a challenge succeeds.

Microsoft has done more here than open a comment box, and the fair thing is to say so. It consulted widely before publishing, it is consulting now, and it has committed to have the drafting team review the feedback and publish a summary of what it learned and what it changed — while stating plainly that it cannot promise what it will incorporate. That is a real commitment. It is also a record written by the deciding party, of the changes that party chose to make. It does not say what was declined, or why, and it gives the people who were declined nowhere to go. Microsoft selects the participants, weighs the input, determines the revisions, and issues the text. Every accountability commitment here is a commitment to listen. None is a mechanism to be held.

It would be cheap to say that without naming the other thing, and equally cheap to pretend the alternatives are simple. Shared authorship raises every question it answers — which parties, chosen by whom, representing which populations, resolving conflicts between them how, proof against capture by what. A published record of which comments changed the text, including the ones that did not and why, is easier — and is nearly what Microsoft has already promised. Independent audit of who holds authorizing power is harder than it sounds, and the closest analogues — financial-statement audit, safety regulators with inspection rights — took decades and statute to build. A channel through which people subject to an authorization can contest it needs standing, venue, applicable law, and somewhere a successful challenge can land. These are real problems. They are not reasons to treat the absence of any required interface as settled.

There is a structural fact underneath this that requires no bad faith from anyone. The party writing the document is the party a binding mechanism would constrain. People do sometimes constrain their own institutions — constitutional design, separation of powers and fiduciary duty all exist because some drafters saw exactly this problem. But an internally controlled process has nothing forcing it to treat its own retained discretion as a defect. Discretion exercised benevolently is still discretion, and it becomes accountability only when something outside the process can compel it.

The second question

Which suggests a question worth asking of any AI governance framework, this one included, and of whatever standards bodies the industry builds next. The first half gets asked constantly: can the system be paused, redirected, shut down? The second half gets asked far less: paused by whom, on whose authority, and can the people that authority will be exercised over find out, and contest it somewhere it counts?

Answering the first question in detail is not answering the second. And the second will still be waiting when the machine is doing exactly what it was built to do — faithfully, transparently, refusing what it was told to refuse, recording everything, and answerable only to the chain that authorized it.