What Must Remain Human?

Imagine a city deciding whether to close a neighborhood school. An AI system compares the alternatives. It forecasts the budget savings, the longer journeys, the staffing changes, and the likely educational results. For the sake of the thought experiment, grant the machine an extraordinary advantage: every forecast is correct.

The decision remains. A saving for the school system becomes a longer morning for particular children. Money released by the closure could improve teaching elsewhere. Keeping the school open would require accepting some other cost. The model can make those consequences visible, including consequences the officials would otherwise miss. The city still has to decide which burdens it is entitled to impose, on whom, and for what public purpose.

This is the harder question behind the book. Measuring errors helps us compare performance. It does not tell us who is entitled to decide, what limits apply, or who must answer to the people affected. If the only reason to keep a human in charge is that the human performs better, that reason disappears when a better model arrives.

What must remain human is the authority to choose and revise our purposes, the judgment required for decisions that belong to accountable people, and the obligation to answer those affected. AI can help with all of that work. Greater assistance makes the terms of that authority more important.

Human Authority Needs a Better Defense

It is tempting to defend human judgment by listing abilities machines supposedly lack. The list usually includes empathy, creativity, intuition, and an understanding of context. Some claims may describe a real limitation of a particular system. As a permanent defense, however, the list is fragile. It invites each technical improvement to become an argument for transferring another part of human authority.

I would rather concede the strongest possibility. A machine may recognize a pattern we miss, propose an option we would never have considered, or explain a difficult subject with greater patience than an overworked professional. Where those abilities improve people’s lives, we should want to use them. Requiring inferior work to preserve a familiar professional role would be an expensive way to protect a title.

The human claim rests on a different foundation. People are entitled to participate in decisions about their lives. Institutions exercising power must be able to justify it. Someone must decide what a successful outcome means, identify the limits of the authority being granted, and remain exposed to review when the decision causes harm. A better forecast can inform those obligations without discharging them.

This also means that being human confers no automatic competence. A negligent professional, a prejudiced official, or an exhausted reviewer can make a capable system worse. An institution invoking human judgment has to provide the conditions that make the judgment worth retaining. Otherwise, the phrase becomes a defense of whoever already holds power.

Someone Must Choose What Counts as Success

Return to the school. The software may be asked to minimize cost while maintaining an agreed educational standard. That objective already contains consequential choices. Which outcomes count as education? Over what period will they be measured? How much additional travel is acceptable? Which needs require protection even when satisfying them raises the average cost?

An advanced system can help discover those questions. It can compare competing definitions of success, expose a hidden assumption, and show how a preferred plan changes when a neglected cost enters the calculation. That is valuable analytical work. The people entitled to govern still have to authorize the standard by which the alternatives will be judged.

Authority can slip away through ordinary habits. A buyer accepts the supplier’s preset goal. A manager keeps an old performance measure without asking whether it still fits. Staff focus on the numbers that appear on the dashboard. Over time, the system’s ranking can become the accepted answer, even though leaders never clearly decided what the institution should value.

The remaining human duty is to bring that choice back into view. The city should be able to explain why a proposed saving justifies a particular burden and what evidence would change its position. Families should have a practical way to supply missing information and challenge the proposed tradeoff. The officials should be able to alter the objective when the public purpose requires it.

No objective is made legitimate simply by attaching the word human to its author. A school board’s authority is constrained by law, rights, public scrutiny, and the interests of the children it serves. Human ownership must include those constraints. It cannot mean that an official’s personal preference becomes immune from challenge because an official, rather than a machine, selected it.

The Person Affected Has a Voice

A treatment decision shows another part of the boundary. Suppose a patient faces options with different effects on survival, independence, discomfort, and time at home. Better AI could make the likely consequences much clearer. It could identify an overlooked treatment or help the patient understand uncertainty. Those improvements would strengthen the conversation between the patient and the clinical team.

The patient’s priorities still belong in that conversation. A prediction of what a person would probably choose does not amount to permission to choose on that person’s behalf. Even an accurate account of past preferences may miss a changed commitment, a new fear, or a different understanding of what makes treatment worthwhile.

The UK’s National Institute for Health and Care Excellence (NICE) gives this principle a practical form in its 2021 guideline on shared decision making. It recommends discussing how the aims of each healthcare option fit the person’s priorities and wider goals, explaining risks and benefits, and allowing time for questions. The options discussed include no treatment or no change to the existing plan. The guidance also calls for additional support when someone finds participation difficult.

AI can make that support more available. It can help a person prepare questions, translate information, organize concerns, or communicate through an accessible interface. Human agency includes the freedom to use assistance. A rule that equated unaided speech or technical fluency with authentic participation would exclude people whose access could be improved by the technology.

The clinical team retains its professional obligations, and the patient need not become the final technical checker of the system. Consent and clinical authorization serve different purposes. Where a patient lacks decision-making capacity, an authorized representative and the applicable safeguards govern the decision. A model’s reconstruction of likely preferences can contribute information within that process; it should not silently acquire the representative’s authority.

The broader lesson extends to work, education, and public services. A person affected by a decision has standing to say that the institution has misunderstood what matters. Sometimes the institution must act despite disagreement. It should still hear the objection, give reasons, and remain open to correction through the appropriate process.

Reserved Means Judgment Over the Decision

The book uses Reserved for decisions whose legitimacy depends on an accountable human making them. The category has a defined purpose and should remain narrow. The claim is strongest when a decision directly confers, removes, or finally determines a fundamental interest such as life, liberty, or a child’s development.

Principle 7 also requires a human signature for irreversible, high-stakes decisions outside Reserved. Reserved adds a further requirement: independent human judgment forms part of the decision’s legitimacy. Better performance by the machine, on its own, cannot satisfy that requirement.

The Doctrine’s established Reserved examples include a criminal sentence, the intentional decision to apply lethal force against a person, removal of a child, denial of liberty, and disclosure of a terminal diagnosis. These decisions differ in law and practice, but each requires more than an institution accepting an output because its system produced it.

Reserved does not cover every calculation or administrative task surrounding the decision. AI may organize evidence, test competing accounts, flag an inconsistency, or help prepare an explanation. Some bounded actions can proceed under deliberate upstream authorization. Chapter Thirteen’s distinction between autonomous defense against incoming objects and the intentional targeting of a person remains important: the authority concerns the consequential choice, rather than every mechanical step used to carry it out.

Nor does independent judgment require a person to reproduce every calculation. A physician can use an instrument whose internal engineering lies outside the physician’s expertise. A judge can consider specialist evidence without becoming a specialist in every field. The requirement is enough relevant competence, evidence, and context to judge whether reliance is warranted in this case, with a real route to further expertise when it is needed.

The reviewer must be able to change the outcome. A person instructed to approve every recommendation within a few seconds has been assigned a workload, not equipped with authority. If an institution cannot provide competent judgment where the decision requires it, that failure is a reason to change the deployment or restore the capacity. It cannot be cured by placing another name beside the result.

Chapter Twenty-One’s four-characteristic test concerns whether a use should be prohibited, restricted, or managed with ordinary safeguards. It weighs the size of the possible harm, whether the harm could be undone, whether anyone can remain meaningfully accountable, and whether the system can be controlled. Reserved concerns who must exercise judgment within an activity that is permitted. Permission to use AI in a field does not settle which decisions in that field may be handed over. Chapter Twenty-Five turns that distinction into the practical Accountability Test.

Judgment Has to Be Maintained

A human role can remain on an organization chart long after the ability to perform it has weakened. Experienced staff leave. Beginners stop encountering the difficult work through which judgment develops. The person authorized to challenge a system gradually becomes dependent on the same system for the explanation of whether its answer is sound.

Chapter Eighteen examined that loss through expertise and the outage test. Its consequence here is institutional. A society that wants some decisions to remain human has to preserve a way for people to learn how to make them. The retained authority needs a budget, a training path, and enough practice to survive contact with a difficult case.

That does not require recreating every inefficient task AI has removed. Institutions can select consequential tasks for supervised practice, compare judgments with later outcomes, and give novices increasing authority as competence grows. Time saved on routine work can support deeper preparation for the decisions that remain. Whether that happens depends on choices about staffing and development, rather than on the tool’s efficiency alone.

There is also a duty to make disagreement usable. An employee who spots a problem needs a route to someone who can act. A reviewer should be able to explain why an override was appropriate without treating every disagreement as insubordination. Repeated overrides should prompt examination of the system and its instructions. Repeated acceptance should prompt a different question: whether the system is performing well or the review has become automatic.

Accountability Includes Repair

Human authority is most convincing when someone is prepared to remain involved after a mistake. An institution that can produce a polished explanation but cannot reconsider the result has preserved the appearance of dialogue. The affected person needs access to an office with the authority and resources to put matters right.

The repair work itself may be heavily automated. AI can search for similar cases, identify inconsistent records, prepare notices, and estimate the consequences of a defect. The human institution must decide what remedy is owed, ensure that it reaches affected people, and change the process that allowed the problem. Chapter Twenty-Three’s account of administrative repair applies beyond government.

A review should ask what kind of failure occurred: an understandable mistake, misconduct, poor design, or a warning someone ignored. The answer determines what should change and who should face consequences. Blaming only the last person who handled the file can shield the people who set the rules or denied staff the resources to do the work. A serious review follows authority as well as proximity.

The same obligation applies to this book. AI helps produce its sentences and assemble evidence. My approval makes a claim part of the published argument, and I remain answerable for explaining, revising, or withdrawing it. That obligation is real only if I can inspect the supporting material, consider criticism, and change what readers receive. Approval carries a continuing commitment to the work.

Current Evidence: Human Participation Must Be Tested

Reviewed September 12, 2026.

A useful empirical warning comes from Michelle Vaccaro, Abdullah Almaatouq, and Thomas Malone’s systematic review, published in Nature Human Behaviour in October 2024. It combined 106 experiments reported in research published from January 2020 through June 2023. On average, human and AI combinations improved on humans working alone but performed worse than the better of the human alone or the AI alone. Adding a person therefore did not reliably produce the best-performing arrangement.

The results varied by task. Decision tasks showed average performance losses against that stronger baseline. Creation tasks had a more favorable pattern, although their pooled gain over the stronger baseline was not statistically significant. The authors identify limitations including possible publication bias and differences among study designs. These experiments measured task performance; they did not test whether an arrangement possessed legitimate authority or satisfied a duty to hear an affected person. Their publication window also limits what they establish about later models.

The finding puts a useful burden on institutions. A human review requirement should identify the judgment the person contributes and provide evidence that the arrangement works. If a reviewer routinely makes a reliable bounded process worse, the process should be redesigned. Where independent human judgment remains necessary, improving the tools, training, and allocation of work is part of meeting that requirement.

A different kind of evidence comes from the Court of Justice of the European Union’s February 27, 2025 judgment in Dun & Bradstreet Austria. The underlying dispute involved an automated credit assessment used in refusing a mobile telephone contract. Interpreting the General Data Protection Regulation (GDPR), the Court required an intelligible explanation of the procedure and principles actually applied to the person’s data. Simply communicating an algorithm would not provide the required concise and understandable explanation.

The ruling concerns specified GDPR rights around automated decision-making, rather than a universal rule for every AI use. It also provides a concrete example of visibility serving a person’s ability to challenge a decision. The important information is what was done with the person’s data and how it produced the result. A generic description of the software cannot substitute for that account. The Court separately required disputed trade-secret or third-party information to be provided to the competent authority or court for the necessary balancing of rights.

Together, the research and the judgment resist an easy assurance. Human participation has to be designed for a purpose, and an explanation has to be usable by someone with a reason to question the outcome. The evidence worth watching is whether those arrangements improve decisions and make correction possible under real operating conditions.

Accountability Analysis

Principles 2 and 7 carry the central argument. Capability can justify assigning more work to a system. The authority to set its purpose and the judgment required for a Reserved decision remain separate obligations. An institution should be able to identify both the person or office that authorized the arrangement and the qualified decision owner for any case requiring an individual judgment.

Principles 4 and 5 determine whether that ownership is practical. When action happens too quickly for individual review, the relevant choices about scope and safeguards must be made earlier. That upstream authorization can support legitimate bounded autonomy. It cannot turn a Reserved decision into an unrestricted machine choice merely by moving a human signature to the purchase order.

Principles 6 and 8 make the obligation durable. Consequences must reach the roles with authority to prevent, examine, and repair failure. Competence and control must survive repeated use. The school board in the opening example would therefore answer for its chosen objective and final decision, while retaining the capacity to revisit the plan as its effects become known. A supplier could answer for the quality and representations of its product without becoming the city’s substitute government.

Separate the forecast from the choice

Return to the school-closure example and accept its premise that every forecast is correct. Name one consequence the AI can describe and one tradeoff the city must justify. Who should have a voice? Who has authority to decide? What could lead that person or office to reconsider? Notice how much responsibility remains after uncertainty about the forecast disappears.

Accountability Verdict

What must remain human cannot be defined as whatever work the machine still performs badly. That would make human authority a temporary exception waiting for the next improvement. The enduring claim concerns our purposes, our standing in decisions that affect us, and the answerability of those empowered to decide.

In practice, retaining that authority requires institutions to do real work. A purpose must be chosen and defended. A consequential objection needs somewhere to go. A Reserved decision needs a competent human judgment over the actual case. When an error occurs, the institution must have the capacity to repair its effects and reconsider the process.

Return to the city with the excellent forecast. The officials may reasonably choose to close the school. They may reasonably decide that its continued presence is worth a cost the analysis has made explicit. Either way, the public should be able to see what the officials valued, challenge their reasons, and hold them to the consequences of the choice. The machine has helped them understand the decision. The city still owes its people an answer.

The next chapter asks how to establish, before granting a delegation, whether that answer will remain possible.

Sources and Notes

1. National Institute for Health and Care Excellence, Shared decision making, NICE guideline NG197, published June 17, 2021, especially recommendations 1.2.3-1.2.6 and 1.2.9-1.2.14. https://www.nice.org.uk/guidance/ng197/chapter/Recommendations. Accessed September 12, 2026. The guideline supports the account of patient priorities, available options, time, and assistance with participation. It is guidance on shared decision making, not a determination of AI authority or a universal statement of consent law. The chapter’s application to AI and its treatment of delegated authority are the book’s argument. Source record: DA-CH24-S1. Archive reference: pending.

2. Michelle Vaccaro, Abdullah Almaatouq, and Thomas Malone, “When combinations of humans and AI are useful: A systematic review and meta-analysis,” Nature Human Behaviour 8, 2293-2303, published October 28, 2024. https://doi.org/10.1038/s41562-024-02024-1. Accessed September 12, 2026. The preregistered review analyzed 106 experiments from 74 papers, with 370 effect sizes, covering publications from January 1, 2020 through June 30, 2023. The chapter distinguishes improvement over humans alone from improvement over the better standalone performer. The creation-task pooled effect was positive but not statistically significant. Findings concern measured performance in the included studies, not the legitimacy of delegation or the capabilities of all later systems. Source record: DA-CH24-S2. Archive reference: pending.

3. Court of Justice of the European Union, Dun & Bradstreet Austria, Case C-203/22, judgment of February 27, 2025, ECLI:EU:C:2025:117, especially paragraphs 57-66 and 67-76, and the operative ruling. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:62022CJ0203. Accessed September 12, 2026. The Court interpreted Article 15(1)(h) GDPR in the context of automated decision-making within Article 22(1), including intelligible explanations and the treatment of protected information. It supplied an interpretation of EU law for the referring court, rather than deciding the underlying national dispute. The chapter draws a wider accountability lesson while retaining the ruling’s legal scope. Source record: DA-CH24-S3. Archive reference: pending.