Red Lines

Most regulation assumes that an activity is allowed if the risk can be managed. Red lines begin from a different premise. Some uses of artificial intelligence may be unacceptable even when the system works exactly as designed.

That distinction matters because safety and legitimacy are not the same thing. A perfectly reliable system can still perform a task that society decides no machine should be authorized to perform. Better accuracy does not settle whether the delegation itself is legitimate.

Chapter Twenty asked how rules can adapt when technology changes. This chapter asks where adaptation should stop. A red line is a boundary that capability does not erase.

Risk Management Has a Limit

The modern language of AI governance is built around risk. Systems are evaluated, classified, tested, audited, monitored, and mitigated. That is sensible for most uses. Cars are dangerous, medicines have side effects, financial systems fail, and aircraft can crash. Society does not ban every consequential technology merely because it creates risk.

Risk management asks how to reduce the chance or severity of harm from an activity we are willing to allow. Mitigation is the work of reducing that risk. A red-line argument asks an earlier question: should this particular power be delegated at all? A more accurate system does not automatically answer that question.

Europe has already accepted this logic in civilian AI law. The EU AI Act prohibits specified practices rather than placing all of them into a high-risk compliance category. The current list includes harmful manipulation, exploitation of vulnerabilities, social scoring, certain predictive-policing uses, untargeted facial-image scraping, specified emotion recognition, some biometric categorization, and tightly constrained real-time remote biometric identification. A ninth prohibition covering systems that generate nonconsensual sexually explicit material or child sexual abuse material is scheduled to apply in December 2026.

The important feature is architectural. Europe did not say every dangerous use should receive a better audit. It said some uses cross a boundary. A red line begins where mitigation stops being the answer.

The Hardest Line Is the Human Target

Chapter Thirteen argued that the intentional decision to apply lethal force against a person belongs in the Reserved category. Defensive autonomy against incoming munitions and bounded action against materiel can be different because the object of the decision is different. Chapter Twenty-One now carries that distinction into the international treaty debate.

The United Nations and the International Committee of the Red Cross have sharpened their position ahead of the November 2026 Review Conference of the Convention on Certain Conventional Weapons. In an August 2026 joint appeal, the UN Secretary-General and ICRC President warned that the autonomous targeting of humans by machines is approaching a moral red line. The ICRC is urging states to use the Review Conference to launch negotiations on a new legally binding protocol.

The 2026 CCW process is not starting from a blank page. The Group of Governmental Experts has a rolling text addressing characterization, international humanitarian law, prohibitions and restrictions, risk mitigation, accountability, legal review, and context-appropriate human judgment and control across the life cycle of lethal autonomous weapon systems.

The unresolved question is what kind of human control is sufficient, over which functions, and whether some target categories should be excluded from autonomous selection and engagement altogether. The red line proposed here is narrower than a ban on military AI: a machine should not receive delegated authority to select a human being and intentionally apply lethal force to that person without a human decision owner for the strike.

Why the Boundary Is About Authority

The case for that line does not depend on claiming that machines will always be worse at targeting. Future systems may identify objects more accurately, react faster, and process more sensor information than a human operator. Those advantages can matter enormously in defense.

The objection is about authority. Killing a person is not merely an optimization problem whose legitimacy rises with accuracy. It is an exercise of coercive power that demands an accountable decision owner.

That is why Chapter Thirteen’s distinction between bounded defensive autonomy and autonomous human targeting survives improvements in capability. A defensive system can be authorized in advance to intercept an incoming object within a defined environment. The intentional selection of a person as the object of lethal force crosses a different moral and legal threshold.

Principle 7 supplies the test. Reserved does not mean a human must touch every button. It means the accountable human must possess authority over the decision itself. A machine may execute force. It should not own the decision that a person is the target.

There Is a Serious Counterargument

A credible red-line argument has to confront the strongest case against it. Military AI can potentially improve precision, reduce reaction time, protect soldiers, strengthen air and missile defense, and process information that overwhelms human operators. A rigid rule can also create perverse incentives if it prevents defensive systems from responding quickly enough to save lives.

The United States has promoted a different international instrument: the Political Declaration on Responsible Military Use of Artificial Intelligence and Autonomy. It is nonbinding and emphasizes compliance with international law, responsible human chains of command and control, senior oversight of high-consequence applications, rigorous testing, explicit uses, trained personnel, and the ability to disengage or deactivate systems showing unintended behavior.

That approach treats responsible military AI as broader than autonomous weapons and preserves the possibility that autonomy can be lawful and beneficial when bounded and governed. The dispute is therefore between different theories of where law should draw the boundary: prohibit specified autonomous decisions in advance, or regulate systems through existing humanitarian law plus testing, command responsibility, and operational controls.

The Delegation Doctrine supports bounded autonomy where the objective, environment, authority, and stop conditions can be specified. It becomes much less permissive when the delegated decision itself is Reserved.

The Concealment Test

Chapter Thirteen introduced another problem: “A prohibition that cannot survive concealment is a policy, not yet a control.” That line becomes more important when AI capabilities can be copied, distilled, routed through intermediaries, or hidden inside ordinary technical workflows.

A red line therefore needs more than moral clarity. It needs an enforcement architecture that can survive attempts to evade it.

The problem is visible outside weapons. Anthropic’s September 2026 threat-intelligence report describes actors using frontier AI in biological research, cyber activity, surveillance, weapons-related intelligence, and other sensitive domains. The company reports that some sophisticated biological users tried to preserve plausible deniability by splitting work into requests that could look individually legitimate. Anthropic says stronger safeguards on its most capable models pushed some activity toward weaker systems, while other dual-use work remained difficult to classify because beneficial and harmful research can use overlapping knowledge.

These are provider reports about activity observed on Anthropic’s own systems, not independent proof of every underlying allegation. They still illustrate the governance problem: intent can be hidden, tasks can be fragmented, and a rule aimed only at obvious malicious prompts can be routed around.

The enforcement question is part of the red line from the beginning. Who can detect the prohibited delegation? What records exist? Who can investigate? What happens when a model is copied or the capability moves outside the original provider? A red line without a concealment strategy invites compliance by the honest and advantage for the evasive.

Biology Shows Why Access Can Be the Boundary

Biology presents a harder problem because the same capability can support medicine or catastrophe. A system able to reason deeply about biological mechanisms may help develop therapies, improve preparedness, and accelerate scientific discovery. Restricting the knowledge itself can also restrict the defense.

Both major frontier providers now describe biological capability as an area requiring stronger safeguards. Anthropic’s Responsible Scaling Policy uses escalating safeguards for models that cross dangerous capability thresholds and describes access controls, monitoring, red-teaming, and security measures. OpenAI’s 2026 biodefense program similarly argues for giving advanced biological capabilities to trusted developers and government partners while building safeguards and evidence around deployment.

This suggests a different kind of red line. The boundary need not always be “the model may never know this.” It can be “this capability may not be provided without a trusted-access architecture.” That distinction preserves legitimate research while refusing unrestricted delegation of a capability whose misuse could create catastrophic consequences. It also reflects Principle 2: capability is not permission.

Some red lines belong at the action. Others belong at access.

Cyber Capability Creates the Same Pressure

Cybersecurity makes the boundary problem even less tidy. The same system that discovers a vulnerability for a defender can help an attacker exploit it. A blanket prohibition on capable cyber reasoning would disable valuable defensive work. Unrestricted release can create a different danger.

OpenAI reported in September 2026 that its Astra model had crossed what the company calls a Critical cybersecurity capability threshold: with appropriate tools and access, the model could find previously unknown vulnerabilities and develop exploitation methods across well-protected systems without a person guiding every step. The company described stronger safeguards and security measures around that capability.

The claim is a provider evaluation and should be read as such. Its importance is the governance implication. As a system becomes more capable, blocking particular answers may no longer be enough. The boundary may also depend on who gets access, which tools the system may use, how its activity is watched, how the model is protected from theft or unauthorized copying, and what it may do without asking a person.

A red line can govern the combination of capability and permission rather than the existence of capability alone.

Red Lines Need Due Process

Hard boundaries create their own risks. A government can call something a red line to avoid ordinary scrutiny. A company can define a safety threshold that conveniently protects its market position. An emergency restriction can become permanent long after the evidence changes.

Hard limits need clear procedures. Define the prohibited act and identify the institution enforcing the rule. Make the supporting evidence available for review where security permits. State any exceptions narrowly and openly. Give affected people and organizations a way to challenge how the rule classifies or treats them. That opportunity to challenge authority is part of what due process means here.

The EU AI Act offers one civilian model. Its prohibited-practices framework identifies categories lawmakers judged unacceptable, while Commission guidance explains how the prohibitions should be interpreted. The guidance is nonbinding, leaving authoritative interpretation to courts. The system therefore separates the statutory boundary, administrative interpretation, and judicial review.

A red line gains legitimacy when the institution drawing it is also constrained by it.

When Does a Boundary Deserve to Be Red?

The phrase red line is rhetorically attractive, which is precisely why it needs a demanding test. Governments, companies, activists, and industries can all benefit from declaring that their preferred boundary is morally nonnegotiable. A serious doctrine needs a way to separate a genuine limit from ordinary policy disagreement.

Four characteristics make the case stronger. First is magnitude: failure could produce catastrophic harm or destroy a fundamental right. Second is irreversibility: once the action occurs, ordinary appeal or compensation cannot meaningfully restore what was lost. Third is accountability failure: delegating the decision would make it impossible or artificial to identify a human owner with authority over the act. Fourth is controllability: the boundary can be defined and enforced with enough precision that it does not become a vague prohibition on disfavored technology.

No single characteristic is sufficient. A catastrophic risk may still be best managed rather than prohibited if the beneficial and harmful uses cannot be separated. An irreversible decision may remain legitimate for automation if the human authority was exercised earlier through a genuinely bounded authorization. A morally compelling prohibition that cannot be detected or enforced may need a different control architecture before it can function as law.

The three forms of boundary answer different questions. A prohibited use says an action must not be allowed. Controlled access limits who may use a capability and under what conditions. A Reserved decision keeps final authority with an accountable human, even when AI supplies substantial assistance. Restrictions on access do not, by themselves, settle which decisions an authorized user may delegate.

The purpose of a red line is not to preserve human activity for sentimental reasons. It is to preserve accountable authority where delegation would otherwise change the moral or institutional character of the decision.

Current Evidence: November Is the Test

Reviewed October 12, 2026.

The clearest limits on delegation may emerge where mistakes cannot be recalled. In September 2026, an American and a Chinese security scholar participating in a Brookings-Tsinghua Track II dialogue proposed safeguards for military AI modeled partly on nuclear-risk controls. Their recommendations would bar AI from autonomously deciding whether nuclear weapons are used or autonomously attacking the other country’s nuclear command, control, and communications systems, and would keep decisions to launch consequential cyberattacks under human control. These were expert proposals, not adopted U.S.-China policy. Their logic is significant. Some decisions are too consequential for accountability after the fact. For those decisions, responsibility requires retaining human authority before the action occurs. The central governance question is therefore not simply how powerful an AI system may become, but which permissions society refuses to delegate at all.

An October 11, 2026 report described Booz Allen Hamilton’s tests of advanced AI models in an isolated industrial laboratory. The company reported completion of eight test objectives, including moving a robotic arm. Human operators approved exploits and actions with physical effects. These results demonstrate laboratory capabilities under supervised conditions.

For an institution, permission to inspect machinery and permission to change its behavior require different boundaries. A useful test must trace which physical processes an assistant can reach through connected systems, verify which changes require authorization, and confirm that an operator can stop the equipment safely. A software access limit should be tested against the physical consequence it is meant to prevent.

The autonomous-weapons debate is approaching a concrete decision point. The Seventh Review Conference of the Convention on Certain Conventional Weapons is scheduled for November 16-20, 2026. The ICRC is urging states to launch negotiations on a legally binding protocol, and the UN Secretary-General and ICRC President renewed their joint call in August for prohibitions and restrictions on autonomous weapon systems.

The diplomatic base is broader than it was several years ago. In December 2025, the UN General Assembly adopted a resolution on lethal autonomous weapons systems by 164 votes to 6, with 7 abstentions. The vote does not establish agreement on treaty language, but it demonstrates that the issue has moved well beyond a specialist arms-control discussion.

The remaining disagreement is substantive. CCW documents show support among many states for new prohibitions and restrictions, including stronger rules around systems targeting people. Other delegations have argued that existing international humanitarian law is sufficient or that new legally binding prohibitions are unnecessary. The June 2026 rolling text continues to work through those differences rather than pretending they have disappeared.

Civilian regulation is also moving from risk management toward prohibition. Eight EU AI Act prohibited practices have applied since February 2025. A ninth prohibition covering AI systems that generate nonconsensual sexually explicit or intimate content or child sexual abuse material is scheduled for December 2, 2026.

Frontier-model providers are simultaneously creating their own capability thresholds. Anthropic updated its Responsible Scaling Policy in 2026 as biological and other capabilities changed. OpenAI reported a critical cybersecurity threshold for Astra in September and has argued publicly for mandatory capability-based national safety regulation. OpenAI systems are used in this Living Edition’s production workflow, and Anthropic systems are used in the external critique workflow, so both providers’ claims and policy positions are treated here as disclosed provider evidence rather than neutral authority.

The pattern is becoming clearer. Governance is moving toward three different kinds of boundary: prohibited uses, controlled access to dangerous capabilities, and Reserved decisions that require a human decision owner. The November weapons conference will test whether states can turn one of those boundaries into international law.

Accountability Analysis

Principles 2, 3, 5, and 7 carry most of this chapter. Principle 2 is the foundation: capability is not permission. A system’s ability to perform a task does not establish that anyone should be authorized to delegate the task to it.

Principle 3 requires the boundary to be written. A red line that exists only as an aspiration will collapse under pressure. The prohibited action, controlled capability, or Reserved decision has to be defined well enough that developers, deployers, operators, regulators, and courts can recognize it.

Principle 5 moves accountability upstream when speed and architecture make last-second human review fictional. If a dangerous capability can operate faster than meaningful intervention, the control has to move earlier: model evaluation, access, deployment conditions, tool permissions, procurement, authorization, and system design.

Principle 7 identifies the hardest category. Some decisions should remain Reserved because the accountable human must possess authority over the decision itself. The intentional decision to apply lethal force against a person is the clearest case developed in this book.

At the highest stakes, red lines still require the Accountability Chain’s five questions: Who answers? What is the boundary? Can the owner see it? Can the owner stop it? Do the consequences reach the owner?

Name the boundary

Choose a proposed restriction described in this chapter. Is it a prohibited use, controlled access, or a Reserved decision? State what it would allow and what it would prevent. Then identify who would enforce it and how a mistaken application could be challenged. Keep the proposal’s status visible: an expert recommendation, a provider policy, and an enacted law carry different authority.

Accountability Verdict

Red lines should be rare. If everything becomes a red line, the phrase loses meaning and ordinary risk management becomes impossible. The category should be reserved for delegations where the harm is catastrophic, the right at stake is fundamental, the action is difficult to reverse, or the delegation itself would destroy meaningful human accountability.

Three forms are emerging. Some uses should be prohibited. Some dangerous capabilities should require controlled access. Some decisions should remain Reserved to a human decision owner.

The burden rises with the hardness of the boundary. A red line needs a clear definition, an enforcement path, a concealment strategy, evidence that can be revisited, and due process for the institution applying it.

The machine’s capability can move. The line around human authority does not have to move with it.

Chapter Twenty-Two turns to a different danger. Even when no single use crosses a red line, extraordinary power can accumulate in the organizations that control the systems.

Sources and Notes

1. International Committee of the Red Cross, “Advocacy paper: A key opportunity to prevent the development of unacceptable autonomous weapons,” June 17, 2026; and ICRC/United Nations, “Renewed call from UN Secretary-General and ICRC President to adopt rules on autonomous weapon systems,” August 25, 2026. https://www.icrc.org/en/article/advocacy-paper-key-opportunity-prevent-development-unacceptable-autonomous-weapons and https://www.icrc.org/en/statement/renewed-call-un-secretary-general-and-icrc-president-adopt-rules-autonomous-weapons. Accessed September 12, 2026. The ICRC urges the November 2026 CCW Review Conference to launch negotiations on a new protocol and, with the UN Secretary-General, identifies autonomous targeting of humans by machines as approaching a moral red line. The ICRC is an advocacy and humanitarian institution with an explicit policy position; its position is attributed as such. Source record: DA-CH21-S1. Archive reference: pending.

2. United Nations Office for Disarmament Affairs, Convention on Certain Conventional Weapons, Group of Governmental Experts on Lethal Autonomous Weapons Systems, 2026 documents, including the Chair’s rolling text as of June 5, 2026 and CCW/GGE.1/2026/WP.2. https://meetings.unoda.org/meeting/79329/documents. Accessed September 12, 2026. The GGE materials address characterization, IHL, prohibitions and restrictions, risk mitigation, legal review, accountability, and context-appropriate human judgment and control across the life cycle. The chapter uses these documents to describe the negotiating architecture, not to imply consensus on a treaty. Source record: DA-CH21-S2. Archive reference: pending.

3. United Nations General Assembly, Resolution A/RES/80/57, “Lethal autonomous weapons systems,” adopted December 1, 2025. https://docs.un.org/en/A/RES/80/57. Accessed September 12, 2026. The resolution was adopted by 164 votes to 6 with 7 abstentions. The vote demonstrates broad engagement with the issue. Source record: DA-CH21-S3. Archive reference: pending.

4. U.S. Department of State, “Political Declaration on Responsible Military Use of Artificial Intelligence and Autonomy,” 2023, archived text and implementation materials. https://2021-2025.state.gov/political-declaration-on-responsible-military-use-of-artificial-intelligence-and-autonomy. Accessed September 12, 2026. The nonbinding declaration calls for compliance with international law, responsible human chains of command and control, senior oversight of high-consequence applications, explicit uses, trained personnel, rigorous lifecycle testing, and safeguards including disengagement or deactivation. The chapter presents this as an alternative governance architecture rather than as a rebuttal to the treaty position. Source record: DA-CH21-S4. Archive reference: pending.

5. European Commission, “AI Act,” current risk-based framework and prohibited-practices guidance. https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai and https://digital-strategy.ec.europa.eu/en/library/commission-publishes-guidelines-prohibited-artificial-intelligence-ai-practices-defined-ai-act. Accessed September 12, 2026. Eight prohibited practices have applied since February 2025. The current framework adds a ninth prohibition, scheduled for December 2026, concerning systems that generate nonconsensual sexually explicit or intimate content or child sexual abuse material. Commission guidance is nonbinding, with authoritative interpretation reserved for the Court of Justice of the European Union. Source record: DA-CH21-S5. Archive reference: pending.

6. Anthropic, “Responsible Scaling Policy,” current version and 2026 update history. https://www.anthropic.com/responsible-scaling-policy. Accessed September 12, 2026. Anthropic describes escalating safeguards tied to dangerous model capabilities, including biological risks, access controls, monitoring, red-teaming, and security measures. These are provider policies and capability judgments. Anthropic systems are used in this Living Edition’s external critique workflow; see the transparency statement. Source record: DA-CH21-S6. Archive reference: pending.

7. Anthropic, “Countering misuse of AI: September 2026,” September 2026. https://www.anthropic.com/threat-intelligence-report-september-2026. Accessed September 12, 2026. Anthropic reports observed misuse and suspected misuse involving biological research, cyber activity, surveillance, weapons-related intelligence, and other sensitive domains. It describes the difficulty of distinguishing malicious from beneficial intent in dual-use biological work and argues that classifiers alone are insufficient for some frontier capabilities. These are provider observations and allegations, not independently adjudicated findings. Anthropic systems are used in this Living Edition’s external critique workflow; see the transparency statement. Source record: DA-CH21-S7. Archive reference: pending.

8. OpenAI, “Biodefense in the Intelligence Age,” June 4, 2026; and “Strengthening societal resilience with Rosalind Biodefense,” May 29, 2026. https://openai.com/index/biodefense-in-the-intelligence-age/ and https://openai.com/index/strengthening-societal-resilience-with-rosalind-biodefense/. Accessed September 12, 2026. OpenAI argues that advanced biological AI can support medicine and preparedness while creating biological-security concerns and describes a trusted-developer and government-partner model for defensive access. These are provider claims and policy choices. OpenAI systems are used in this Living Edition’s production workflow; see the transparency statement. Source record: DA-CH21-S8. Archive reference: pending.

9. OpenAI, “Path to Astra: critical capabilities and frontier safeguards,” September 1, 2026; and “Pacing model development in an era of cyber-critical capabilities,” August 18, 2026. https://openai.com/index/path-to-astra/ and https://openai.com/index/pacing-model-development-cyber-capabilities/. Accessed September 12, 2026. OpenAI reports that Astra meets its Critical cybersecurity capability threshold and describes safeguards and security measures around that capability. The chapter uses this provider evaluation to illustrate governance moving from content restrictions toward access, tool permissions, monitoring, and model security as capability increases. OpenAI systems are used in this Living Edition’s production workflow; see the transparency statement. Source record: DA-CH21-S9. Archive reference: pending.

10. OpenAI, “The AI policy window is open. We need to act,” September 9, 2026; and “OpenAI’s Frontier Governance Framework,” May 28, 2026. https://openai.com/index/ai-policy-window/ and https://openai.com/index/openai-frontier-governance-framework/. Accessed September 12, 2026. OpenAI calls for mandatory capability-based national safety regulation and describes governance covering cyber, CBRN, harmful manipulation, loss of control, incident response, external input, and framework updates. These are provider policy positions, not independent evidence that the proposed framework is sufficient. OpenAI systems are used in this Living Edition’s production workflow; see the transparency statement. Source record: DA-CH21-S10. Archive reference: pending.

11. Melanie W. Sisson and Tianjiao Jiang, “Advancing human control of military AI,” Brookings Institution, September 9, 2026; and Eduardo Baptista and Laurie Chen, “US, China security experts propose nuclear-style safeguards for AI risks,” Reuters, September 17, 2026. https://www.brookings.edu/articles/advancing-human-control-of-military-ai/ and https://www.reuters.com/world/china/us-china-security-experts-propose-nuclear-style-safeguards-ai-risks-2026-09-17/. Accessed September 18, 2026. The Brookings page presents recommendations by American and Chinese participants in a Brookings-Tsinghua Track II dialogue and states that the recommendations reflect the authors’ views. They include human authority over decisions to initiate cyberattacks on nuclear command, control, and communications systems and critical infrastructure, red lines against autonomous decisions to use nuclear weapons or attack those systems, a shared meaning of human control, and an AI-incident hotline. Reuters reported that neither government had publicly endorsed the proposals. This chapter treats them as expert proposals, not bilateral policy. Source record: DA-CH21-S11. Editorial MODIFY 017; formal Update 032. Archive reference: pending.

12. Booz Allen Hamilton, “Booz Allen Tests Super Intelligence in Operational Technology, Industrial Control Systems, and Physical Equipment,” undated public research summary, https://www.boozallen.com/insights/cyber/frontier-si-reaches-ot-systems.html; and Jessica Lyons, “AI systems are fully capable of carrying out nightmare attacks against infrastructure and nobody’s ready,” The Register, October 11, 2026, 12:30 UTC, https://www.theregister.com/security/2026/10/11/ai-systems-are-fully-capable-of-carrying-out-nightmare-attacks-against-infrastructure-and-nobodys-ready/5302450. Accessed October 12, 2026. Company-reported supervised lab capability; independent coverage is not experimental validation. The report describes supervised tests in an isolated industrial laboratory, not an uncontrolled attack on a working factory. Practical recommendations are the author’s analysis. Source record: DA-CH21-S12. Update 087. Archive reference: pending.