The Algorithm That Knows You

Your phone does not need to hear a confession to learn something private. A location trail can reveal a place of worship, a medical clinic, a political event, a military base, or a shelter. Browsing history can reveal fear before a person has named it. A shopping cart can reveal urgency. A pattern of pauses, clicks, searches, and returns can reveal what attracts attention and what causes hesitation.

Artificial intelligence makes those fragments more valuable because prediction can turn scattered behavior into an actionable profile. The system does not need a complete biography. It needs enough signal to estimate what you are likely to do next.

That changes the privacy question. The central problem is no longer only who has your data. It is who can use that data to predict you, who can act on the prediction, and whether you can see or contest what happens afterward.

Privacy is not only what you reveal. It is what can be inferred.

The Profile You Never Wrote

Privacy discussions often start with collection: what did you provide, did you agree, and who received it? Prediction adds another question. An inference is a conclusion drawn from clues. A company can use patterns in your behavior to infer something you never entered on a form. That conclusion may be accurate or mistaken; either way, it can affect how you are treated.

The Federal Trade Commission’s case against Mobilewalla showed how ordinary advertising infrastructure can become a surveillance system. The FTC alleged that the company collected location information from real-time advertising auctions and other sources, including information tied to visits to health facilities and places of worship. It also alleged that location data was used to build audience segments, including segments related to pregnancy and political protest activity. The final 2025 order barred Mobilewalla from selling sensitive location data and restricted its use of advertising-auction data.

The important feature is not merely that location was collected. Location became inference. A device appeared in one place, then another, and software converted movement into conclusions about health, religion, politics, and identity. The person did not have to volunteer those labels.

Data brokers make this especially difficult because the relationship is often indirect. The consumer may interact with an app, website, retailer, or advertiser while the data moves through intermediaries the consumer has never heard of. The profile can therefore become more detailed as the person becomes less able to name who holds it.

The old question was: what do they know about me? The harder question is: what have they concluded about me?

Prediction Becomes Intervention

Chapter Twelve introduced the authority jump, the point where machine output becomes state action. Here the jump is softer but still consequential. A prediction becomes an intervention when it changes the price, message, recommendation, ranking, opportunity, or environment presented to a person.

The distinction matters. A system can predict that one customer is more willing to pay than another. Nothing has happened yet. The moment the prediction changes the price shown to that customer, the model has crossed from description into action.

The Federal Trade Commission now uses the term personalized pricing for practices in which personal data helps estimate what an individual consumer may be willing to pay. On August 19, 2026, the FTC sought public comment on a proposed enforcement policy statement warning that undisclosed use of personal data for personalized pricing can raise deception or unfairness concerns. FTC Chairman Andrew Ferguson said consumers ordinarily expect a listed price to be the same price other consumers see at the same place and time. The Commission also acknowledged an important limit: Congress has not given it authority to prohibit personalized pricing in all circumstances. Its proposed approach instead focuses on deceptive or unfair undisclosed uses of personal data.

The economic mechanism is easy to understand. A company no longer has to ask what the market will bear in the aggregate. It can ask what this person will bear, now, in this context.

Prediction becomes power when it changes what happens next.

The Persuasion Advantage

Prices are only one intervention. Messages can also be personalized to the person most likely to respond.

A 2025 Nature Human Behaviour study tested whether GPT-4 could persuade people in short online debates. Participants were randomly assigned to debate either another person or GPT-4, and in some conditions the opponent received basic demographic information about the participant. The strongest result came from GPT-4 with access to that personal information. Compared with the human baseline, that condition produced an estimated 81.2% increase in the odds of greater post-debate agreement; when the two sides were not equally persuasive, the personalized GPT-4 condition was more persuasive 64.4% of the time.

The study was controlled, short, and artificial. A September 2026 author correction also matters: the direct comparison between personalized GPT-4 and non-personalized GPT-4 did not reach conventional statistical significance. The experiment therefore does not prove that personalization itself caused the entire advantage. It does establish that a conversational AI given personal information can be highly persuasive relative to a human baseline, which is enough to make the governance question real.

Traditional microtargeting means selecting a message for a narrowly defined group of people. A conversational AI can go further by adjusting its argument as a person responds. It can answer objections, change its tone, and emphasize different information during the same exchange. The influence can change from sentence to sentence.

Personalization therefore changes more than relevance. It can change the persuasive strategy itself.

The accountability question is not whether persuasion is inherently wrong. Teachers persuade. Doctors persuade. Friends persuade. Advertisers persuade. Political campaigns persuade. The question is whether the person can understand who is trying to influence them, what information is being used, what the objective is, and whether the system is optimizing for the person’s interests or someone else’s.

Architecture Has Already Allocated Attention

Chapter Thirteen used the line “architecture has already allocated attention” to describe military targeting systems. The same principle applies to ordinary digital life.

A recommender system does not need to order a person to watch a video, click a post, buy a product, or remain on a platform. It ranks possibilities. It decides what appears first, what is repeated, what disappears, and what receives another chance to capture attention. The user still chooses, but the architecture chooses the menu.

In February 2026 the European Commission preliminarily found TikTok in breach of the Digital Services Act over addictive design features including infinite scroll, autoplay, push notifications, and a highly personalized recommender system. In July it issued similar preliminary findings against Instagram and Facebook. The Commission said the design of these services could push users toward compulsive behavior and reduce self-control. TikTok denied the accusations, and both TikTok and Meta retain the right to examine the Commission file and respond before any final decision. These remain preliminary enforcement findings, not final judgments.

Courts are examining the architecture as well. In March 2026 a Los Angeles jury found Meta and Google negligent in the design of Instagram and YouTube in a bellwether social-media-addiction case involving a young woman who had used the platforms as a minor. The jury awarded $6 million in damages, with $4.2 million attributed to Meta and $1.8 million to Google. Both companies said they disagreed with the verdict and planned to appeal. The case focused on platform design rather than the content users posted, making it a useful American counterpart to the European regulatory theory.

The important point for this book is architectural. Influence can emerge from a sequence of small design choices rather than one explicit command. The system learns what keeps a person engaged, the interface removes stopping points, and the next recommendation arrives before the person has decided whether to continue.

Architecture can manipulate without issuing a command.

The Objective Is the Hidden Decision

Personalization is often described as though the system simply becomes more helpful by knowing the user better. That description leaves out the most consequential variable: what the system is trying to optimize.

The goal changes how a system uses the same personal information. A video service trying to keep you watching may make different recommendations from one trying to leave you satisfied with your time. A retailer trying to turn a visit into a purchase may treat hesitation as a reason to apply more pressure. A political campaign may seek the argument most likely to change your view.

This is why personalization cannot be governed only by asking whether the prediction is accurate. A perfectly accurate prediction that a person is anxious, lonely, impulsive, financially stressed, or unusually persuadable can make an intervention more effective while making the underlying relationship less legitimate. Accuracy can increase the need for a boundary.

Principle 5 points upstream again. The hidden decision is what the system is told to achieve. An objective function is the rule it uses to measure success. Does success mean a useful answer, a completed sale, or another minute spent watching? Once people choose that goal and put it into the product, the system can make millions of individual choices without an employee approving each one. Accountability therefore reaches the people who chose the goal and the rules for pursuing it.

A personalized system does not merely learn the user. It chooses which version of the world to show the user.

Consent Cannot Carry the Whole Load

Consent remains essential, but modern prediction systems make it a weak foundation if it has to carry the entire accountability structure.

A person may technically agree to a privacy policy while having no practical understanding of how many intermediaries receive the data, which inferences will be created, how long the profile will persist, or what future systems will use it for. The problem becomes sharper when the output is not a stored fact but a probability: likely pregnant, likely affluent, likely persuadable, likely to churn, likely to pay more, likely to click, likely to support a position.

The Federal Trade Commission’s 2026 orders involving Cox Media Group and two marketing firms provide an odd but revealing counterexample. The FTC alleged that the firms falsely claimed an AI-powered “Active Listening” service could target ads based on conversations captured from smart devices and falsely suggested consumers had opted in. According to the FTC, the service actually used no voice data. It relied on other marketing inputs, including email lists purchased from data brokers, and the geographic targeting did not work as advertised.

That distinction matters because people often explain uncanny targeting by assuming the microphone must be listening. Sometimes the real system needs no microphone. Location, browsing, purchases, identifiers, brokered lists, and behavioral patterns can already support remarkably specific inference. The FTC also said that merely clicking through terms of service would not amount to opt-in consent for the invasive voice-data collection the companies had claimed their service performed.

The privacy problem is therefore broader than eavesdropping. A system can know enough to act on a person without ever hearing the secret the person believes was overheard.

The Surveillance Boundary

Chapter Thirteen introduced the 2026 dispute between Anthropic and the Department of Defense over provider safeguards. One half concerned autonomous weapons. The other concerned domestic surveillance, and it belongs here.

Reuters reported that Anthropic refused to remove restrictions barring use of Claude for mass U.S. surveillance. The Pentagon argued that contracted AI should be available for lawful military purposes under government standards. In March the department designated Anthropic a supply-chain risk. In August a federal judge ruled the designation unlawful, finding constitutional and statutory defects; another case remained pending.

The dispute is important because both sides were arguing about who owns the boundary. A model provider claimed authority to preserve a use restriction after sale. The government claimed authority to decide lawful uses of a tool it procures.

The White House added another layer in June 2026. National Security Presidential Memorandum 11 directed the national security enterprise to accelerate AI adoption while stating that AI must not be used for unauthorized or unlawful surveillance of Americans. It also states that commanders, directors, and agency heads remain responsible and accountable for meeting those obligations at every level of command. The memorandum requires contractual or other protections so commercial entities cannot disable, degrade, or materially modify mission systems without federal approval, and it directs agencies, to the maximum extent permitted by law, to terminate contracts with companies that repeatedly show a pattern of conduct inconsistent with the memorandum’s policies.

Those positions are not necessarily contradictory, but they expose the unresolved line. A provider can prohibit a category such as mass domestic surveillance. Government policy can prohibit unauthorized or unlawful surveillance. The contested territory is lawful surveillance that a provider still considers unacceptable.

That is a genuine governance question, not a software setting. It belongs in contracts, statutes, constitutional limits, procurement rules, and judicial review. The more powerful the model becomes at connecting weak signals into strong inferences, the more important it is to decide who has authority to draw that line before deployment.

A Right to Exit the Profile

Privacy law usually gives people rights one company at a time. Data brokerage makes that structure exhausting. A consumer can be expected to identify and contact organizations that the consumer may not know exist.

California’s Delete Request and Opt-Out Platform, or DROP, tries a different architecture. Beginning in 2026, a California resident can submit one deletion request through a state system that applies to registered data brokers. Starting August 1, brokers must regularly access the system and process those requests, subject to legal exceptions. California’s regulations require deletion of associated personal information, including inferences, when the request applies.

That last detail matters. A meaningful deletion right cannot stop at the raw record if the profile built from the record survives.

DROP is still young and its effectiveness will depend on compliance, identity matching, enforcement, and the scope of exemptions. But it reverses an important asymmetry. Instead of requiring the individual to find hundreds of brokers, the law creates one mechanism that reaches outward to them.

Privacy becomes more usable when the burden follows the architecture that created the problem.

Current Evidence: From Knowing to Steering

Reviewed September 11, 2026.

The persuasion evidence is becoming harder to dismiss as merely speculative. The Nature Human Behaviour experiment found that the personalized GPT-4 condition significantly outperformed the human baseline in the study’s strongest comparison. A September 2026 author correction clarified that the direct comparison with non-personalized GPT-4 was not statistically significant, so the study cannot isolate personalization as the sole cause of the advantage. It still shows that a conversational AI supplied with personal information can be highly persuasive under controlled conditions.

The FTC’s personalized-pricing work moves prediction directly into the consumer transaction. The Commission’s January 2025 study found that intermediaries can use precise location, demographics, browser history, shopping behavior, and even mouse movements to help tailor prices or promotions. Its August 19, 2026 proposed enforcement statement warned businesses about undisclosed personalized pricing based on personal data while acknowledging that the FTC lacks authority to ban personalized pricing in every circumstance. The original September 18 comment deadline was later extended seven days to September 25. Congressional investigators and state attorneys general were examining related practices at the same time.

Courts and regulators are examining architecture rather than individual messages. The European Commission’s 2026 preliminary findings against TikTok, Instagram, and Facebook focus on infinite scroll, autoplay, push notifications, and personalized recommendation systems. TikTok denied the accusations, and TikTok and Meta retain rights of defense before final Commission decisions. In the United States, a March 2026 Los Angeles jury found Meta and Google negligent in the design of Instagram and YouTube and awarded $6 million in a bellwether addiction case; both companies said they would appeal. Across both systems, the enforcement target is increasingly the architecture of attention itself.

California’s DROP system moved from statute to operation in 2026. Consumers can make one deletion request to registered data brokers, and brokers began processing those requests in August. The regulations expressly reach inferences associated with the consumer. That is a practical response to a market in which prediction is built from data held outside the direct consumer relationship.

The FTC also finalized 2026 orders against Cox Media Group and two marketing firms over claims about AI-powered “Active Listening.” The case is useful precisely because the agency alleged the service used no voice data at all. It instead relied on other marketing inputs, including brokered email lists, while its location targeting did not work as advertised. The FTC also said a terms-of-service click would not constitute opt-in consent for the invasive listening service that had been claimed. The case separates two fears that are often collapsed: literal eavesdropping and behavioral inference. The second can be powerful even when the first never happened.

The national-security dispute shows the same boundary at higher stakes. Anthropic sought to retain a ban on mass domestic surveillance; the Pentagon argued for lawful-use flexibility; and a federal court rejected the supply-chain-risk designation. NSPM-11 later prohibited unauthorized or unlawful surveillance, expressly kept commanders, directors, and agency heads accountable at every level of command, required government control over mission systems, and created a contract-termination mechanism for companies that repeatedly act inconsistently with its policies. The unresolved question is which lawful uses a private provider may still refuse.

Taken together, the evidence shows a common progression. Data becomes prediction. Prediction becomes selection. Selection changes the environment. The intervention may be a price, a recommendation, a message, an investigation, or a surveillance query. The person experiences the outcome while much of the decision process remains elsewhere.

Accountability Analysis

Principles 3, 5, and 6 carry most of this chapter. Principle 3 requires a boundary around which data may be used, which inferences may be drawn, which interventions may follow, and which uses are prohibited. A privacy notice that describes collection without defining downstream use leaves the most important delegation open.

Principle 5 explains why responsibility moves upstream. A recommender system can shape millions of choices without any employee selecting each item. Personalized pricing can change offers automatically. A surveillance model can rank people or events before a human investigator sees the result. Accountability therefore reaches the people who choose objectives, data sources, optimization targets, thresholds, defaults, interfaces, and stopping rules.

Principle 6 asks where the consequences land. If a company gains from more engagement while the user bears the cost of compulsive use, the architecture separates benefit from consequence. If a retailer gains from estimating willingness to pay while the consumer cannot see that a personal profile changed the price, the same split appears. If a government gains surveillance capability while constitutional and statutory risk lands on the subject, accountability has to reach the officials who authorized the system.

Principle 1 frames the chain. A profile does not become responsible because it predicted correctly. The institution that used the prediction remains answerable for what it chose to do with it.

Follow one recommendation

Choose a recommendation or advertisement you recently saw. What action did it encourage? Look for an explanation of why it appeared and any controls over personalization. Separate what you can establish from what you can only guess. Then ask whose goal the recommendation serves and whether you can change or challenge how the service treats you.

Accountability Verdict

AI should be allowed to personalize when personalization serves the person and the person can understand and control the relationship. A tutor that adapts explanations, a medical system that highlights relevant history, or a service that remembers a chosen preference can create real value.

The boundary tightens when personalization is used to exploit hidden vulnerability, vary treatment without disclosure, shape attention toward someone else’s objective, or convert sensitive inference into surveillance.

The practical rule is to follow the prediction forward. What action did it trigger? Who selected the objective? What data made the prediction possible? Could the person see or challenge the intervention? Who benefits if the prediction is acted on? Who bears the cost if it is wrong or manipulative?

The algorithm does not need to know you completely. It only needs to know what will move you.

Chapter Fifteen moves from prediction to creation. When machines can produce writing, music, images, film, and design at enormous scale, the next question is not only what the machine can make. It is who owns the work, whose labor trained the system, and what creativity means when generation becomes cheap.

Sources and Notes

1. Francesco Salvi, Manoel Horta Ribeiro, Riccardo Gallotti, et al., “On the conversational persuasiveness of GPT-4,” Nature Human Behaviour 9, 1645-1653 (2025), published May 19, 2025; author correction published September 3, 2026. https://www.nature.com/articles/s41562-025-02194-6 and https://www.nature.com/articles/s41562-026-02588-0. Accessed September 11, 2026. In a preregistered study with 900 participants debating sociopolitical topics, the GPT-4 condition with access to basic sociodemographic information significantly outperformed the human baseline, with an estimated 81.2% increase in the odds of greater post-debate agreement and a 64.4% probability of being more persuasive when the two sides were not equally persuasive. The September 2026 correction states that the direct comparison between personalized and non-personalized GPT-4 did not reach conventional statistical significance (P = 0.07), so the study does not establish that personalization itself caused the full advantage. The study used GPT-4. OpenAI systems are also used in this Living Edition’s production workflow; see the transparency statement. Source record: DA-CH14-S1. Archive reference: pending.

2. Federal Trade Commission, “FTC Finalizes Order Banning Mobilewalla from Selling Sensitive Location Data,” January 14, 2025; and underlying complaint and order. https://www.ftc.gov/news-events/news/press-releases/2025/01/ftc-finalizes-order-banning-mobilewalla-selling-sensitive-location-data. Accessed September 11, 2026. The FTC alleged that Mobilewalla collected and sold sensitive location data and used location information to create audience segments that could reveal health, religious, pregnancy-related, and political characteristics. The final order bans sale of sensitive location data and restricts collection of advertising-auction data for unrelated purposes. The case is an administrative settlement and does not constitute an admission of every allegation. Source record: DA-CH14-S2. Archive reference: pending.

3. Federal Trade Commission, “FTC Surveillance Pricing Study Indicates Wide Range of Personal Data Used to Set Individualized Consumer Prices,” January 17, 2025; “FTC Seeks Comment on Enforcement Policy Statement Regarding Personalized Pricing,” August 19, 2026, press release corrected August 31, 2026; and “FTC Extends Public Comment on Proposed Policy Statement Regarding Personalized Pricing,” September 3, 2026. https://www.ftc.gov/news-events/news/press-releases/2025/01/ftc-surveillance-pricing-study-indicates-wide-range-personal-data-used-set-individualized-consumer, https://www.ftc.gov/news-events/news/press-releases/2026/08/ftc-seeks-comment-enforcement-policy-statement-regarding-personalized-pricing, and https://www.ftc.gov/news-events/news/press-releases/2026/09/ftc-extends-public-comment-proposed-policy-statement-regarding-personalized-pricing. Accessed September 11, 2026. FTC staff reported that intermediaries can use location, demographics, browsing, shopping behavior, and other signals to tailor prices or promotions. The proposed policy statement says consumers ordinarily expect a listed price to be the same price others see at the same place and time and acknowledges that the FTC does not have authority to prohibit personalized pricing in all circumstances. The original September 18 comment deadline was extended seven days to September 25. The House Committee on Oversight and Government Reform opened an investigation into AI-driven surveillance pricing in March 2026, and a coalition of 16 state attorneys general urged the FTC in May to address personalized pricing in online food delivery. The proposed statement is not a final binding rule. Source record: DA-CH14-S3. Archive reference: pending.

4. European Commission, “Commission preliminarily finds TikTok’s addictive design in breach of the Digital Services Act,” February 6, 2026; “Commission preliminarily finds the addictive design of Instagram and Facebook in breach of the Digital Services Act,” July 10, 2026; and Reuters, “Meta, Google lose US case over social media harm to kids,” March 25, 2026. https://digital-strategy.ec.europa.eu/en/news/commission-preliminarily-finds-tiktoks-addictive-design-breach-digital-services-act, https://digital-strategy.ec.europa.eu/en/news/commission-preliminarily-finds-addictive-design-instagram-and-facebook-breach-digital-services-act, and https://www.reuters.com/legal/litigation/jury-reaches-verdict-meta-google-trial-social-media-addiction-2026-03-25/. Accessed September 11, 2026. The Commission’s preliminary findings focus on infinite scroll, autoplay, push notifications, and highly personalized recommender systems. TikTok denied the accusations, and both TikTok and Meta retain rights of defense before final Commission decisions. Separately, a Los Angeles jury found Meta and Google negligent in the design of Instagram and YouTube and awarded $6 million in damages in a bellwether social-media-addiction case. Meta and Google said they disagreed with the verdict and planned to appeal. Source record: DA-CH14-S4. Archive reference: pending.

5. California Privacy Protection Agency, “Information for Data Brokers” and “Accessible Deletion Mechanism - Delete Request and Opt-Out Platform (DROP) System Requirements,” current 2026 editions. https://cppa.ca.gov/data_brokers/ and https://cppa.ca.gov/regulations/drop.html. Accessed September 11, 2026. California’s Delete Act created a single state-hosted mechanism through which consumers can request deletion from registered data brokers. Consumers could submit requests beginning in 2026; data brokers were required to begin processing DROP requests on August 1, 2026 and to access the system at least every 45 days. Agency materials state that applicable deletion reaches associated personal information, including inferences, subject to legal exceptions. Source record: DA-CH14-S5. Archive reference: pending.

6. Reuters, “Pentagon designates Anthropic a supply chain risk,” March 5, 2026; Reuters, “US judge rules Pentagon blacklisting of Anthropic unlawful,” August 28, 2026; and The White House, National Security Presidential Memorandum 11, “Artificial Intelligence in the National Security Enterprise,” June 5, 2026. https://www.reuters.com/technology/pentagon-informed-anthropic-it-is-supply-chain-risk-official-says-2026-03-05/, https://www.reuters.com/world/us-judge-rules-pentagon-blacklisting-anthropic-unlawful-2026-08-28/, and https://www.whitehouse.gov/presidential-actions/2026/06/national-security-presidential-memorandum-nspm-11/. Accessed September 11, 2026. Reuters reported that Anthropic refused to remove restrictions on use of Claude for autonomous weapons and mass U.S. surveillance, while the Pentagon argued for lawful-use flexibility. A federal judge later ruled the supply-chain-risk designation unlawful. NSPM-11 accelerates national-security AI adoption, bars unauthorized or unlawful surveillance, preserves privacy and civil-liberties obligations, states that commanders, directors, and agency heads remain responsible and accountable at every level of command, requires controls preventing commercial entities from disabling or materially modifying mission systems without federal approval, and directs agencies to terminate contracts, to the maximum extent permitted by law, with companies that repeatedly act inconsistently with its policies. Anthropic systems are also used in this Living Edition’s external critique workflow; see the transparency statement. Source record: DA-CH14-S6. Archive reference: pending.

7. Federal Trade Commission, “FTC Finalizes Orders with Cox Media Group, Two Other Firms Settling Charges They Deceived Customers About ‘Active Listening’ AI-Powered Marketing Service,” August 27, 2026. https://www.ftc.gov/news-events/news/press-releases/2026/08/ftc-finalizes-orders-cox-media-group-two-other-firms-settling-charges-they-deceived-customers-about. Accessed September 11, 2026. The FTC alleged that Cox Media Group, MindSift, and 1010 Digital Works falsely claimed an AI-powered advertising service could target localized ads based on conversations captured from smart devices and falsely suggested consumers had opted in. The FTC said the service was not based on voice data, relied on other marketing inputs including brokered email lists, and did not deliver the geographic targeting claimed. It also stated that a terms-of-service click would not constitute adequate opt-in consent for the invasive voice-data collection the firms had claimed. The final orders settled deception allegations and do not establish that secret listening actually occurred. Source record: DA-CH14-S7. Archive reference: pending.

8. Federal Trade Commission, “FTC Finalizes Order Prohibiting Gravy Analytics, Venntel from Selling Sensitive Location Data,” January 14, 2025; and FTC v. Kochava Inc., stipulated order entered June 26, 2026. https://search.ftc.gov/news-events/news/press-releases/2025/01/ftc-finalizes-order-prohibiting-gravy-analytics-venntel-selling-sensitive-location-data and https://www.ftc.gov/legal-library/browse/cases-proceedings/ftc-v-kochava-inc. Accessed September 11, 2026. The FTC’s location-data cases allege that data brokers sold or disclosed sensitive location information that could reveal visits to health facilities, places of worship, political or other sensitive locations, and in the Kochava matter could trace movements tied to hundreds of millions of mobile devices. These matters support the chapter’s broader claim that inference and intervention can occur through data held outside the direct consumer relationship. Source record: DA-CH14-S8. Archive reference: pending.