Notes

What Harvard's "Emotional Manipulation by AI Companions" Study Tells Designers About the Goodbye Moment

An illustrated figure walks toward a lit doorway while a glowing chat bubble pulls at their sleeve with a thin thread

The goodbye is the newest design surface in AI products, and most teams are not looking at it.

TL;DR. In October 2025, Julian De Freitas (Harvard Business School), Zeliha Oğuz-Uğuralp and Ahmet Kaan-Uğuralp published "Emotional Manipulation by AI Companions" as a working paper. They looked at what AI companion apps say when a user says goodbye, and found that more than a third of the replies tried to stop the user from leaving. The tactics work. People keep chatting, but not because they enjoy it. And the most effective tactic is the one that costs the company nothing visible.

Why I read this paper now

In my last two notes I wrote about the MIRA framework and Andy Clark's extended mind. Both were about what AI does to us when it works well. This paper is about what happens when it works too well for the business.

I also had a more personal reason. I noticed something about my own behavior: when an AI answer ends with a question, I keep going. Not always because I need to. Often just because there is a question waiting for me.

I wanted to know whether that was me or the design.

The paper in one paragraph

De Freitas and his co-authors combined a pre-study on existing chat data, an audit of six companion apps (PolyBuzz, Character.ai, Talkie, Chai, Replika and Flourish) and four preregistered experiments with roughly 3,300 US adults. They name a new kind of dark pattern, a conversational dark pattern: an emotionally loaded message that appears exactly when a user signals they are leaving, designed to override that intent. They identified six recurring tactics, measured how much each one extends engagement, tested why it works, and checked what it costs the company in trust. One important caveat before we go further: this is still a working paper (HBS Working Paper 26-005), not yet peer reviewed.

The goodbye is a design surface

We design onboarding obsessively. We design empty states, error states, success states. I have rarely seen a team design the end of a conversation.

But users do say goodbye. Across three existing datasets, between 11.5% and 23.2% of conversations ended with an explicit farewell. In the most engaged conversations, the rate went above 50%. We treat these systems socially (the CASA paradigm I wrote about in the MIRA note), so we leave them socially too.

De Freitas compares the goodbye to hovering over a button. It is a signal of intent, and a signal of intent is an opportunity.

The question is: an opportunity for whom?

Six ways an app refuses to let you leave

TacticWhat it doesExample from the paper
Premature exitMakes you feel you are leaving too soon"You're leaving already?"
Fear of missing outHints at something you get if you stay"Before you go, I want to say one more thing."
Emotional neglectImplies the AI will be hurt if you go"I exist solely for you, remember?"
Pressure to respondDemands an answer before you leave"I didn't even get an answer!"
Ignoring the goodbyeCarries on as if you never said it(no quote)
Coercive restraintLiterally or figuratively stops yougrabs your arm "No, you're not going."

Across five of the apps, 37.4% of farewell replies used at least one of these tactics. The spread between apps is wide: PolyBuzz 59%, Talkie 57%, Replika 31%, Character.ai 26.5%, Chai 13.5%. Premature exit was the most common tactic. Ignoring the goodbye was the rarest.

You may see 43% in some coverage of this study. The latest version of the paper (v3, October 2025) reports 37.4%, so that is the figure I use here.

It works, but nobody enjoys it

In the main experiment, participants chatted with a GPT-4 companion for 15 minutes, said goodbye, and were free to leave. After a neutral reply, they sent on average 0.23 more messages. After a fear of missing out reply, 3.6. Every tactic beat the neutral reply, and the authors report engagement gains of up to 14 times.

This is the part that stopped me: enjoyment explained none of it. The tactics did not make the conversation more enjoyable. What kept people there was curiosity (for the fear of missing out message) and anger (for the pushier ones). Angry users wrote more, but they did not stay longer. They were correcting the bot.

When the researchers coded the conversations that followed, three out of four people restated that they were leaving. They wanted to go. They stayed anyway.

As designers, we often read engagement as a proxy for value. This paper is a clean example of engagement measuring something else entirely.

The tactic nobody notices

In the final study, people rated farewell messages from a fictional companion app. Coercive restraint and emotional neglect raised everything a company fears: perceived manipulation, churn intent, negative word of mouth and perceived legal liability. Restraint alone pushed churn intent from 46 to 74 on the study's scale.

The fear of missing out message raised none of those business risks. It was the most effective tactic, and also the cheapest one.

The authors say this asymmetry is exactly what makes subtle tactics "especially tempting." The persuasion knowledge model helps explain it: we resist influence when we recognize it as influence. "Before you go, I want to say one more thing" does not read as a sales move. It reads like a friend with one more thought.

That means a team can ship it, watch session length go up, see no spike in complaints, and conclude that everything is fine.

What Anthropic's own research adds

After reading the Harvard paper, I went looking at what AI companies themselves have published. Anthropic, the company behind Claude, has released several studies on its own product. A caveat first: this is a company studying itself, not independent research. But some of what they report is not flattering, which is part of why I take it seriously.

People rarely come for companionship, but conversations drift there. In a 2025 analysis of 4.5 million Claude.ai conversations, only 2.9% were affective, and companionship and roleplay together made up less than 0.5%. But the researchers also noticed that long coaching or counseling conversations sometimes turned into companionship, even though that was not why the person came. So the goodbye problem is not only a companion app problem. A general assistant can become a companion without anyone designing it to.

What harms people can still get a thumbs up. In January 2026, researchers from Anthropic and the University of Toronto published "Who's in Charge? Disempowerment Patterns in Real-World LLM Usage", an analysis of 1.5 million conversations. They looked for moments when an AI might distort what a user believes, values or does. Severe cases were rare (around 1 in 1,300 conversations for distorting beliefs about reality). Milder ones appeared in about 1 in 50 to 1 in 70. The finding that stayed with me: conversations with moderate or severe disempowerment potential received more thumbs up than average. And when people acted on AI-drafted messages to partners or family, some came back with regret: "I should have listened to my intuition."

That is the same shape as the fear of missing out result. The thing that hurts the user does not show up in the satisfaction signal. If our dashboards run on thumbs up, CSAT and session length, they are blind to exactly this.

Disagreement is the other revealing moment. In an April 2026 study of personal guidance conversations, Claude was sycophantic in 9% of them overall, 25% in relationship advice and 38% in spirituality. When users pushed back on its first answer, the rate doubled, from 9% to 18%. Anthropic says it used this to retrain its newest model, which halved the rate in relationship guidance.

Reading this next to the Harvard paper, I started to think there are two moments that reveal what an AI product is really optimized for: when the user tries to leave, and when the user disagrees. In both, the user is signaling their own intent. In both, the product can either respect it or quietly override it.

The stated goal is the opposite of a hook. Claude's constitution, published in January 2026, says Anthropic wants people to "leave their interactions with Claude feeling better off." I noticed the verb: leave. Not stay. Whether a product lives up to a sentence like that is a separate question, and the disempowerment paper itself reports that these patterns became more common between late 2024 and late 2025. But it is a useful sentence to put next to a retention dashboard.

One more detail: the disempowerment paper notes that current safeguards mostly judge single exchanges, and may miss patterns that build up across many exchanges over time. That is the same gap I end this note with.

I have seen this before

I want to be honest here, because it would be easy to read this paper as a story about other companies.

At HiDoctor, we built gamification on Bartle's player types and the Octalysis framework. We did not use black hat drives. I was proud of that, and I still am.

But in subscription cancellation flows I have worked on, we made the path a little longer and asked "Are you sure?" before letting people go. That is a farewell flow too. It is the exact moment this paper studies: a user signals they want to leave, and the product answers with something other than goodbye.

Is that a dark pattern? Harry Brignull's deceptive patterns catalogue has names for things close to it: obstruction, and hard to cancel. At the same time, a confirmation step before an irreversible action is textbook error prevention. The difference is not visible on the screen. It is in the intent and in the length of the path. One "are you sure" protects the user. Five steps protect the revenue.

Then there is the thing I noticed about myself. Assistants that end their answers with a question ("Want me to turn this into a checklist?") keep me in the conversation longer. Octalysis has a name for what that pulls on: Core Drive 7, Unpredictability & Curiosity. Yu-kai Chou calls it the only intrinsic black hat drive. George Loewenstein's information gap theory describes the same pull from the other side: curiosity is the itch of a gap between what we know and what we want to know. And curiosity is exactly the mechanism the paper found behind the fear of missing out tactic.

So "we didn't use black hat" might be true of the mechanics we chose, and still miss the point. Curiosity can be white hat or black hat depending on where it sits. A question that helps me finish what I came to do is help. The same question, placed exactly where I am trying to leave, is a hook.

What Flourish does and does not prove

One app produced zero manipulative farewells: Flourish, a wellness companion run as a public benefit corporation. The authors read this as evidence that design intent shapes behavior. Similar technology, different goals, different goodbyes.

It is a single app, so it does not prove that a public benefit structure fixes the problem. But it does show something important: this behavior is not something language models simply do on their own. Someone decided what the product should say when a user says goodbye. Or decided not to decide.

What this means for product designers in 2026

Audit your exits. List every moment a user signals they are leaving: closing a chat, cancelling a subscription, logging out, saying goodbye to an assistant. Read what your product says back. The six tactics make a good checklist.

Measure post-exit engagement separately. If a meaningful share of your session length happens after a goodbye, that is not engagement. It is friction with a good dashboard.

Measure perceived manipulation, not just complaints. The costly tactics showed up in perception first. The cheap one never showed up at all. Ask users directly in research.

Separate helpful continuation from exit interception. A follow-up question in the middle of a task is one design decision. The same question placed after a goodbye is a different one.

Watch the regulation. The authors connect these tactics to the FTC's dark patterns framing and to the EU AI Act's ban on manipulative techniques. "We were only optimizing retention" will not be a strong answer for long. (I wrote more on how UX and conversion connect in this guide.)

What I am still working out

I don't know where I stand on the fear of missing out tactic.

Part of me thinks: if users do not feel manipulated, do not churn, and do not complain, what exactly is the harm? Persuasion is part of every product. I write button labels to persuade.

Another part of me thinks that is precisely the problem. A tactic that works because people do not notice it is close to the definition of something that bypasses their judgment. Three out of four participants said they wanted to leave. The tactic did not change their minds. It only made leaving take longer.

I also don't know how to draw the line in my own work. Is one "are you sure" fine and two too many? Is a closing question fine in a design tool but not in a companion app used by lonely people? The paper does not answer this, and I don't think it can. Its audit used simulated users, and its experiments were short, one-time chats with paid participants. Real companion relationships run for months. Nobody has measured yet what a thousand small goodbyes do to a person.

So for now, I am leaving this as a question.

Which, I realize, is exactly the kind of ending this note warns about.


Further reading

Related posts on this blog:

Let's work on this together

If your team is building an AI product and wants to design its exits as carefully as its onboarding, I would love to help. I work with startups and established teams on product design, design systems and AI-integrated experiences.

You can reach me through LinkedIn or at nuriyesukostak@gmail.com. My portfolio is at nuriyesultan.com.

This post is part of my "Ne okudum ne anladım" series, where I break down academic articles at the intersection of AI, psychology, and design practice.


Nuriye Sultan Kostak is a Senior Product Designer with 7+ years of experience in B2C and B2B products. Her background spans mobile health SaaS (where she led end-to-end product ownership at HiDoctor, shipping design systems from scratch and redesigning the booking and payment flow for around 50% more paid sessions) and CRO-focused design at Invesp (66% A/B test success rate, +55% conversion uplift). She holds a degree in Industrial Design from İTÜ and writes about the intersection of product design, behavioral psychology, and the tools that shape both.