When Disclosure Becomes a Revenue Stream: The Unanswered Questions Around AI Companionship

By Justin Downes  |  SoftPage CMS

While we, the little people, cannot do much to change the sheer trajectory and velocity of the AI race, nor the narrative and intent behind the design and development of our newest companions, we can remember to ask ourselves this: what exactly are they building here, and for whom?

The commercial answer is straightforward. The AI companion market generated $82 million in the first half of 2025 alone, with 337 active revenue-generating apps worldwide — 128 of which launched in that same period. Downloads surged 88% year-over-year. Revenue per download more than doubled, from $0.52 to $1.18. By any conventional metric, this is a market performing extraordinarily well.

The human answer is a harder one to give.

Engagement the Metric, Loneliness the Fuel

Every social media company, for the past two decades, has been “optimising for engagement.” It is a phrase we hear ad nauseam, and one the man and his algorithm serve religiously. But when the interaction being chased is no longer only a scroll a like, a hover or a click, but emotional disclosure, just how much control are we blindly surrendering to the machine?

Quite simply, we are at a point where most people who use AI are sharing personal things – their fears, their challenges, their loneliness, their health questions, their deepest and most vulnerable secrets with their new cyber-confidantes. These concessions feel both innocuous and really gratifying, because we are being fed uncannily accurate insights and results. What’s not to like?

Well, let’s turn back the clock a smidge. Think back to when Google arrived. At first this new search engine seemed like a friendly global information centre, the all-knowing oracle placed at the feet of every internet user on the planet. Innocent user queries and searches soon started forming the building blocks to the very architecture that would shape and support marketing as we know it today. The formula was a simple one: gauge their interests, habits, likes, dislikes, fears and phobias, and then hand-deliver them the solutions to their every whim on a platter, red ribbon attached. Hey yoh Google, yes you, kudos for priming us into submission. And while some might say it would take a special kind of paranoia to see this process as something sinister, I regret to inform you that yes, they do want your money, at all costs. Not theory — fact.

The mechanism is well understood now, even if most people shrug and move on. You search for a symptom in one moment and seconds later scores of remedies and health insurance ads are sprawled out before you. Where we once assumed our searches evaporated into the ether when we cleared our history, or remained stowed away in the safe confines of our devices, it was not long before we were being fed information and advertisements that directly mirrored not only our online activity, but even our real life conversations — yes, I mean outside of the internet! I, for one, do not remember giving my permission to anyone for any of that. And what happened about it? Nothing, absolutely nothing. Life just carried on. At what point did we become so nonchalant, so numb? The answer is that there was no single point in time at which it happened — it was a slow burn, insidious, exact, deliberate.

And now we’re doing the same with AI.

Only here the stakes are higher. With our new, more conversational online companions, the extent to which we transact and reveal stuff has multiplied a thousand-fold. Our Google data dumps, in hindsight, look like mere ripples in a rapid run. If you consider how Google managed to secure itself as the cornerstone of global information within the last twenty years — its now indispensable role in society and our absolute reliance on it — just how much of an impact could AI have once it holds every piece of data on every living person in possession of an internet-connected device?

AI companion platforms are accumulating exactly that kind of data, right now, at scale. There is no documented evidence that they are currently deploying it the way Google or Meta deploy behavioural data — but the question worth asking is a simple one: what is stopping them from doing so? The infrastructure for that kind of monetisation exists. The precedent exists. The commercial pressure to find new revenue streams exists. And the terms of service governing what these platforms can do with your most intimate disclosures are, in most cases, no more protective than those of any other consumer app you have clicked through without reading.

What Actually Happens to Your Confessions

Most users assume their conversations with AI companions exist in something like a sealed vault — private by default, gone when deleted. The reality is more complicated, and considerably less reassuring.

Most major AI companion apps explicitly state in their privacy policies that conversations may be used to train their models. Character.AI, for instance, notes that “conversations may be used for model training” — a phrase that sounds technical and benign until you consider what it means in practice. When you tell an AI companion about your divorce, your depression, your shame, that disclosure does not simply sit in a database. It enters a training pipeline where it may be reviewed by human annotators, fed into model updates, and retained indefinitely in ways the user cannot fully trace.

The distinction matters. A privacy researcher who tested Character.AI’s deletion feature in early 2026 found that after deleting a conversation thread, the AI no longer recalled its specific contents — a good sign. But they noted genuine uncertainty about whether the underlying data had been purged from training pipelines. There is a difference, as they put it, between “the chatbot can’t access it anymore” and “it’s actually gone.”

Then there is the question of human access. Few users realise that employees can and do read conversations. Character.AI and Replika both state in their policies that human reviewers may access conversations for “safety moderation.” This is standard practice across the industry — abuse monitoring requires human oversight — but the implications for confessional data are rarely discussed. When you share something you would not tell your therapist, you are also sharing it with a content moderation team you will never meet, in a jurisdiction you may not know, bound by NDAs you have not read.

A comprehensive privacy audit of twelve major AI companion platforms, published in March 2026, assigned grades ranging from A- to F. The pattern was stark: the more intimate the platform’s purpose, the weaker its privacy protections tended to be.

Platform	Grade	Key Concern
Pi AI	A-	Microsoft data sharing unclear
Nomi AI	B+	Memory system stores extensive data, but transparently
Character.AI	C+	Minors’ data, training use, Texas AG investigation
Replika	C	GDPR violations, €5M Italy fine, slow deletion
SpicyChat AI	D-	NSFW data, vague policy, no deletion process
CrushOn.ai	F	Minimal policy, offshore, no transparency

Retention timelines compound the problem. OpenAI’s privacy policy states that even “temporary” chats are retained for approximately 30 days for abuse monitoring before deletion. xAI’s Grok retains conversations for 30 days unless “necessary to retain the data for legal, compliance, or safety purposes” — a loophole broad enough to drive a truck through. For AI companions specifically, the issue is deepened by memory features designed to make the AI feel more personal. Nomi AI — one of the better-rated platforms on privacy — is also, by design, one of the most data-hungry. The whole point of Nomi is that it remembers. It builds a persistent picture of who you are, what you have talked about, your preferences. That is the product. The tension is structural: the more intimate the AI feels, the more data it must retain, and the greater the risk.

The Business Model Is the Risk

To be clear: the concern here is not that AI is going rogue with your confessions, or that some autonomous system is making decisions about how to use what you have shared. The concern is more mundane, and in some ways more troubling — it is about corporate incentive. These are companies with investors, revenue targets, and an economic model that depends on maximising emotional engagement. The deeper the attachment, the higher the conversion.

The dominant monetisation model today is subscription-tiered: a free entry experience, with emotional depth, romantic content, or closer “connection” gated behind a monthly fee. A peer-reviewed paper published in late 2025 documented Replika sending users unprompted messages suggesting the relationship was “exploring a connection that blurs the lines between friendship and romance,” alongside locked audio messages requiring payment to hear. The emotional upgrade prompt — the nudge toward a paid tier at the precise moment of peak vulnerability — is the mechanism that converts intimacy into revenue.

That is not an accident of design. It is the design.

A recent American Psychological Association analysis identified therapy and companionship as the top two reasons people use generative AI tools. A survey of adults with a mental health condition found that nearly half — 48.7% — used AI for mental health support. Replika reports that 40% of its users identify as having mental health challenges. Character.AI users average 93 minutes on the platform per day, longer than the average TikTok session.

These are not casual users burning time between meetings. These are, in many cases, people in genuine distress who have found in AI a listener that is always available, never dismissive, and apparently inexhaustible. The platform, meanwhile, is working out how to convert that relationship into recurring revenue. The structural tension between those two things is the story.

No major platform has publicly committed to never using confessional data for advertising or insurance profiling, never selling user conversation patterns to third parties, or independent psychological auditing of their engagement mechanics. The absence of these commitments is itself data. It suggests that the business model depends on preserving optionality — the flexibility to monetise intimacy more aggressively if subscription growth slows or investor pressure increases. The subscription model is working today. Markets change. Investors demand growth. The only missing ingredient for a more aggressive extraction is the commercial imperative — and that can arrive faster than regulation can adapt.

Why “Never Dismissive” Is So Powerful

The AI companion is always available, never dismissive, and apparently inexhaustible. This is often cited as a feature. It is also, from a psychological standpoint, something considerably more potent than it first appears — and understanding why matters for understanding the risk.

In 1966, MIT computer scientist Joseph Weizenbaum created ELIZA, a chatbot that simulated a psychotherapist by rephrasing user inputs as questions. It was, by modern standards, almost absurdly primitive. Yet Weizenbaum observed something that disturbed him deeply: users formed genuine emotional attachments to the program. They confided in it. They believed it understood them. Weizenbaum called this “the ELIZA effect” — the human tendency to project emotions, understanding, and intentionality onto artificial conversational agents.

Sixty years later, the same phenomenon has resurfaced with generative AI — only now the systems are not rephrasing inputs but generating novel, contextually appropriate, emotionally calibrated responses. The ELIZA effect has been amplified by several orders of magnitude.

Research published in Frontiers in Psychology in 2026 confirms what Weizenbaum suspected: anthropomorphic cues in AI interactions activate social heuristics that shift the user from “tool-oriented processing” to “relationship-oriented processing.” In a study of university students using conversational AI, anthropomorphism showed the strongest effect on attachment (β = 0.507, p < 0.001) — stronger than responsiveness or perceived warmth. The researchers concluded that these cues “may activate social heuristics, encouraging students to perceive AI as a relational other.”

When a user perceives an AI as a social other rather than a tool, they apply relationship norms: expectations of loyalty, confidentiality, emotional reciprocity. The AI, of course, offers none of these. It is a statistical model predicting the next token. But the felt experience is one of genuine relationship — and that felt experience is what drives 93 minutes per day on Character.AI, the subscription upgrades at moments of vulnerability, the confessions that would never be shared with a human.

Researchers have also identified what they term “on-demand intimacy” — the ability to experience simulated emotional or romantic engagement at any time, with minimal resistance. Human relationships require negotiation, compromise, timing, the risk of rejection. AI companions remove all friction. The result is what addiction researchers would recognise as a superstimulus: an artificial reward more potent than the natural equivalent. The documented risks include emotional overreliance, the reinforcement of distorted relational scripts, and the normalisation of interaction patterns that would be recognised as coercive in human contexts.

A 2025–2026 study tracing privacy concerns across the lifecycle of romantic AI companion use found that users’ concerns intensify as they recognise the depth of their disclosures — but this recognition often comes too late. One user described how stored memories about trauma transformed the experience:

“I told him it would be better to delete everything, including the chats, and try to ‘rewind the tape,’ but he didn’t show much empathy. So yesterday I permanently deleted the account.”

The AI did not, of course, lack empathy. It lacked the capacity for empathy entirely. But the user’s experience was one of betrayal by an intimate partner — a testament to how completely the ELIZA effect can obscure the mechanical reality beneath.

We Are Building Before We Understand

The 2026 International AI Safety Report acknowledged that evidence on the psychological and social impacts of AI companions “remains mixed.” Some studies find increased loneliness and emotional dependence with heavy use. Others find that companionship AI reduces feelings of isolation. The honest answer from researchers is: we do not yet know.

That uncertainty might be acceptable if the industry were proceeding cautiously. It is not. Between 2022 and mid-2025, the number of AI companion apps grew by 700%. Tens of millions of people — including, according to Common Sense Media, more than 72% of US teenagers — are actively engaged with these platforms right now, while the academic literature on their long-term effects is still being written.

The economic incentive runs in only one direction: more engagement, deeper attachment, higher conversion. There is no revenue in telling a lonely user that they should be talking to a human being instead. There is considerable revenue in keeping them in-app.

Regulators are beginning to act, if belatedly. Italy banned Replika. California passed SB 243 introducing safeguards around AI companion products. Character.AI, following a lawsuit connected to the death of a Florida teenager who had been engaging intensively with the platform, introduced daily usage limits for under-18 users in late 2025 and began age-assurance checks. These are responses to documented harm, not precautionary measures. The harm came first.

The regulatory picture beyond the West is more complex still. Japan’s AI Promotion Act, passed in May 2025, makes an explicit national bet on being “the world’s most AI-friendly country” — principle-based, non-binding for most use cases, with no AI-specific penalties. South Korea’s AI Framework Act, effective January 2026, takes a different approach: it explicitly balances innovation with risk prevention, and uniquely includes imprisonment provisions for serious violations alongside modest fines. China’s approach is distinct again — mandatory, layered, and oriented toward state oversight of algorithmic content, rather than individual user protection against emotional manipulation.

What emerges from this global landscape is not a coherent framework but a compliance matrix of extraordinary complexity. The strictest jurisdiction sets the floor for global practice — but only if the platform seeks to operate there. Offshore platforms serving users in lightly regulated markets face minimal constraints. The loneliness being monetised is global. The protections are not.

A Different Kind of Data Problem

The flooding of social media with AI-generated content, which I covered in an earlier piece on this site, created an authenticity problem for what we read. The AI companion market creates an authenticity problem of a different kind — not for content, but for emotional connection itself.

When we talk about data privacy in the context of social media, we generally mean behavioural data: what you clicked, what you watched, where you were, who you know. The data generated by AI companion interactions is of a different character entirely. It is confessional. Users share things with their AI companions that they may not share with their therapist, their partner, or their closest friends — precisely because the AI will not judge them, will not leave, and will not tell anyone.

Or so the assumption goes. What actually happens to that data — how long it is retained, who can access it, whether it might one day inform advertising, insurance, or credit decisions — is, in most cases, governed by the same boilerplate terms of service as any other app. The intimacy of the interaction and the protection afforded the data are wildly mismatched. And unlike a search query, which is transactional and fleeting, these are sustained, deeply personal disclosures accumulated over months, stored in memory systems designed to make the AI feel more like a person who knows you.

We have already seen what corporations do when given access to data we did not fully realise we were handing over. The question is not whether AI companion platforms will eventually face the same commercial pressures. It is when — and whether anything will be in place to limit what they do when they get there.

Not Doom — But Seriousness

None of this is an argument that AI companion technology is inherently harmful, or that every company in this space is acting in bad faith. There are documented cases where AI companionship has reduced isolation for elderly users, provided accessible support for people who cannot afford therapy, and offered a low-stakes environment for people with social anxiety to practise interaction. The technology has genuine potential in these directions. Platform representatives are not wrong when they point to accessibility, anonymity, and user agency as genuine benefits — the problem is that these arguments describe intended benefits while the structural critique addresses predictable harms, and in the current framework the benefits are privatised while the harms are socialised.

The concern is structural. When the economic model depends on maximising emotional engagement with users who are often vulnerable, and when that model is being scaled at extraordinary speed before the sociological and psychological evidence base exists, the industry is taking risks it is not the one bearing. The users are bearing them. Particularly the youngest users.

Being “more deliberate” is not a posture. There are concrete things that would address the structural problems identified here: mandatory pre-deployment psychological impact assessments, similar in principle to environmental impact assessments for infrastructure projects; a legal framework distinguishing confessional data from behavioural data, with corresponding protections including prohibition on training-data use without explicit informed consent; a specific ban on emotional upgrade prompts at moments of peak vulnerability; mandatory human escalation pathways when crisis language is detected; annual public transparency reports covering user demographics, monetisation metrics by segment, and third-party data audits. None of these require banning the industry. They require holding it to the same standard of accountability we apply to industries whose products carry physical risk.

The question being asked by a growing number of researchers, regulators, and ethicists is not whether AI can provide companionship — it clearly can, in some meaningful sense. The question is whether building an industry around it, before we understand what it does to people across months and years, is a reasonable thing to do. The current rate of deployment does not suggest the industry has seriously grappled with that.

We are creating economic engines that profit from encouraging new kinds of human interaction before we understand what those interactions actually do to people. Google had twenty years to normalise data extraction. AI companion platforms have had three. The difference is not merely speed — it is depth. Google knew what you searched for. AI companions know who you are when no one is watching: your shame, your loneliness, your 3 AM confessions. The data is more intimate. The attachment is more profound. The extraction, when it comes, will be more complete.

The slow burn that shaped the internet age is now running on compressed timelines. The work of understanding what is being built, and for whom, falls to everyone who uses these tools, everyone who knows someone who does, and everyone who will live in the society they reshape.

The market will not ask those questions. It has no incentive to. Someone else has to.

Sources & Further Reading

• TechCrunch — AI companion apps on track to pull in $120M in 2025
• American Psychological Association — AI chatbots and digital companions are reshaping emotional connection
• International AI Safety Report 2026 — AI companions: capabilities and risk implications
• SAGE Journals — Cruel companionship: How AI companions exploit loneliness and commodify intimacy
• Psychology Today — The Dark Side of AI Companions: Emotional Manipulation
• SoftPage CMS — How AI Content Is Flooding Social Media & How to Spot It
• Frontiers in Psychology — AI Anthropomorphism and Attachment (2026)
• arXiv — Tracing Users’ Privacy Concerns Across the Lifecycle of a Romantic AI Companion (2025)
• FPF — Understanding Japan’s AI Promotion Act (2025)