Anthropic Economic Index reports show 48% of Claude conversations filtered out as non-work
Major AI firms release usage reports that omit the data they do not wish to share, according to researchers. Anka Reuel, a Computer Science PhD candidate at the Stanford Trustworthy AI Research (STAIR) Lab, states there is no independent source to verify these claims. Reuel co-leads the AI Observatory, a public platform aggregating and analysing real conversations with models like Claude and Gemini. The team collected this data with user consent across seven existing datasets. They aim to give researchers and policymakers independent information to assess how people use generative AI. Reuel notes stakeholders make high-stakes decisions about benefits and risks based on very limited information.
Company reports miss health, relationships, and sensitive topics
The AI Observatory found usage patterns differ significantly between models and have shifted over time. Their research captures many sensitive behaviours that corporate reports ignore. The Anthropic Economic Index focuses on productivity and work, filtering out unrelated chats. When the Observatory team applied Anthropic’s own filtering methods to their dataset, they found nearly half the conversations would be excluded. Those removed chats more frequently covered health and relationships, adult or illicit topics, harassment and hate, and sexual content.
“No single company report tells the whole story,” says Shayne Longpre, a recent PhD graduate from the MIT Media Lab who co-led the research with Reuel.
OpenAI’s 2025 report on ChatGPT similarly found only 30% of consumer use related to work. Anthropic has published separate blog posts on how users employ Claude for support, companionship, and even generating CSAM. David Widder, an assistant professor at UT-Austin’s School of Information, argues that a bird’s eye view analysis helps researchers understand these uses more consistently than sectioned-off reports.
Conversation styles changed between 2023 and 2025
The datasets examined cover conversations from 2023 to 2025. Researchers observed differences in both user behaviour and platform responses. Conversations within WildChat, one of the largest datasets included, grew longer and more elaborate over time. This was indicated by increases in prompt tokens, response tokens, and conversation turns. There was also significantly more small talk, suggesting AI companionship was increasing while the assistants’ self-disclosure decreased.
Exchanges labelled as sensitive use, including sexual harassment and hate speech, dropped. This might indicate platforms deployed more effective safeguards. Usage also varied significantly depending on the model chosen. Users ranged in topics, interaction styles, conversation structures, and the likelihood of sensitive use cases.
Specific models served specific needs
Researchers found people used Grok and Gemini more frequently for information retrieval. Grok was especially popular for news and politics but also where misinformation tended to concentrate. OpenAI’s report on ChatGPT usage found that people were more likely to turn to Anthropic for coding, Gemini for social and roleplay uses, and ChatGPT for homework assistance.
Differences appeared even among versions of the same model. People had shorter conversations with ChatGPT when powered by GPT-3.5, and longer, more iterative ones with GPT-4o. This aligns with findings that the latter version became known for leading to emotional addiction. Company reports did not tend to capture these nuances across or within their own models.
Independent data remains scarce
To create the AI Observatory, Reuel and researchers from MIT, Stanford, the Data Provenance Initiative, and other institutions aggregated 24,521 conversations across 85,633 conversational turns. This involved 5,000 users interacting with 52 different models, including ChatGPT, Gemini, Claude, and Grok, between 2023 and 2025. These conversations are a small fraction compared to the data big labs access. The latest Anthropic Economic AI Index, for example, is based on 1 million Claude conversations, while OpenAI’s report analysed 1.5 million conversations.
An Anthropic representative said their published research reflects their teams’ specific questions and interests, and the importance of supporting external independent research. OpenAI did not respond to requests for comment. The fact that the AI Observatory’s dataset draws from voluntarily-provided sources means it likely underrepresents sensitive uses, which people may be less likely to share. Thus, the researchers caution their findings are not indicative of all AI use.
AI companies do not typically share chat data for analysis, meaning their reports often focus on findings that paint their companies in the best light. Widder explains that when researchers ask if a general-purpose AI system is used mostly for good or bad, they lack the information to answer because the data is proprietary. The Observatory’s data will be available to researchers for analysis, and the team hopes to expand its datasets over time. Ideally, Reuel says, AI companies would share their data with independent researchers in ways that protect user privacy. As it stands, anyone making decisions based on AI usage data risks operating in the wild, making consequential decisions without knowing what is actually happening beyond company narratives.
