Readers rate AI-generated short stories higher than human ones until they learn a machine wrote them
Participants in a study could not distinguish between fiction written by people and fiction written by ChatGPT. They rated the machine-generated stories as better, but only when they were unaware of the author.
Three separate tests involving more than 2,500 people found that subjects performed no better than random guessing when trying to identify the source of the text.
The first experiment involved 1,682 participants. Each person read one of six short stories, roughly 1,000 words in length. Three came from established literary magazines and collections. The other three were created using ChatGPT 4.0. The prompts for the AI were designed to match the theme, style, and narrative perspective of the human originals.
Half the group was told the story was written by a human. The other half was told it came from ChatGPT. This information was accurate for only half the participants in each group. Researchers Sydney Sears and Deena Skolnick Weisberg published the findings in the journal Judgment and Decision Making.
When told the story came from a machine, participants gave it higher scores for both perceived quality and immersion. On a scale from minus 3 to plus 3, the mean score for AI stories was 1.54 compared to 0.97 for human stories. For immersion, the gap was 1.42 versus 1.00.
Personal attitudes toward artificial intelligence also influenced the ratings. Participants gave higher scores when told a human was the author, regardless of the actual origin. Those with a positive view of AI generally gave higher ratings across the board. When they were also told the story came from ChatGPT, their scores rose even further. Among participants who were sceptical of AI, this effect reversed. An earlier study on AI-generated poems found a similar bias.
In two further experiments with 905 participants, the researchers made the task harder. Each person read both a human-written and an AI-generated story, then had to decide which was which. Even with a direct comparison, participants performed no better than chance.
Self-reported experience with AI systems correlated positively with the ability to correctly identify the stories’ origins. Experience with fiction, on the other hand, did not help participants tell them apart.
Why the ratings are higher
AI-generated texts tend to be smoother, easier to read, and more emotionally upbeat than human-written texts. The authors suggest these traits explain the higher ratings without the AI stories actually being better in a literary sense. People tend to prefer material that is easier to process.
High-quality literary fiction is often intentionally difficult to access and pushes readers to work for meaning. A story can be high quality but not very engaging, and vice versa, the researchers suggest.
The short story format also likely works in AI’s favour. Telling a coherent story in 1,000 words is a very different challenge than doing so across hundreds of pages. The researchers’ conclusion is clear: AI can generate creative works people perceive as at least on par with human work, yet people do not believe AI is capable of that.
All data and materials from the study are freely available on the Open Science Framework.
A study from last October by Stony Brook University and Columbia Law School shows that reader expertise matters only up to a point. With simple prompts, professional readers clearly preferred the human-written texts. But when models were specifically trained on individual authors’ styles, the experts preferred the AI-generated texts eight times more often for style imitation and twice as often for writing quality.
