Wikipedia's AI Cleanup Volunteers Hunt 'Stands as a Testament' Prose as Pangram Flags About 20% of New Articles as AI-Written

The first tell was the vocabulary. Ellia Lebleu and a handful of other Wikipedia editors kept running into new articles that “repeated phrases like ‘stands as a testament’ or ‘rich cultural heritage.’” Nobody writes an encyclopedia entry that way. Chatbots do.

Around December 2023, a few of them talking on Wikipedia’s Discord decided, in Lebleu’s words, “let’s create an official project on Wikipedia.”

The project’s founding signatures are dated December 4, 2023. Its member list now has more than 300 entries. Their main tool is a field guide the community keeps, a page called Signs of AI writing, that catalogs the habits a language model tends to leave on the page.

Some are words: forms of “stands/serves as,” “is a testament/reminder,” and “a crucial/pivotal/vital/significant/key role/moment.” Some are structures. Lebleu told Spectrum about articles that end with a tidy conclusion section, headings like “Why is this important?”, lists that always come in threes, and “negative parallelisms,” the “Not only … but …” and “It is not just …, it’s …” patterns. Some are behavior: one editor produced five articles of thousands of words each in about five minutes.

The guide carries its own warning: “The patterns listed here are also only potential signs of a problem, not the problem itself.”

The rules caught up. English Wikipedia’s guideline on large language models now says “the use of large language models (LLMs) to generate or rewrite article content is strictly prohibited,” with exceptions for copyediting your own writing and translating from another language’s Wikipedia. Under a speedy-deletion criterion the project page cites, “any pages (not just articles) that are clearly entirely LLM-generated can be nominated for speedy deletion.”

Nobody knows yet why the share dropped from about 20% to 15 or 16%. It could be the stricter policy, or it could be that newer models write less like the old tells. Spectrum notes it’s unclear which.

There’s also a strange twist in the models themselves. “We’ve seen Claude sometimes say,” Lebleu told Spectrum, “I can’t write a Wikipedia article; it would go against Wikipedia’s policies.”

Many people have called Wikipedia “the last good place on the internet,” Lebleu said. The Wikimedia Foundation has added a tool called Paste Check, and one editor has built an AI source verifier, but whether the share of machine-written drafts keeps falling or bounces back with the next generation of models is something the next Pangram scan will have to show.