AI-Enabled Fakery Has ‘Infiltrated Academic Publishing,’ Say Researchers

'Vegetative electron microscopy’ appears in dozens of papers, but it does not exist.

The use of AI to create bogus scientific papers is becoming an increasingly serious problem, researchers say, threatening life-saving medical research as well as the integrity of academic publishing.

Last year Computing reported that AI tools such as ChatGPT were being used to assist with a significant number of research papers, mostly without proper disclosure.

In that study, researcher Andrew Gray of UCL found a substantial increase in the frequency of words such as commendable, innovative, meticulous, lucidly and impressively in recently published scientific papers, stating that this is a fingerprint of LLM-generated output. He suggested that suggests the lack of transparency over the use of these tools could ultimately undermine trust in academic publications.

Now a new study by researchers at Retraction Watch, Université Grenoble Alpes and Institut de Recherche en Informatique de Toulouse has found that this is just the tip of a very large and dangerous iceberg. the production of fake research papers by so-called paper mills has become a rapidly growing industry, further threatening the integrity of scholarly work.

“Fraudsters have infiltrated the academic publishing industry to prioritise profits over scholarship,” they say in an article in The Conversation. “Equipped with technological prowess, agility and vast networks of corrupt researchers, they are churning out papers on everything from obscure genes to artificial intelligence in medicine.”

They continue: “Just as highly biased websites dressed up to look like objective reporting are gnawing away at evidence-based journalism and threatening elections, fake science is grinding down the knowledge base on which modern society rests.”

In many cases, fraudsters use GenAI to create plausible sounding content. Using a tool called Problematic Paper Screener the researchers scanned the literature for “tortured phrases” such as “creepy crawlies” instead of insects and “preparation information” in place of training data. These tortured phrases are used to avoid direct plagiarism which is easier to detect.

There are other signs of GenAI fakery too.

Retraction Watch, an organisation that monitors the retraction of papers from academic journals, mentioned a strange phrase “vegetative electron microscopy” cropping up in several papers, including some published by high profile journals, such as Springer Nature. It appears in another paper, still available, in Elsevier’s ScienceDirect, which was co-authored by Vijay Kumar Thakur, a special content editor at Elsevier.

“The ludicrous phrase is what sleuths call a ‘fingerprint’: an offbeat characteristic found in one or more publications that suggests paper-mill involvement,” said Retraction Watch contributing editor Frederik Joelving.

Vegetative electron microscopy is not a recognised term, so where had it come from?

Russian chemist “Paralabrax clathratus” and Alexander Magazinov, a software engineer from Khazakstan who first noticed the phrase think they know.

They speculated that it could have been the result of the inability of an AI model to properly process the two-column format, still commonly used in academic papers. They found a paper from 1959 in which “vegetative” appears in the left-hand column opposite “electron microscopy” in the right. The model could have seen the column separator as a space.

The industrialised fakery enabled by new tools, together with the pressure on academics to publish, risks degrading academia. The sheer volume of papers submitted makes it very difficult for editors and reviewers to keep up. Publishers estimate that 2% of submitted papers are fake, but it is likely far higher than that, particularly in certain areas such as biology, medicine and computer science.

Papers are being retracted by publishers at record rates and in some cases entire journals have been shut down because of the amount of fraudulent paper-mill content they were found to contain.

It’s a serious problem. Fake papers are slowing down research into medicines, with cancer research particularly badly hit, according to the researchers.

And it’s an issue that is likely to get worse as the pressures to publish increase and tools enabling fakery improve.

This article originally appeared on our sister site Computing.