Artificial intelligence chatbot ChatGPT has made quite an entrance with its capacity to produce human-like textual dialogues. However, there are numerous subtle indicators that allow us to separate AI chatbots from human conversation, as a study published today in the journal Cell Reports Physical Science illustrates. From these indications, the team devised a tool that can identify AI-created scientific academic writing with a precision of over 99%.
Heather Desaire, the first author and a professor at the University of Kansas, said: “We tried hard to create an accessible method so that with little guidance, even high school students could build an AI detector for different types of writing.
“There is a need to address AI writing, and people don’t need a computer science degree to contribute to this field.”
Desaire highlighted current challenges with AI-generated writing, noting that, “One of the biggest problems is that it assembles text from many sources and there isn’t any kind of accuracy check — it’s kind of like the game Two Truths and a Lie.”
While many AI text detectors function reasonably well and are freely accessible on the internet, they are not specifically crafted for academic writing. To bridge this gap, the research team endeavored to devise a tool optimized for academic writing, with a focus on ‘perspectives’ – articles that overview specific research topics penned by scientists. They trained the model on 64 perspectives and 128 ChatGPT-produced articles on the same research subjects. A hallmark of AI writing, predictability, was discerned upon comparison.
In contrast to AI, human-created paragraphs exhibit more complexity in structure, with varied sentence and word counts, and a changing sentence length. Choice of punctuation and vocabulary can also reveal AI writing. For instance, scientists tend to use words like “however,” “but,” and “although,” while ChatGPT frequently employs “others” and “researchers.” The team incorporated 20 such traits into the model.
The model achieved a perfect score in identifying full-perspective articles generated by AI and distinguishing them from human-written ones. It displayed 92% accuracy in recognizing individual AI-created paragraphs within these articles. The tool also vastly outperformed an existing AI text detector in similar tests.
The team’s future plans involve assessing the breadth of the model’s application. They aim to evaluate it on larger datasets and across diverse forms of academic scientific writing. As AI chatbots continue to evolve and improve, they are interested in determining if their model will remain robust.
On the applicability of the model for educators, Desaire commented, “The first thing people want to know when they hear about the research is ‘Can I use this to tell if my students actually wrote their paper?'”.
She clarified that while the tool excels at differentiating between AI and scientific writing, it was not designed to detect AI-created student essays. However, she mentioned that their methods could be easily replicated to create models serving specific needs.
Image Credit: Jaap Arriens/NurPhoto via Getty Images