Since November 2022, generative AI writing has transformed many facets of life and education is no exception. Students, educators, administrators, unis, schools, parents, and the edtech industry as a whole have been working towards not only understanding AI-generated writing technology, but also finding ways to safely leverage it in the classroom.
In response to educators’ request for a tool that could help them identify AI writing similarities in students’ writing submissions, Turnitin quickly answered the call with its AI writing detection feature in April 2023. Since its debut, Turnitin has successfully reviewed over 76 million papers globally.
The need for this type of support is not slowing down—especially with so many students starting a new term this autumn. According to an investigation by youth media outlet The Tab, “More than 40 percent of unis who responded to their inquiry said they have investigated their students for using ChatGPT or a similar AI bot in an assessed piece of work.”
Furthermore, of the 76 million papers reviewed by Turnitin, over 4 million—3.3 percent—have been flagged as having at least 80 percent AI writing present. Nearly 7.8 million—10.6 percent—have over 20 percent AI writing present.*
ENAI researchers investigate AI writing detection effectiveness
Therefore, it's no surprise that a group of veteran European researchers got to work from March to May 2023 investigating the effectiveness of AI writing detection tools. The research group is part of the European Network for Academic Integrity (ENAI), one of Turnitin’s regional partners. Most recently, Turnitin participated in their annual European Conference on Ethics and Integrity in Academia (ECEIA).
According to the study’s abstract, “recent advances in generative pre-trained transformer large language models have emphasised the potential risks of unfair use of artificial intelligence (AI) generated content in an academic environment and intensified efforts in searching for solutions to detect such content.”
In the spirit of transparency, and as a partner committed to continually improving its technologies to meet consumer demand, Turnitin gave the research group access to their own Turnitin account to test fifty-four different academic writing submissions in English, alongside twelve other publicly available tools and one commercial tool.
Their test sought to evaluate the performance and quality of AI writing detection tools with a special focus on whether or not the “detection tools can reliably differentiate between human-written text and ChatGPT-generated text, and whether machine translation and content obfuscation techniques affect the detection of AI-generated text.”
The methodology for the test cases included the following writing types:
- Human-written
- Human-written in a non-English language with a subsequent AI/machine translation to English
- AI-generated text
- AI-generated text with subsequent human manual edits
- AI-generated text with subsequent AI/machine paraphrasing
While the study found that “the available detection tools are neither accurate nor reliable and have a main bias towards classifying the output as human-written rather than detecting AI-generated text,” Turnitin’s detection feature scored higher than the others. Furthermore, the study emphasises the need for a broader discussion on AI writing detection.
Opportunity to fundamentally rethink assessment
While there is clear value in objective research to help educators make evidence-based decisions on whether to employ AI detection tools in their classroom, the phenomenon of AI writing needs to be viewed in the wider educational context.
The study’s authors assert AI writing tools offer institutions and educators the opportunity to fundamentally rethink the way educators assess students’ knowledge and learning. With our current assessment practices, are we equipping students effectively to make the next step in their educational career? Surely this is the real opportunity offered by this research.
AI writing tools, just like the internet and later, social media, which have grown exponentially over the last 20 years, will continue to grow in reach and capability. Just as students have grown to be technologically literate in many aspects of their social and academic lives, they will now be required to develop a similar level of AI literacy.
As a former librarian, I have seen and continue to see a clear need for developing students' information-handling and literacy skills, so they can discern quality sources to use in their studies, and to establish which voices are both authoritative and credible. These skills go beyond the classroom and become fundamental to our everyday, online professional lives.
Therefore, the conversation may need to be less about whether students have used AI writing tools in their assessed work, but rather whether they use them correctly, ethically, and with the knowledge and permissions of the educator.
As educators, we need to focus our attention on developing these skills to help students navigate AI writing tools.
At Turnitin, our guidance is, and has always been, that there is no substitute for knowing a student, their writing style, and their educational background. Educators should be using AI reports as resources, not deciders, and educators should always make final determinations based on all of the information available to them.
In cases where there are indications of AI use or other unauthorised resources in academic work, we recommend that the first step be a conversation with the student, in the form of an informal follow-up. In most cases, these actions will resolve the issue in a way that is both safe for the student, and impactful for their growth and learning.
Turnitin is committed to helping educators adapt to this ever-evolving technology. For more information about the AI writing detection feature and to check out the latest, regularly updated, resources and guides, visit Turnitin’s AI writing landing page and the Turnitin blog.
*As of July 2023.