Machine Learning Effective in Catching Contract Cheating Finds New Research

Three in five University students who engage in contract cheating will be caught by markers using machine learning software, new academic research has found.

The research conducted by Associate Professors Phillip Dawson and Wendy Sutherland-Smith from Deakin University, in collaboration with Principal Product Manager Mark Ricksen from academic integrity solutions provider Turnitin, is the first of its kind into the potential promise of machine learning to address the problem of contract cheating.

Contract cheating occurs when students outsource their assessed work to a third-party and submit it under the pretense of being their own work. Contract cheating is problematic and prevalent in Australia, with six percent of University students admitting to having obtained an assignment through a third-party.

The research presents the first quantitative empirical study into the use of either authorship analysis technology or machine learning to improve detection rates of contract cheating. Using an early alpha version of Turnitin’s new Authorship tool, which evaluates whether an assignment was written by the same student or not, 24 experienced markers were asked to spot contract cheating in bundles of 20 student assignments, which included 14 legitimate assignments and six purchased from cheating sites.

When markers were paired with a machine learning system and provided with authorship report for an assignment, their accuracy of detecting contract cheating increased significantly, from 48 percent to 59 percent, without a significant increase in false positives (incorrectly flagging contract cheating); in other words, markers were able to accurately identify three out of five cases of contract cheating.

The authorship report generated for markers provides an analysis of the student assignments’ linguistic attributes, such as sentence complexity, sentence length, and other stylometrics, as well as document information such as date created and last modified. The report does not specify whether contract cheating has occurred, but rather provides a recommendation for investigation based on statistical measures.

Phillip Dawson, Associate Director, Centre for Research in Assessment and Digital Learning, at Deakin University said the research demonstrated how machine learning can be an effective component of institutional strategies to address contract cheating.

“When markers were provided with a copy of the authorship report to review evidence and corroborate information, they were able to more accurately determine whether assignments had been authored by the student or if contract cheating had occurred. In addition to potentially improving detection rates, authorship analysis approaches using machine learning also offer benefits in terms of raising awareness about contract cheating and efficiently providing evidence if they wish to take their suspicions further.”

Following this alpha version, Turnitin is in development of a machine learning prediction model to offer a more robust recommendation system that prioritizes papers across a corpus of documents or students in a cohort by level of suspicion of inconsistent behavior. Turnitin believes this will replicate the “gut feeling” a marker experiences when suspicion of authorship of a student’s work occurs.

“Whilst Authorship Investigate was in early stages of development when this study was conducted, we’re pleased to see the value of the tool in the detection process, in bringing together all submissions made by a student and allowing rapid scanning of key points of evidence,” said Mark Ricksen, Principal Product Manager at Turnitin.

“Collaboration with higher education institutions and industry enables us to constantly test and iterate our tool so it can be used in a better, faster and more impactful way by markers to address contract cheating. There’s also potential for the software to speed up the investigation process by highlighting submissions of concern by a student and potentially determining the direction and focus of any investigation.”

This study was published in Assessment & Evaluation in Higher Education on 23 September 2019.

You can access the report here.

About Deakin University:

Established in 1974, Deakin University is one of Australia’s largest universities, with five campuses including an online Cloud Campus. Globally connected with internationally recognized quality research and teaching programs, Deakin ranks 261 in the prestigious Academic Ranking of World Universities (ARWU) putting Deakin in the top 1% of the world’s universities.