Copyleaks Blog

Your learning destination for all things responsible AI, plagiarism and beyond.

Featured Image pen

Beyond The Red Pen: Writing Tools & GenAI

  • Copyleaks analyzed whether certain writing assistant tools and their key features get flagged as AI content.
  • The study found that content altered by writing assistant tools that do not use genAI features to correct grammar, tone, mechanics, etc., does not get flagged as AI-generated content.
  • The Copyleaks data revealed that copy rewritten by writing assistant tools using genAI-powered features like GrammarlyGo will likely be flagged as AI.
  • A smaller analysis from another AI detector flagged 20% of Grammarly’s non-AI features, which include spell check and grammar, as AI content at an astounding 20% false positive rate. Such results highlight the importance of utilizing more sophisticated, enterprise-level detection platforms like Copyleaks that can decipher what functions are truly AI to avoid false positives.

Writing Assistant Tools In the Age of AI

Writing assistant tools are nothing new. 

MIT created the first spell-check tool in 1961, and by the early 1980s, spell-check had become part of PCs for individual users. In 2009, Grammarly arrived as a ‘writing partner’ that checked your spelling and helped you fix grammar mistakes, adjust tone, and more. In short, writing assistants have been a part of our daily lives for a while now.

Then, generative AI crashed the party.

Many of these writing assistant tools have been utilizing AI to help with essential functions for a while, so what’s different? 

Yes, writing assistant tools have been using AI, but now, some have evolved to use large language models (LLMs) to rewrite content, fill in sentences, and more. 

What does that mean? Well, depending on what writing assistant features you use, your content could get flagged as AI-generated, posing problems for students, marketers, bloggers, and more. 

Do Writing Assistants Get Flagged As AI?

To determine what exactly makes content edited by writing assistant tools get flagged as AI, we tested using two AI-powered writing tools: Grammarly and our tool, the Copyleaks Writing Assistant.

We collected 1,000 random files from a public essay dataset of English-language text. The dataset is designed to be English only and does not contain AI. We then ran that content through the Copyleaks Writing Assistant and Grammarly. 

Here is what we found.

  • One thousand human-created files were run through the Copyleaks Writing Assistant, with each one averaging around 35% of changes made. These updated files were scanned through the Copyleaks AI Content Detector. All 1000 came back as human content.
  • The same original files were run through Grammarly for correctness and clarity (Note: We used only the more commonly used corrections portion of Grammarly; this does not include their GenAI platform, GrammarlyGo).  These updated files were also processed through the AI Content Detector. All 1000 returned as human content.
  • We then took 500 of the same original files and used Grammarly’s GenAI  ‘Improve’ feature.  These updated files were processed through the AI Content Detector. 31.6% returned as AI Content and 68.3% as human.  
Chart of Copyleaks Writting Assistant vs. Grammarly Original and Grammarly's GenAI 'Improve' Feature

Things to Note

  1. In the last experiment, the content coming back as human-written had fewer overall changes than those coming back as AI.
  1. A high cosine similarity between the original text and the ‘Improved’ means they are very similar. “1” is nearly identical, and the texts are less similar as the value gets lower. The average cosine for the files detected as AI was .76, while the files identified as human-written was .88. This means the text detected as AI was significantly changed, resulting in a lower cosine score than the source. In contrast, texts with minimal changes had a higher cosine score than the source and were not detected as AI.
  1. A much smaller analysis, similar to ours, was done by another AI detector on the market. The crucial difference is that in their study, 20% of Grammarly’s non-AI features, which include spell check and grammar, were flagged as AI content, resulting in an astounding 20% false positive rate. We took the exact text performed in their analysis and ran it through the AI Content Detector, and it came back with 0% AI content detected. These results highlight the importance of utilizing more sophisticated, enterprise-level detection platforms like Copyleaks that can decipher what functions are truly AI. 
Originality.AI Grammarly Test Chart

The Takeaway

Only certain features of writing assistants can cause your content to be flagged by AI detectors as AI-generated.

When it comes to tools such as Grammarly, which offers a genAI option to help re-write your content entirely, whether you’re asking it to improve or shorten the text, it may very well get flagged as AI because, simply put, it is AI-generated. 

On the other hand, the Writing Assistant did not get flagged as AI, nor did any of the content that Grammarly changed to fix grammatical errors, tone issues, etc., because they do not use genAI to power these features or functionalities.

Bottom line: AI changed the writing assistant landscape, as it has with many others. So, while using writing assistants in the traditional sense does not get flagged as AI, to be on the safe side, you’d be better off using a writing tool, such as the Writing Assistant, that does not offer a genAI feature to ensure that your content remains crisp, clear, and original, and free of AI content.

Copyleaks AI Content Detector at the top of the Studies' Research

AI Content Detector Continues To Be Confirmed As Most Accurate By Third-Party Studies

In July 2023, four researchers from across the globe published a study on the Cornell Tech-owned arXiv, declaring Copyleaks AI Content Detector the most accurate for detecting Large Language Models (LLM) generated text. 

For the study, the researchers collected 124 submissions from computer science students written before ChatGPT was created. They then generated 40 ChatGPT submissions and used the data to evaluate eight publicly available LLM-generated text detectors. 

Here were their key findings,

That was just the beginning. 

Since then, additional independent third-party studies have been released, each one highlighting the accuracy and efficiency of the AI Content Detector.

Check out the findings below. 

Study: Detecting AI Content in Responses Generated by ChatGPT, YouChat, and Chatsonic: The Case of Five AI Content Detection Tools

Author: Chaka Chaka, University of South Africa

Published: July 2023

Multiple Languages
check mark
check mark

Read the full study here.

Study: How Hard Can It Be? Testing the Reliability of AI Detection Tools

Author: Daniel Lee, University of Adelaide

Published: September 2023

ai detection results after human modifications

Read the full study here.

Study: The Effectiveness of Software Designed To Detect AI-Generated Writing: A Comparison of 16 AI Text Detectors

Author: William H. Walters, Manhattan College

Published: October 2023

copyleaks most accurate ai content detector

Percentage of all 126 documents for which each detector gave correct, uncertain, or incorrect responses.

Read the full study here.

Study: Accuracy Pecking Order – How 30 AI detectors stack up in detecting generative artificial intelligence content in university English L1 and English L2 student essays

Author: Chaka Chaka, University of South Africa

Published: April 2024

Only two AI detectors, Copyleaks and Undetectable AI, managed to correctly detect all of the essay sets of the two English language categories (English L1 and English L2) as human written. As a result, these two AI detectors jointly shared the first spot in terms of the GenAI detection accuracy ranking.

Read the full study here.

Study: How Sensitive Are the Free AI-detector Tools in Detecting AI-generated Texts? A Comparison of Popular AI-detector Tools

Author: Sujita Kumar Kar, Teena Bansal, Sumit Modi, and Amit Singh, Indian Journal of Psychological Medicine

Published: May 2024

Read the full study here.

Study: Boston University AI Task Force Report on Generative AI In Education and Research

Author: BU AI Research & Teaching Taskforce

Published: April 5, 2024

Study Quote

Read the full study here.

As the studies show, with Copyleaks, you can have complete peace of mind with a solution that covers all of your AI detection bases, evolves with the ever-changing technology, and ensures originality while inspiring authenticity.