You can now detect whether text was written by humans or AI… sort of

We’ve been covering a lot of different AI utilities of late. We’ve introduced you to ChatGPT, which writes for you, Dall-E which paints for you, and Vall-E which speaks for you. However, it is now time we switch gears and shatter a few dreams that these AI utilities may have inspired. There are ways to test whether something has been written by AI, and below, we’re going to show you how. 

Disclaimer: I have spent a considerable amount of time on these tools and others like them, and, unfortunately, there’s a massive issue with how they work. We’ll discuss the issue once I’ve reviewed the three best products for the job, but I just want you to enter this article with the knowledge that AI detection tools are not perfect. 

GPT-2 Output Detector

This first utility hails from the same stable as ChatGPT. Created by OpenAI, GPT-2 Output Detector is one of the foremost detection tools in the industry. OpenAI has been working on mimicking and duplicating human artistic ability and intelligence since 2015. The company went viral in 2022 on the back of the release of ChatGPT, a tool that inspired so many sleepless nights and cold sweats until I realized that it couldn’t replace me. 

However, while the company was working on the tool I thought would replace me, t was also working on a way to make sure that didn’t happen to anyone in my field. OpenAI created a way to make ChatPGT less usable in professional circles, and a way to detect if it had been used. 

You can use the online demo version of this tool right now, although we don’t know how long before it’s replaced by the final software. All you need to do is head to the website in the link above, paste your text into the box, and the tool will determine whether the text was written or generated. 

GLTR (Giant Language Model Test Room)

Another company that took rather diligent note of the OpenAI and its AI utilities is the MIT-IBM Watson AI Lab. In 2019, when OpenAI released GPT-2, the Lab joined forces with the Harvard Natural Language Processing Group to create a utility that could detect computer generated text. This tool is GLTR, and it’s relatively clever. See, it uses an approach known as ‘takes one to know one.’ This means that it uses its prowess as an AI to process a piece of text and determine each next word. If the word that GLTR predicts matches the actual text, it’s most likely generated by AI. If the text deviates and GLTR isn’t able to predict what comes next, the text is most probably written by a human. 

GPTZero

We’ve actually written about this one before. Edward Tian is a senior at Princeton University, and he has previously expressed great concern for students using tools like ChatGPT in their university assignments. Edward believes that professors would have a keen interest in a tool that makes it easier to determine whether any text has been written by human hands or generated by AI bots. Therefore, he created GPTZero.

Edward Tian is a very clever young individual, at only 22 years old. He explains that GPTZero works by detecting randomness in sentences. Two other words he used were ‘perplexity’ and ‘burstiness.’ I won’t pretend to understand how GPTZero works, but I can tell you it has issues, just like the rest of these tools, including those not on this list. 

The trouble with AI detectors

As alluded to previously, I have spent a lot of time with both ChatGPT and the detectors that are meant to make it more difficult to use. My method was to write a piece of text myself and generate one using ChatGPT. I then ran both pieces of text through as many AI detector models as I could find. The results were always the same. One after the other, the detector utilities identified the AI-generated text 100% of the time. This doesn’t sound like an issue, and it isn’t. The issue comes in with how my own work compared. 

Again, one after the other identified my work as either written entirely by AI or in large part. I became anxious and tried a few more pieces of my own original work, with largely the same results. 

The issue is that people like me write for algorithms. We write for humans first, otherwise, there’d be no joy in this job. But, we also write for algorithms. Algorithms determine whether our choices of words are too complex for the average audience, whether our sentences are too long, whether our articles are concise enough to keep the audience’s attention, and whether Google and other search engines will regard our articles favorably. We’re being taught, inadvertently, to write how algorithms and AI would prefer. And, when we use these detectors, it shows. The other aspect to keep in mind is that AI was trained in our writing. Therefore, if an AI’s generated text seems robotic and pleases the Google overlords, what does that say about those of us who inadvertently trained it? 

To summarize, AI detectors are really good at detecting AI. Unfortunately, they often detect AI in human hands.

Source