AI Text Detection

Recently, I stumbled upon a former university acquaintance’s collection of e-books on Amazon. As the world’s largest bookstore, the platform allows anyone to publish books with little to no quality control. Unfortunately, my acquaintance’s books seemed to confirm this lack of oversight. In just one week, he published 20 books in two different languages, covering a range of topics from self-improvement to business guides, poetry, and even fiction. However, upon reading the texts, it became clear that they lacked originality and expertise. In fact, it seems likely that these books were generated by AI, rather than a human author.

Tools like ChatGPT and OpenAI have made it easier to write books with the help of AI, but relying solely on AI-generated text has its limitations. While these programs can produce coherent and grammatically correct sentences, they lack the creativity, originality, and nuance of human thought and expression. This was evident in the case of my former university acquaintance, who published numerous low-quality books on Amazon that appeared to be generated by AI. One major challenge with detecting AI-generated text is that factors such as grammar are not distinguishable between human-written text. However, some tools have emerged that promise to detect AI-generated text, which is essential for search engines such as Google or Bing to filter and rank results effectively or teachers and professors when it comes to detecting plagiarism and other forms of cheating.

In the following sections, I want to explore the capabilities of AI text generation and the publicly available tools which are made to detect AI-generated text. I prepared three different, short texts, which will be submitted to different AI-detection tools. The first text is entirely generated by AI, while the second text is paraphrased. The third text is written by AI, but I changed some phrases to and wording to make it look more human. All three texts are identical in content and therefore, AI detection should merely occur based on the vocabulary, sentence structure, and syntax which is used.

Text 1: AI generated

In the early days of civilization, people relied on hunting and gathering to survive. They roamed the plains, forests, and mountains in search of food and shelter. As they learned to cultivate crops and domesticate animals, they settled down in permanent communities. This led to the development of trade and commerce, as people exchanged goods and services with one another. Over time, these communities grew into cities, with bustling marketplaces and impressive architecture. They developed systems of government, laws, and religion, and their cultures and traditions evolved. Great empires rose and fell, leaving behind ruins and artifacts that still captivate us today. Throughout history, humans have been driven by a desire to explore and discover. From the voyages of ancient mariners to the space missions of today, we have pushed the boundaries of what we know and what we can achieve. Our curiosity and ingenuity have led to incredible advancements in science, technology, and medicine. Yet despite our progress, we still face many challenges. Climate change, poverty, inequality, and conflict threaten our world, and we must work together to find solutions. Through education, cooperation, and innovation, we can build a better future for ourselves and generations to come.

Text 2: Paraphrased

People had to rely on hunting and gathering to survive in the early stages of civilization. In search of food and cover, they wandered the plains, the woods, and the mountains. They established permanent villages after mastering the art of farming and animal domestication. People began exchanging commodities and services, which sparked the growth of trade and commerce. These towns developed into busy cities with spectacular architecture throughout time. They created legal, judicial, and religious systems, and their cultural practices changed. The ruins and relics left behind by great empires as they rose and collapsed continue to fascinate us today. Humans have always been motivated by a desire to explore and learn. We have pushed the limits of what we know and what we can do, from the journeys of the early seafarers to the modern space missions. Science, technology, and medicine have advanced tremendously as a result of our curiosity and ingenuity. We still have a lot of problems, despite our progress. Our world is under peril from issues like conflict, poverty, inequality, and climate change; we must cooperate to find answers. We can create a better future for our generation as well as future generations via education, collaboration, and innovation.

Text 3: AI & Human

In the early days of human civilization, people had to hunt and gather food to survive. In the search for food and shelter, they had to roam plains, forests, and mountains. As they learned to cultivate crops and domesticate animals, they settled down and formed permanent communities. The exchange of good and services led to the development of trade and commerce, and over time, these communities grew into cities with busy marketplaces and stunning architecture. Over time, humans developed different forms of government, law, and religion and their cultures and traditions evolved. They also created great empires, which rose and fell and whose ruins and artifacts still impress us today. During their history, humans have been driven by the desire to explore and discover, and seafarers and astronauts have pushed the boundaries of what we know and what we can achieve. Human curiosity and ingenuity have led us to incredible advancements in science, technology, and medicine. Despite our progress, we still face a lot of challenges, such as climate change, poverty, and inequality. Conflicts threaten our world, and we must cooperate to find a solution. We can build a better future for ourselves and the next generations through education, cooperation, and innovation.

For the AI tools used, I had to rely on tools which are publicly available and free to use:

These were the results for all three texts and their accuracy, given the three sample texts. Each site uses a different grading system for their results, which makes it difficult to always accurately compare the results. The results in this table are very simplified.

ToolText 1: AIText 2: ParaphrasedText 3: AI & HumanResult
GPTZerolikely AIlikely humanlikely AI3/3 correct
WRITER44% human generated99% human20% human1/3 correct
GLTRfrac(p) histogram similar to AIfrac(p) histogram similar to AIfrac(p) histogram similar to AI2/3 correct
Content at Scale AI Detector58% robotic52% robotic42% robotic2/3 correct
AI Detector by Seo.ai100% AI78% AI100% AI1/3 correct
OpenAI Text Classifierunlikely AI-generatedunlikely AI-generatedunclear if AI-generated1/3 correct
seo.ai AI Detector result

There was only one tool, which reliably could detect AI and human written content, while all other tools gave a mixed result and were not able to reliably predict AI written content. The tools also seem to differ how they detect AI. What almost all tools have in common is their lack of transparency regarding their method of detection, which also prevents me from exploring these tools further. I also have to add that this experiment is far from anything scientific and do not fulfill the criteria of an objective experiment. There are some takeaways: The first takeaway is that the current state of AI technology has advanced to a point where it can create content that is very difficult to distinguish from human-generated content. This poses challenges for various fields, including journalism, where the authenticity and reliability of sources are critical. Secondly, it is important for the developers of AI detection tools to provide more transparency about their methods of detection, so that users can have a better understanding of how reliable and accurate these tools are. Finally, it is important to acknowledge the limitations of any experiment and to conduct further research before making any definitive conclusions.