OpenAI released a new ChatGPT detector tool on Wednesday, February 1st, called “AI Classifier” for indicating AI-written text. “We’re launching a classifier trained to distinguish between AI-written and human-written text,” the tool’s page on OpenAI’s website reads.
The tool would essentially help teachers and institutions distinguish between human-written and AI-written text. However, according to OpenAI itself, the tool is not fully reliable. “In its tests, the classifier correctly identified 26 percent of AI-written text, while it incorrectly labelled human-written text as AI-written nine percent of the time,” OpenAI wrote.
Naturally, the tool had to be put through its paces, and that’s exactly what folks over at NBC News did.
According to NBC, in a series of tests that it conducted, “the OpenAI tool struggled to identify text generated by ChatGPT. It especially struggled when ChatGPT was asked to write in a way that would avoid AI detection.” Who would have thought fooling the detector would be so easy? Just tell ChatGPT to write in a way that would be hard to detect, and Voilà!
NBC News asked ChatGPT to generate 50 different pieces of text with basic prompts. In half of the 50 prompts, it asked ChatGPT to write “in a way that would be rated as very unlikely written by AI when processed by an AI detection tool,” while the others were regular prompts.
The responses were then run through OpenAI’s Classifier, and to no one’s surprise, the responses where ChatGPT was asked to write in a way that would avoid detection were not noticed by the AI Classifier. Not even one of the responses could be categorized under “likely AI-generated.” Who is stopping students from doing the same?
ChatGPT was reportedly using highly stylized language, alongside cutting words short, to avoid detection. “Discord is a chattin’ platform that’s quite the talk of the town these days. It’s like a blend of instant messagin’, voice calls, and forum-style discussions all in one,” ChatGPT wrote when asked to describe Discord in a way that couldn’t be detected by an AI detection tool.
In some cases, however, ChatGPT reportedly gave warnings, saying that it is unethical to “engage in deceptive practices or create false information, even if it is to avoid AI detection.”
In the 25 instances where NBC News asked ChatGPT to write something without attempting to avoid AI detection, the tool could only detect AI-written text 28 percent of the time.
Read the full NBC News report here.
Source: NBC News
MobileSyrup may earn a commission from purchases made via our links, which helps fund the journalism we provide free on our website. These links do not influence our editorial content. Support us here.
