We know it can be difficult for human beings to spot AI-generated content, but can detection software fare any better? Pangram is one of the leaders in the field, promising a "near-zero false positive rate" by looking at "subtle cues" in the writing, based on a training and feedback model comparing human-made content with the AI equivalent.

According to recent reports, you'll find arguments on both sides of the debate as to whether or not Pangram's claims stand up. After a short story was flagged by Pangram and then judged to be free from AI, the detection engine has been accused of bias against text from minority writers.

Proponents of Pangram would point to an ongoing reward offer—at the moment at $3,950—available to anyone who can find a pre-2019 (and pre-LLM) piece of writing that Pangram falsely flags as being AI-written. So far, the reward is unclaimed.

Industry observers note that as the AI detector has recently added AI image and video detection to its capabilities, it seemed a good time to put Pangram's analysis skills to the test for myself. Here's how it performed in the three different content types.

On the human side, I picked out 10 of my most recent articles for Lifehacker, known to be written by a person because I personally wrote them—in the traditional way, I fretted over word choices, went back and changed sentences, and accidentally left in a variety of typos that editors thankfully fixed.

Sources indicate that for the AI content, I rotated between ChatGPT, Gemini, and Claude, and got the LLMs to write 10 articles with the same headlines and the same approximate number of words, for a tech-savvy audience. I even asked the AI apps to vary their language use, so that the output would "read less like an LLM has written it.".

Recent findings suggest that pangram's AI text detector lets you upload files, but to make the task a little harder, I copied and pasted all of the text into the online article, so it wouldn't be able to rely on any metadata included in the file. The scans got the correct result each and every time: Human-written text came back as 100% human, and AI-written text came back as 100% AI.

Sources indicate that what's more, Pangram was certain about its results. All of the human writing was marked as highly likely to be human, and the vast majority of the AI writing was marked as highly likely to have come from an LLM. There were only a handful of phrases or sentences across thousands of AI words that got a "medium" rather than a "high" ranking for AI.

It has been revealed that now, there have been some edge cases where Pangram has got it wrong—like a poem repeating the same six words thousands of times—but as far as this sample goes, it can't be faulted. Clearly, my writing is too idiosyncratic and too clunky to be confused for the smooth sheen of AI-produced prose.

To be fair, I do have another advantage over AI: I can actually test out devices and apps in a way that LLM bots can't. ChatGPT can't experiment with One UI 9 (at least not yet), though it can scrape the web for various experiments and tests that real people have done.

In a recent development, however, I was able to fool Pangram to some extent by alternating AI paragraphs with human-written paragraphs. The text was flagged as AI, but the percentage estimate was typically lower (20-25%) than what it actually was (around 50%). If you're offloading some but not all of your writing to AI, some of it might slip though as real.

Sources indicate that image and video detection is the new string to Pangram's bow, so I sifted through my phone's photo and video library for some pieces of content to test the AI on. Again, I used copy and paste to get the files into Pangram rather than uploading them, to limit how much information the AI detector could get from metadata.

Sources indicate that again, all the human-made content was recognized as such, with a very high level of certainty—whether it was complicated crowd shots or simpler images of natural scenes. Presumably, there are even more signals to go off than there are with writing, including camera movements, colors, style, consistency, and composition.

According to recent reports, next up, I generated a few images and videos in Google Gemini and ChatGPT, requesting results that would look as photorealistic and "non-AI" as possible—the resulting files covered people, animals, and buildings. As before, I used copy, paste, and a few screengrabs to try and limit the metadata Pangram had access to.

Industry observers note that here, too, the AI identification was spot on, with all of the artificial content recognized as such, to a high level of confidence. This is perhaps not surprising: I think AI images and especially AI video still have more "tells" than AI writing, especially if you've seen a lot of it, so I wasn't expecting Pangram to fail here.

Recent findings suggest that I did try experimenting with real photos that had a little bit of AI in them, what with tools like Gemini and ChatGPT now so good at image editing, as well as creating pictures from scratch. These were marked as "unsure" by Pangram, though to cut it some slack, only a small amount of the images were altered.

Sources indicate that as with the text, the best way to beat Pangram's detection systems seems to be to include some AI embedded in human-made content. It will most likely still fail the "100% human" test, but the tests I ran showed that Pangram is less certain in these cases, or underestimates the amount of AI there actually is.

It has been revealed that increase the level of AI, and the detection gets more reliable—but if you're only using AI for a small part of whatever it is you're making, it's less worthwhile using AI tools anyway. As far as these tests go, Pangram's claims about accuracy check out, making it a useful tool if you need to ensure something is AI-free.

→ Read original article