AI Detection's High False Positive Rates and the Psychological and Material Impacts on Students

The Fundamental Impossibility Of “Ai-Detection” For Writing

We begin by breaking down the “GPT” in ChatGPT, which stands for “Generative Pre-trained Transformer”. The word “transformer” refers to a type of neural network architecture first introduced by the famous paper “Attention is all you need” (Vaswani et al., 2017). Models using transformer architectures can learn about what words are likely in various contexts by processing sequences of text; the model can attend not just to the immediately preceding word, but to words in other positions in the sequence, and even to intermediate representations in other layers of the neural network.

The learning occurs during “pre-training”. In essence, the model learns by playing a “fill in the blank” game: given a sequence of text with a mask in place of one of the words, it predicts what word should replace the mask, and updates its parameters to account for how close or not the guess was to the actual masked word. “Closeness” is based on the representations within the neural network, which are called “embeddings”. A word embedding is a vector of values representing information about the contexts in which that word appears, which can be thought of as representing coordinates in a massively multidimensional space. For any pair of words, their “embedding” vectors can be compared to generate a measurement of how “similar” they are. While there is good reason to be skeptical about precisely how much world knowledge is actually captured in these word embeddings (Bender & Koller, 2020), some cognitive scientists argue that word embeddings closely align with human conceptual knowledge (Piantadosi & Hill, 2022). Research suggests that transformer models can predict nearly all of the variance in human neural responses to sentences (Schrimpf et al., 2021), and that alignment with humans at both the neural and behavioral level is possible even with “developmentally-realistic” amounts of training data (Hosseini et al., 2022). To be clear, this does not mean that LLMs are equivalent to human minds, but it does suggest that we should not be surprised when the output of an LLM is indistinguishable from human writing.