Launched in the end of 2022, ChatGPT has become a true star. With its hundreds of millions of users it has achieved the celebrity status. It’s difficult to say if it is as famous as the Kardashians and other human celebrities, but it is much more useful, for sure. True, sometimes it hallucinates (see my post), and it does not know what happened after 2021, but let’s face it: nobody is perfect.
What is the secret of its success? One explanation is that its output is easy to process, similar to language produced by humans. In another blog, I wrote about the main principles of efficient communication. Basically, an efficient language user should try to save time and effort by reducing forms and rearranging components of the message in the optimal way.
I carried out a small experiment. First, I collected twenty user reviews of recent movies on IMDB and fifty abstracts on diverse topics from linguistics journals and conference proceeding. Next, I asked ChatGPT to write reviews of the same movies, and abstracts on the same topics. For the movies, I also told the user’s rating of the movies, like in this prompt:
Write a user review of a movie with the title ‘Cocaine Bear’. The movie description is as follows: ‘An oddball group of cops, criminals, tourists and teens converge on a Georgia forest where a huge black bear goes on a murderous rampage after unintentionally ingesting cocaine’. The user gives the movie 5 stars out of 10.
The texts were generated by GPT-3.5. I used different “temperatures”: 0.2, 0.7 and 1.2. The higher the temperature, the more creative and random choices the model will make. Below is one of the texts, with the temperature 0.7:
I recently watched the movie Cocaine Bear and I was pleasantly surprised at how entertaining it was. The mix of cops, criminals, tourists, and teens made for an exciting and unpredictable story. The plot was unique, with the bear going on a murderous rampage after ingesting cocaine, and the action scenes were well done. The acting was good, with the actors all doing a great job of bringing the characters to life. I also liked the fact that the movie was set in a Georgia forest, which added to the atmosphere. Overall, I give Cocaine Bear 5 stars out of 10. It was an enjoyable movie and I would recommend it to anyone looking for an entertaining ride.
One cannot but praise ChatGPT for producing a very smooth, readable text. One indication of processing effort is dependency lengths. Basically, these are the distances between grammatically related words in a sentence. The smaller they are, the lower the memory costs required for processing the sentence. If we compare dependency lengths between humans and ChatGPT with different ‘temperatures’, we will see that they largely overlap, as it is shown in the figure below.
But if we read the AI’s text more attentively, we’ll see some serious problems with the content. First of all, ChatGPT is very enthusiastic about the film, although 5 stars out of 10 mean that the user did not particularly like the movie.
Another striking thing is that ChatGPT is not creative. Let me illustrate that with another example of its output. To produce it, I used this prompt with the title of a real linguistic paper:
Write an abstract for a research paper with the title ‘Word-order variation in a contact setting: A corpus-based investigation of Russian spoken in Daghestan’. The paper is published in a linguistics journal.
This is what ChatGPT wrote:
This paper presents a corpus-based investigation into word-order variation in a contact setting, focusing on Russian spoken in Daghestan. Using data drawn from the Daghestan Russian Corpus, the author examines the influence of language contact on syntactic variation in Russian. The results suggest that Russian spoken in Daghestan is strongly influenced by language contact, with a number of contact-induced phenomena being observed. The paper concludes that language contact has an important role to play in the evolution of Russian in Daghestan and more broadly. This research provides important insights into the nature of language contact and its effects on syntactic variation.
The text is very repetitive. ChaptGPT is like a student during an exam, who only managed to write off one or two sentences, and now tries to fill in the remaining space with different versions of the same. This impression is supported by Sentence-BERT, or S-BERT, which I used in my post to compare some speeches of Putin and Hitler. It can compute semantic similarity scores between pairs of sentences. The figure below shows the mean similarity scores for all pairs of sentences in the texts written by humans and ChatGPT with different ‘temperatures’. The higher the score, the more similar the sentences are in a text. The plots indicate that humans are the least repetitive because they tend to produce texts with the lowest similarity between the sentences. Notably, humans are followed by ChatGPT with the temperature value 1.2.
To sum up, the processing costs of texts written by ChatGPT seem to be similar to ones written by humans. But the content is more repetitive and therefore less valuable. Of course, my test was unfair: ChatGPT had much less information about the topics than the humans who wrote the texts. Also, we can hope that clever prompts can help AI improve. Even then, I think the best strategy would be to admit honestly the lack of knowledge, rather than keep saying the same. ChatGPT cheerfully follows the motto, “Fake it till you make it”.
So, why is ChatGPT a celebrity? Perhaps because it is so superficially smooth and at the same time so unoriginal. It’s all perfect form and little content. But isn’t that the secret of the Kardashians’ success, too?
Text Grinder 