On May 28, a paper with a straightforward title was posted to arXiv: "Language Models are Few-Shot Learners". Its thirty-one authors from OpenAI describe GPT-3, a language model with 175 billion parameters. On June 11, OpenAI opened an API to it as a private beta, with a waiting list and selective admission.
We have followed this field since GPT-2 last year, when we wrote about machines that write, and we have requested access. In the meantime, we have studied the paper closely and followed what the first testers are publishing.
What is the GPT-3 API?
The term "AI" often suggests a chatbot, but this is different. OpenAI describes the API as "text in, text out." The user provides the beginning of a text and the model continues it. That is the entire interface.
To obtain an answer to a question, the user does not ask directly. Instead, the user writes the beginning of a document in which that answer would naturally appear: a Q and A page with the question and an empty answer, the opening of an interview, or the first line of an email. The model then writes the rest, in whatever tone has been established.
Described this way, the process may seem awkward. Based on the samples being shared, it is not. The output reads like the work of someone who knows the subject, paragraph after paragraph. After years of chatbots that failed at the second question, this represents a significant advance.
How convincing is the text?
The most striking figure in the paper comes from a test on short news articles. The researchers had GPT-3 write articles from a title and subtitle, mixed them with real articles, and asked people to identify those written by the machine. With the largest model, participants answered correctly about 52 percent of the time, compared with 50 percent by chance.
In other words, most readers cannot reliably distinguish the two, even in a controlled test where they know they are expected to look for differences.
For professional writers, and for those who hire them, this is a significant finding. For small business owners who dislike writing, it may appear to be good news. In our assessment, it is both.
Why is fluent text not necessarily true?
The concerns emerged at the same time as the initial impressions, and they stem from the same capability.
GPT-3 does not know things as a person does. It has learned, from an enormous volume of text, what text typically looks like. When asked about something real, it produces something that resembles a correct answer. Sometimes it is correct. In other cases, we expect it to produce a fluent fabrication: a statistic that does not exist, a quotation no one said, or a book the author never wrote.
Errors in themselves are not the issue, since people make mistakes regularly. The issue is that incorrect answers will look exactly like correct ones: the same tone, the same confidence, the same polished grammar, with no hesitation or qualification. The text alone will not reveal which parts are accurate.
The authors acknowledge this openly. The paper's limitations section states that samples can repeat themselves, lose coherence over long passages and contradict themselves. A full section addresses misuse, including misinformation and spam. When the developers of a tool devote several pages to these concerns, they warrant serious attention.
Fluent is not the same as true. This is the central lesson.
Why do prompts determine the result?
The paper's title points to the other major lesson: few-shot learning. When GPT-3 receives two or three examples of the desired output before the request, the result improves considerably. Given three good product descriptions and asked for a fourth, it produces something close to the examples. Given a single vague line, it produces something average.
This is consistent with established practice in briefing a writer or a designer: the brief determines the result. Audience, tone, length and an example of good work all matter. A machine that learns from examples makes this even more true, because it has no other guidance.
We expect this to be where small businesses either gain value or encounter problems. The tool will reward those who provide their own facts, knowledge of their customers and their own voice. It will penalize those who type "write me a blog post about plumbing" and publish the result unchanged. Business owners will need guidance on using these tools properly. Practical AI training for people who are not developers does not yet exist in a useful form, and we expect the need for it to arise sooner than anticipated.
What should GPT-3 be used for, and what should it not be used for?
If access is granted, our intended approach is as follows.
Drafts are an appropriate use: moving past the blank page, generating five different openings for an article in order to select the one closest to the intended meaning, rephrasing a paragraph that does not read well, or brainstorming names. These are the areas where we expect it to perform best.
Facts are not: figures, dates, and anything legal, medical, or where an error could harm someone. These should be written by a person, or every line should be verified.
At present, GPT-3 is a developer tool behind a waiting list, and most business owners will not use it this year. However, we expect it to be integrated into everyday tools sooner than most organizations anticipate. When that happens, one practice is essential: review everything it produces as if it had been written by a very confident new employee. The work is probably good, but possibly wrong, and the business, not the tool, is accountable for it.