This fall, we launched a small project called ChatGPT RPG. The name came later; the project began with a persistent question.
Can a language model run a game for a single player?
Two weeks ago, OpenAI released ChatGPT, and a large share of internet users are now attempting to make it tell stories. This is therefore an appropriate time to document our observations so far, before public discussion intensifies further.
Why test a language model as a game master?
Because it could fill a gap in solo play. Solo role-playing already exists and is growing. Players play alone using oracle tables, dice and their own imagination. The approach works, but the player is both participant and referee, which makes genuine surprise difficult.
A language model appears to be the missing element: a tool that can describe a scene, play the characters the player encounters and respond to any action, in plain language. AI Dungeon, running on GPT-2, demonstrated in 2019 that there is demand for this. Our objective was to determine how far newer models can go when treated less as a novelty and more as a game master with a defined role.
Our focus differs somewhat from the purpose of AI Dungeon, which is open by design: any input produces any outcome. This is entertaining for an evening, but a real game requires limits. Actions must have a cost. Characters who die must remain dead. The world must offer resistance. The question is whether a model can adhere to a set of rules, or whether it inevitably reverts to accepting whatever the player types.
What have the first experiments shown?
We began with GPT-3 in OpenAI's playground and have used ChatGPT since its release. Three patterns recur consistently, regardless of the model used.
It improvises well, then forgets
The quality of the improvisation is striking. Given a situation, the model produces atmosphere, a character with a distinct voice and an unexpected development. It can portray an irritable innkeeper and a nervous radio operator in the same session while keeping them distinct. For a solo player, this capability alone is highly valuable.
It then forgets. These models can retain only a limited amount of text at a time, approximately 4,000 tokens for the current GPT-3 models, or a few pages at most. Anything older falls outside that window. A character who died an hour earlier reappears in the scene. A lost sword is back in the player's hand. The village changes its name.
The most accurate description is that of a brilliant but forgetful game master: the most creative partner one could play with, yet unable to remember the previous week.
The rules must be written out for the AI
This point took longer to understand. A human game master retains the rules. A language model retains nothing from one request to the next, except what it is given.
Everything it requires must therefore be written down and provided again: the rules of the game, the character's current state, a summary of past events reduced to the essentials, the tone, and what it must never do, such as deciding the player character's actions, which it frequently attempts.
This is proving to be the majority of the work. It is less a matter of writing a clever prompt than of writing a clear rulebook the model can follow consistently. The task is closer to game design than to programming, which aligns well with our expertise.
The player must still direct the story
Left to itself, the model drifts toward the predictable: the expected twist, the happy ending, lengthy dramatic speeches. It also tends to agree with everything and yields under the slightest pressure.
The successful sessions are those in which the player sets goals, makes difficult choices and pushes back when the story loses tension. The AI is an excellent partner, but not the author.
What does this project reveal about AI in general?
A game project has proven unexpectedly instructive about the tools themselves.
A model knows only what it has been given. For it to remain consistent, consistency must be defined in writing. Memory will likely improve: context windows will grow, and methods for retaining notes between sessions will be developed. In our assessment, the underlying principle will remain the same.
The same applies to rules. Whether the model serves as a game master or answers questions for a business, the quality of its output depends mainly on how clearly its permitted and prohibited actions have been defined. Most of the screenshots circulating this week come from users who skipped this step and are surprised by the results.
A person must also remain in control. The model can assist, suggest, draft and participate, but it should not be the one determining direction.
How does a game master relate to a business assistant?
The challenge is identical, even though it may seem far removed from running a local business. Consider a chatbot on a campground's website. It must remember the site's rules, prices and dates, and must not invent a swimming pool that does not exist. Its rulebook must be written out, exactly as for the game master.
In our view, this is where most of the real work in AI chatbots and assistants will lie. The model is becoming the simpler component. Documenting what it needs to know, clearly and accurately, is the core of the work.
In the meantime, testing of the forgetful game master continues. It regularly brings deceased characters back into play, which keeps each session unpredictable.