ChatGPT turns two this weekend. At around the time of its release, in the fall of 2022, we launched a small site called ChatGPT RPG. The concept was straightforward: use a language model as the game master for solo role-playing, giving a single player a story that responds to their choices.
Two years later, the models have changed considerably. ChatGPT now has a memory feature. Claude offers Projects, where reference files can be kept alongside the conversation. Context windows have grown from a few pages to something closer to the length of a book. Some tasks have become much easier, while a few problems remain unresolved.
This subject is somewhat outside our usual topics. It is nonetheless relevant, because nearly everything we have learned from running stories with an AI applies directly to business chatbots.
What does an AI game master do well?
The first strength is improvisation. A good model responds to any input: an unusual decision, an unplanned character, or a plan that makes no sense at all. A human game master might need a moment to think. The model responds immediately.
The second strength is description. A request for the smell of a harbour at dawn or the sound of a jungle at night produces usable, and often good, results. In solo play, no other participant at the table contributes energy, so this quality is particularly valuable.
The third strength is patience. The model never tires, never loses interest in the player's character, and never needs to leave at eleven o'clock.
During the first few sessions, the experience is impressive. The limitations then become apparent.
Why does an AI game master need a memory system?
From the outset, memory was the most significant problem. A model only "remembers" what is present in the current conversation. Two years ago, that window was small. In a long campaign, the beginning of the story was pushed out of the window, and the innkeeper the player had bribed in session two suddenly had no record of the encounter. A lost sword would reappear in the character's hand. These inconsistencies undermined the experience.
The larger windows available today help, but they do not solve the problem. Even when the entire story technically fits, models become imprecise about details located in the middle of a long conversation. Moreover, the built-in memory features are designed to retain preferences, such as a liking for short answers. They are not designed to track forty characters and a dozen open plot threads.
The effective approach is to build the memory outside the model:
- A campaign sheet, kept up to date, listing characters, places, items and open plot threads.
- A short summary at the end of each session, written or at least reviewed by the player.
- The sheet and the most recent summaries pasted in at the start of every session.
This approach requires little technology and works very well. The model tells the story; the notes preserve it.
How do clear rules keep the story fair?
Without guidance, a language model tends to try to please the user. In a game, this is a significant problem. Every risky plan succeeds, every lock opens, and every enemy is slightly slower than the player. After an hour, nothing matters, because nothing can go wrong.
The solution is a set of rules the model must follow, and a method for determining outcomes that does not depend on the model's tendencies. Dice, cards and tables all serve this purpose: a simple system in which the result comes from outside the model, and the AI's role is to narrate it honestly, even when the outcome is unfavourable to the player.
Solo tabletop games established this principle long before language models appeared. The rules determine what happens; the storyteller determines how it feels. When these two roles are kept separate, the game becomes fair again. When they are combined, the result is a very polite system that tells the player they are doing wonderfully.
How should the rules be written?
Rules should be short and specific. When rules are vague, the model fills the gaps in whichever direction appears most agreeable. Short, specific rules with examples consistently outperform long paragraphs of good intentions. This applies to game prompts, and equally to the custom GPTs that people have been building since OpenAI made them available last year. We have reviewed many of these instructions, and most express wishes rather than rules.
Why do players want surprise rather than chaos?
The opposite failure also occurs. When asked to "make it more interesting", an AI may introduce a dragon, a betrayal and an earthquake in the same scene. The result is surprising, but not satisfying.
Good surprise comes from elements prepared earlier: a quiet character who turns out to be important, or a poor decision from session three that has consequences later. This again requires memory, as well as restraint.
The best AI game master, like the best human one, knows when to remain silent: allowing the player time to think and giving a scene room to develop. Not every reply needs a twist.
What does this mean for business chatbots?
These lessons matter to small business owners because a customer service chatbot faces exactly the same three problems.
- Memory. The chatbot needs accurate, current information about your business provided every time, rather than relying on partial recollection. Prices, hours and policies should be kept in one place and supplied to the chatbot.
- Rules. Without clear limits, the chatbot will try to please the customer, for example by promising a refund you do not offer or confirming a date that is not available. It needs written rules, and a handoff to a person when it is uncertain.
- Restraint. A good chatbot answers the question and stops. It does not improvise a policy on a Sunday evening.
Many businesses are currently being offered chatbots. Before purchasing one, we recommend asking the vendor three questions. Where does the chatbot obtain its information? What is it not permitted to say? What happens when it does not know the answer? If the answers are unclear, the chatbot is likely to be unclear as well. A well-designed AI chatbot or assistant has a clean knowledge base, precise rules and a real person available within one click.
What have two years with an AI storyteller shown?
We continue to use this approach. The models are clearly better storytellers than they were in 2022, and the improvement is undeniable. We nevertheless still maintain a campaign sheet and roll the dice ourselves, and we expect to continue doing so, because these solutions reflect sound design.
The site is available here for those interested: ChatGPT RPG. It began as a personal project and has become the source of most of our expertise in guiding the behaviour of an AI.