Anthropic released Claude 3 at the start of this month, in three models: Haiku, Sonnet and Opus. Opus is the largest, and it is the first time in some while that an alternative to ChatGPT has stood out to us for the quality of its writing.
This changes a question we used to answer immediately. Clients often ask us which AI tool to pay for. We have been using OpenAI's models since GPT-3, and we wrote a beginner's guide to ChatGPT last year, so for a long time the answer was simply "ChatGPT." Today, the answer depends on several factors, and a useful recommendation has to explain what those factors are.
Which AI assistants are worth paying for this spring?
Three assistants stand out. Here is an overview of the situation this spring, since it changes every few weeks.
ChatGPT remains the default choice. Most people who use AI use it. The paid Plus plan provides GPT-4, image generation, file analysis, web browsing and custom GPTs, and since January a GPT Store has offered a large selection of them. Some are of good quality; many are little more than a prompt with a logo. ChatGPT is the most versatile of the three, much like a Swiss army knife.
Gemini is Google's assistant, and it has carried that name only since February. Bard has been retired, and a paid tier, Gemini Advanced, is included in a Google One plan. Its main advantage is its integration with Gmail, Docs and Search. Google also presented Gemini 1.5 Pro, which can process a very large amount of text at once. The past month has been difficult for the product, however, as Google paused image generation of people after several highly public errors.
Claude 3 is the notable newcomer. Opus writes in a way that sounds less like a brochure, and it handles long documents well. It is also the most willing of the three to say "I'm not sure," a quality we value more than any benchmark score.
All three paid plans cost about the same, roughly twenty US dollars a month, so price will not determine the choice.
How should you compare AI assistants?
On your own tasks rather than on benchmarks. Every launch comes with a chart showing one model outperforming another on a particular exam, coding test or math set. The charts are accurate, but they measure tasks a small business owner never performs.
The effective method is simple. Take five real tasks you have this month and give exactly the same prompt to all three assistants. Ours look like this:
- Rewrite an English web page in Quebec French and keep the tone.
- Summarize a long PDF and identify what's missing from it.
- Fix a small piece of PHP in a WordPress theme.
- Draft a reply to a difficult email that doesn't sound like a robot wrote it.
- Take a messy list of numbers and find the mistake.
Then read the answers side by side, carefully rather than at a glance.
For businesses in Quebec, the translation task is usually the most revealing. Last year we conducted a short consulting session with PADI Americas on AI translation tools, and our observations from that work still apply. The models perform well in French, but not always in Quebec French. Some output reads as though it came from a Paris agency. Some uses "courriel" correctly, then introduces "e-mail" two lines later. These issues only become apparent through careful reading by someone who knows how the text should sound.
Which assistant do we recommend for which task this month?
For long-form writing and careful editing, we currently favour Claude. For tasks that require a built-in tool, such as a chart, a quick analysis of a spreadsheet or an image, ChatGPT offers more features. Gemini is a logical choice if your work is already centred on Google's ecosystem.
This assessment reflects our own tasks. Your results may differ, which is precisely why we recommend conducting the test.
Is it worth using more than one model?
Yes. Paying for two subscriptions may seem wasteful, but in our view the cost is low compared with the time saved.
The key is cross-checking. Draft with one model, paste the draft into another, and ask it to identify errors. The models make different mistakes. When two models disagree on a fact, verify the source yourself. When they agree, a check remains advisable if the matter is important; agreement simply offers additional confidence.
Three subscriptions are not necessary to begin. Use the free version of each, run your five tasks, and pay for the one that performs best on the work you do most often. Repeat the exercise in six months, and schedule it in your calendar. Six months is a long period in this field, and the current leader will not necessarily remain ahead in the fall.
What should you check in AI privacy settings before entering client data?
This step is frequently overlooked, and we consider it essential.
The three companies do not apply the same defaults. On ChatGPT, conversations from a personal account can be used to train the models unless this option is disabled in the data controls. Google states that some Gemini conversations can be read by human reviewers and advises against entering confidential information. Anthropic states that it does not train on consumer Claude conversations by default.
These are the rules in effect this month. Terms are updated regularly, and users are not always formally notified when they change. We therefore recommend reviewing the settings of each tool you use, and reviewing them again whenever the company announces a change.
A client's personal information should not be entered into a consumer chat application for convenience. In Quebec, with Law 25 already in force, this is a significant issue. If you handle client files every day, a business or team plan with stricter terms is a worthwhile investment.
How should a small business choose an assistant this week?
We recommend setting aside launch posts and YouTube rankings. Anthropic's own Claude 3 family announcement is worth reading to understand the company's claims, while noting how much of it consists of benchmark charts. That content is marketing, and it should be read as such.
Then set aside an afternoon with the five tasks, the three free versions and a pot of coffee. That afternoon provides more insight than a month of reading on the subject. For a team that wants to learn together rather than have each person experiment alone, hands-on AI training built around your real work is highly effective.
The current trend does not suggest that any of these three assistants will "win" and displace the others. They continue to overtake one another in turn. This requires ongoing attention, and it also benefits the businesses paying for them.