On October 11, a team at Google posted a paper to arXiv introducing BERT, which stands for Bidirectional Encoder Representations from Transformers.
Few people outside the machine learning field have noticed it so far. Our own activity is in dive tourism rather than machine learning. Nevertheless, this development is significant for anyone who writes for the web, which includes most business owners.
What does BERT actually do?
Most language models read text in one direction: left to right, predicting the next word. This approach works, but it means the model only knows what came before a word when it tries to determine that word's meaning.
BERT reads in both directions. During training, the researchers hide random words in sentences and have the model predict them using all the surrounding context, before and after. Trained this way on English Wikipedia and a large collection of books, the model becomes very effective at interpreting context.
Consider the word "bank." In "I sat on the bank and watched the river," the word "river" at the end indicates which kind of bank is meant. A model that reads only left to right has not yet seen "river" at that point. BERT has.
The results reported in the paper are what drew attention. The researchers state that BERT set new best scores on eleven different language tasks, including question answering, in which the model reads a passage and must locate the answer in it. Researchers in the field have responded with considerable and unusually public enthusiasm.
For the primary source, the BERT paper by Devlin and colleagues is available on arXiv. It is technical, but the abstract and introduction are accessible to non-specialists.
Where does BERT come from?
BERT is built on the Transformer, introduced in another Google paper last year titled "Attention Is All You Need." It is not an isolated result. Earlier this year, the Allen Institute published ELMo, which also assigns words different meanings depending on context. In June, OpenAI released a model it calls GPT, also built on Transformers, which reads left to right and is trained to predict the next word.
These are three major results in a single year, all pointing to the same idea: a large model trained on very large amounts of ordinary text learns enough about language to be useful for many tasks it was not specifically designed for. This is new. Until recently, a separate system had to be built for each task.
The long-term direction remains uncertain, but our assessment is that its impact will be substantial. This spring, Google demonstrated Duplex booking a hair appointment by phone in a voice that sounded human, and Gmail now suggests entire phrases as users type. These applications are early and still imperfect, but the underlying research is advancing quickly.
What could this mean for Google Search?
Google already uses machine learning in search. RankBrain has been part of it since 2015, helping with queries Google has never encountered before. However, search still relies heavily on the main words in a query and gives little weight to the small ones.
Consider a search such as "dive courses for kids." The main words are "dive" and "courses," while "for kids" is treated as a minor addition. The page that ranks may be a general course page that mentions children once, in the fine print.
A model that reads context as BERT does could understand that "for kids" is the central point of the query: minimum ages, programs for young divers, whether parents can accompany them. Similarly, in "can you pick up a prescription for someone else at the pharmacy," the phrase "for someone else" is the actual question. Keyword matching cannot reliably detect this.
Google has not announced that BERT will be integrated into search. We nevertheless expect it to be, in some form, and possibly soon. The research is Google's own, and understanding what people mean is the core of its business.
Why should businesses write naturally?
Much of SEO still consists of filling pages with exact phrases, such as "Best dive shop Bali dive shop Nusa Lembongan dive shop." Pages like these are familiar, and some of them currently rank well.
We do not expect this approach to remain effective. When a search engine understands context, a page that reads as a genuine answer to a real question should outperform a page that repeats a phrase. Google has been advising site owners to "write for people" for years. With models of this kind, it can begin to verify whether they do.
In practice, this means:
- Answer the actual question people ask, using the words they use.
- Include the small words. "For beginners," "near Sanur," "without a car": these details are what people search for.
- Write one clear page per real topic instead of five thin pages targeting variations of a keyword.
This advice is not new, but it is about to become much harder to ignore. If writing clearly for real readers is a challenge for your business, professional web copywriting addresses exactly that need. The writing does not need to be elaborate; it needs to answer the question.
For about twenty years, people have learned to write in unnatural ways for machines, and now the machines are learning to read. The most effective strategy may once again be to write naturally, for people.