We take a deep dive into the inner workings of the wildly popular AI chatbot, ChatGPT. If you want to know how its generative AI magic happens, read on.
Google, Wolfram Alpha, and ChatGPT all interact with users via a single line text entry field and provide text results. Google returns search results, a list of web pages and articles that will (hopefully) provide information related to the search queries. Wolfram Alpha generally provides mathematically and data analysis-related answers.
ChatGPT, by contrast, provides a response based on the context and intent behind a user’s question. You can’t, for example, ask Google to write a story or ask Wolfram Alpha to write a code module, but ChatGPT can do these sorts of things.
Fundamentally, Google’s power is the ability to do enormous database lookups and provide a series of matches. Wolfram Alpha’s power is the ability to parse data-related questions and perform calculations based on those questions. ChatGPT’s power is the ability to parse queries and produce fully-fleshed out answers and results based on most of the world’s digitally-accessible text-based information — at least information that existed as of its time of training prior to 2021.
In this article, we’ll look at how ChatGPT can produce those fully-fleshed out answers. We’ll start by looking at the main phases of ChatGPT operation, then cover some of the core AI architecture components that make it all work.
In addition to the sources cited in this article (many of which are the original research papers behind each of the technologies), I used ChatGPT itself to help me create this backgrounder. I asked it a lot of questions. Some answers are paraphrased within the overall context of this discussion.
Let’s use Google as an analogy again. When you ask Google to look up something, you probably know that it doesn’t — at the moment you ask — go out and scour the entire web for answers. Instead, Google searches its database for pages that match that request. Google effectively has two main phases: the spidering and data gathering phase, and the user interaction/lookup phase.
Roughly speaking, ChatGPT works the same way. The data gathering phase is called pre-training, while the user responsiveness phase is called inference. The magic behind generative AI and the reason it’s suddenly exploded is that the way pre-training works has suddenly proven to be enormously scalable.
Generally speaking (because to get into specifics would take volumes), AIs pre-train using two principle approaches: supervised and non-supervised. For most AI projects up until the current crop of generative AI systems like ChatGPT, the supervised approach was used.
Supervised pre-training is a process where a model is trained on a labeled dataset, where each input is associated with a corresponding output.
For example, an AI could be trained on a dataset of customer service conversations, where the user’s questions and complaints are labeled with the appropriate responses from the customer service representative. To train the AI, questions like «How can I reset my password?» would be provided as user input, and answers like «You can reset your password by visiting the account settings page on our website and following the prompts.» would be provided as output.
In a supervised training approach, the overall model is trained to learn a mapping function that can map inputs to outputs accurately. This process is often used in supervised learning tasks, such as classification, regression, and sequence labeling.
As you might imagine, there are limits to how this can scale. Human trainers would have to go pretty far in anticipating all the inputs and outputs. Training could take a very long time and be limited in subject matter expertise.
But as we’ve come to know, ChatGPT has very few limits in subject matter expertise. You can ask it to write a resume for the character Chief Miles O’Brien from Star Trek, have it explain quantum physics, write a piece of code, write a short piece of fiction, and compare the governing styles of former presidents of the United States.
It would be impossible to anticipate all the questions that would ever be asked, so there really is no way that ChatGPT could have been trained with a supervised model. Instead, ChatGPT uses non-supervised pre-training — and this is the game changer.
Non-supervised pre-training is the process by which a model is trained on data where no specific output is associated with each input. Instead, the model is trained to learn the underlying structure and patterns in the input data without any specific task in mind. This process is often used in unsupervised learning tasks, such as clustering, anomaly detection, and dimensionality reduction. In the context of language modeling, non-supervised pre-training can be used to train a model to understand the syntax and semantics of natural language, so that it can generate coherent and meaningful text in a conversational context.
It’s here where ChatGPT’s apparently limitless knowledge becomes possible. Because the developers don’t need to know the outputs that come from the inputs, all they have to do is dump more and more information into the ChatGPT pre-training mechanism, which is called transformer-base language modeling.
