Having found myself using ChatGPT daily, mostly to help me write code, I recently decided to upgrade to the Plus service and try GPT-4. I was curious about what improvements were incorporated into the new model, and while OpenAI’s GPT-4 Technical Report describes GPT-4’s capabilities, limitations, and benchmarks, it does not go into detail about how the model was improved. So I asked ChatGPT-4 to describe its improvements over the original GPT-3.5 model, and it provided me a list of advancements, including:
- An improved model architecture with more parameters, layers, and attention heads
- More extensive and diverse training data
- Fine-tuning advancements, such as prompt engineering, and reinforcement learning from human feedback
However, when I attempted to dig deeper into these topics, I ran into some roadblocks.
ChatGPT-4’s Limited Knowledge
When I asked it to provide details on the larger number of parameters, layers, and attention heads, it stated it could not provide specific numbers and only explain in general terms the significance of the improvements. Then when I asked it how the training data used for GPT-4 is more extensive and diverse than the one used for GPT-3.5, it similarly answered that it doesn’t have exact details on the datasets used for its training. But more interestingly, at that point, it simply stated that its training data could be more extensive and diverse compared to GPT-3.5 (in essence backpedaling its previous response).
Not giving up, I then tried to get more details on the improvements in prompt engineering, as this is a very interesting area of conversational AI because it allows the user to exert control over the response by structuring the question in a specific way, But ChatGPT-4 was equally unhelpful and its response was again qualified with the statement that it does not have exact details on the prompt engineering improvements.
Enter ChatGPT-3.5
At this point, I was a little frustrated and decided to try the GPT-3.5 model with the same questions. To my surprise, the GPT-3.5 model was aware of the GPT-4 model and more eagerly answered my questions. While it still could not provide details on the model architecture improvements, it volunteered a lot more information and was able to provide much more detailed information on the training data and the fine-tuning advancements. As you can see below, GPT-3.5 produced a very informative response for the training data:
The data set used to train ChatGPT 4 is larger and more diverse than the one used to train ChatGPT 3. This data set is called the Pile (the "Pile of Interesting Language Examples") and it consists of over 800GB of text data from a wide range of sources, including books, articles, websites, and scientific papers.
The Pile is larger and more diverse than the data set used to train ChatGPT 3, which was primarily sourced from the internet. The Pile was created specifically to include a diverse range of language examples, including rare and unusual language use cases.
To train ChatGPT 4, OpenAI used a combination of supervised and unsupervised learning techniques. The model was first pre-trained on a large corpus of text using unsupervised learning, and then fine-tuned on specific tasks using supervised learning.
The improved feedback system in ChatGPT 4 is due in part to the fact that the model was trained on a diverse range of tasks, including question answering and conversation generation. This means that the model is better equipped to understand the context and intent of user interactions, and can generate more accurate and relevant responses as a result.
Additionally, ChatGPT 4 uses a technique called "prompt engineering" to generate more relevant responses. This involves providing the model with a specific prompt or starting point for generating a response, which allows it to focus on the most relevant information and generate more coherent and accurate responses.
Overall, the larger and more diverse data set used to train ChatGPT 4, combined with its improved feedback system and prompt engineering technique, has led to a more advanced language model that is better equipped to understand and generate human-like language.And this was its response to the fine-tuning advancements:
ChatGPT 4 takes prompt engineering to a new level by introducing a technique called "prompt tuning". Prompt tuning involves adjusting the prompt in real-time based on user input to generate more accurate and relevant responses.
Specifically, ChatGPT 4 uses a technique called "adaptive prompt tuning" that allows it to fine-tune the prompt on the fly based on the input it receives from the user. This means that ChatGPT 4 can adjust the prompt in real-time to account for new information or changes in context, leading to more accurate and relevant responses.
For example, if a user asks ChatGPT 4 a question and then provides additional information or context, the model can adjust the prompt based on this new information to generate a more accurate response.
Overall, while both ChatGPT 3 and 4 support prompt engineering, ChatGPT 4 takes this technique to a new level by introducing adaptive prompt tuning. This allows the model to generate more accurate and relevant responses in real-time, making it a more effective and versatile language model.
ChatGPT-3.5 Hallucinations
While I welcomed the details provided by the GPT-3.5 model, I suspect some of the information provided by the GPT-3.5 model may be hallucinations, meaning responses that are nonsensical or untruthful. For example, prompt tuning is a technique used to fine-tune a pre-trained language model to better respond to specific prompts. Instead of training the model on a large dataset, prompt tuning focuses on a smaller set of carefully chosen prompts, often paired with the desired response, to guide the model’s behavior when presented with similar prompts.
When I asked ChatGPT-4 to explain how “adaptive prompt tuning” is different from prompt tuning, it explained that in adaptive prompt tuning, the prompts are treated as learnable parameters, just like the weights and biases in the model, which allows the prompts to adapt to the specific task or domain, effectively “learning” the best way to elicit the desired response from the model. While ChatGPT-3.5 states that “adaptive prompt tuning” fine-tunes the prompt on the fly based on the input it receives from the user, ChatGPT-4 counters that the adaptive prompt tuning process does not involve on-the-fly fine-tuning of the prompts when the model is generating responses based on user input.
Exercise Caution
OpenAI has said that GPT-4 is 82% less likely to respond to requests for disallowed content and 40% more likely to produce factual responses than GPT-3.5. it is therefore not surprising that GPT-4 was guarded in its responses while GPT-3.5 freely provided detailed information. Since OpenAI has not made public the specific improvements to the model architecture, or any details on the training process or data for GPT-4, other than to say that it was trained using publicly available data and data they’ve licensed, it is possible that much of the information GPT-3.5 provided is inaccurate. The stark difference in the responses from these models is fascinating and serves as a valuable reminder to exercise caution when relying on their responses.