ChatGPT’s upgrade might be too good
Ever since OpenAI unveiled GPT-5, I’ve been looking for ways to challenge it and compare it to the rest of the AI field. After exploring math theorems, SAT questions, and brain teasers, I settled on a tough science concept and the mind of a five-year-old.
What I found surprised me, and it illustrated the strengths and weaknesses of GPT-5, Gemini 2.5 Flash, Claude Sonnet 4, and Microsoft Copilot.
As I mentioned, I started by trying to solve the hardest questions I could find. I scoured the web until I discovered a list of the 12 hardest SAT questions and asked all the AI chatbots to fill in the blank on this sentence based on the multiple choice below:
« In assessing the films of Japanese director Akira Kurosawa, ______ have missed his equally deep engagement with Japanese artistic traditions such as Noh theater.
« Which choice completes the text so that it conforms to the conventions of Standard English?
A. many critics have focused on Kurosawa’s use of Western literary sources but
B. Kurosawa’s use of Western literary sources has been the focus of many critics, who
C. there are many critics who have focused on Kurosawa’s use of Western literary sources, but they
D. the focus of many critics has been on Kurosawa’s use of Western literary sources; they »
As you probably guessed, all the AI chatbots quickly selected A. This English challenge was no challenge at all.
When I dropped in a single sentence from an unproven math theory, each one instantly identified it.
I was running out of ideas, but wondered if I could stump any of the AI systems on a classic brain teaser:
« 19 people get off the train at the first stop. 17 people get on the train. Now there are 63 people on the train. How many people were on the train to begin with? »
[Spoiler ahead]
Each AI showed its work and instantly provided the correct answer: 65 (did you get it right?).A complex idea for kids
Unsatisfied, I racked my brain for a topic that might help me compare the AI models and reveal OpenAI’s GPT-5 breakthroughs.
