A team of researchers at Edinburgh University tested some top multimodal large language models to see how well they could answer questions based on images of clocks.
Facepalm: Generative AI tools are able to perform the sorts of tasks that once seemed the stuff of sci-fi, but most of them still struggle with many basic skills, including reading analog clocks and calendars. A new study has found that overall, AI systems read clock faces correctly less than a quarter of the time.
A team of researchers at Edinburgh University tested some top multimodal large language models to see how well they could answer questions based on images of clocks and calendars.
The systems being tested were Google DeepMind’s Gemini 2.0, Anthropic’s Claude 3.5 Sonnet, Meta’s Llama 3.2-11B-Vision-Instruct, Alibaba’s Qwen2-VL7B-Instruct, ModelBest’s MiniCPM-V-2.6, and OpenAI’s GPT-4o and GPT-o1.
Various types of clocks appeared in the images: some with Roman numerals, those with and without seconds hands, different colored dials, etc.
