By commenting, you agree to the
A team of technology experts issued a global call on Monday seeking the toughest questions to pose to artificial intelligence systems, which increasingly have handled popular benchmark tests like child’s play. Dubbed «», the project seeks to determine when expert-level AI has arrived. It aims to stay relevant even as capabilities advance in future years, according to the organizers, a non-profit called the (CAIS) and the startup . The call comes days after the maker of previewed a new model, known as OpenAI o1, which «destroyed the most popular reasoning benchmarks», said , executive director of CAIS and an advisor to Elon Musk’s xAI startup. Hendrycks co-authored two 2021 papers that proposed tests of AI systems that are now widely used, one quizzing them on undergraduate-level knowledge of topics like U.S. history, the other probing models’ ability to reason through competition-level math. The undergraduate-style test has more downloads from the online AI hub Hugging Face than any such dataset.
