By commenting, you agree to the
A team of technology experts issued a global call on Monday seeking the toughest questions to pose to artificial intelligence systems, which increasingly have handled popular benchmark tests like child’s play. Dubbed « », the project seeks to determine when expert-level AI has arrived. It aims to stay relevant even as capabilities advance in future years, according to the organizers, a non-profit called the (CAIS) and the startup . The call comes days after the maker of previewed a new model, known as OpenAI o1, which « destroyed the most popular reasoning benchmarks », said , executive director of CAIS and an advisor to Elon Musk’s xAI startup. Hendrycks co-authored two 2021 papers that proposed tests of AI systems that are now widely used, one quizzing them on undergraduate-level knowledge of topics like U.S. history, the other probing models’ ability to reason through competition-level math. The undergraduate-style test has more downloads from the online AI hub Hugging Face than any such dataset.
