Apple Intelligence is the product of more than a year’s worth of tireless testing. Here’s what Apple engineers used to ensure the quality of their AI software.
Apple Intelligence is the product of more than a year’s worth of tireless testing. Here’s what Apple engineers used to ensure the quality of their AI software.
For Apple, 2024 was undoubtedly the year of artificial intelligence. The company has long been working on machine learning features, with its most recent operating systems ushering in an entirely new set of AI-powered enhancements. They are known collectively under the moniker of Apple Intelligence.
While the generative AI tools themselves were announced in June, at WWDC 2024, only a handful of them made their public debut with the first developer betas of iOS 18.1 and macOS 15.1. Since then, Apple has rolled out more and more of the AI-powered enhancements with subsequent beta releases.
At the time of writing, the iOS 18.1 and macOS 15.1 updates are nearing the end of beta testing, while the first developer beta of iOS 18.2 has only just arrived. Months after the big announcement, some Apple Intelligence features are still only available on beta versions of Apple’s operating systems.
According to people who spoke with AppleInsider and accurately revealed many Apple Intelligence features months ahead of launch, the company spent a year working on its in-house generative AI tools before they were finally released to the general public.
During development, Apple tried to keep the full scale of its AI endeavors a secret. Individual AI projects received their own codenames, as was the case with the email categorization feature, known as Project BlackPearl.
Apple Intelligence as a whole, however, was known by the codename Greymatter — an unmistakable reference to a type of tissue found in the human brain. Some of Apple’s internal test applications also had names that concealed their overall purpose.
The internal tools Apple used for testing before Apple Intelligence became public
During the development of Apple Intelligence, Apple used at least two dedicated test applications and environments to test its AI software.
The two apps in question are known as 1UP, a reference to the ever-popular Super Mario series by Nintendo, and Smart Replies Tester. The name of the latter is self-explanatory, given that AI-powered Smart Replies have since made their way into release versions of Apple’s operating systems, in the Mail and Messages applications.
We were told that internal distributions of iOS 18.0 and macOS 15.0 Sequoia featured many of the underlying Apple Intelligence frameworks used in the publicly available betas of iOS 18.1 and macOS 15.1.
The frameworks were necessary for testing and were included alongside the standard development and configuration utilities found in Apple’s internal-use operating systems.
Different AI-related features could be toggled through feature flags, with the use of the Livability application. 1UP and Smart Replies Tester, the two known AI applications, were used by Apple’s engineers to test the different aspects and use cases of Apple Intelligence.
1UP — Text-generation testing with AI models
Found even in the earliest internal-use builds of iOS 18 and macOS Sequoia, the 1UP application was used for testing text-related generative AI features. The application itself featured a variety of different test options and parameters, which could be adjusted as needed.
People familiar with the application have told AppleInsider that it contains direct references to Apple’s long-rumored in-house LLM or large language model, known as Ajax, which can function on-device.
The 1UP app features multiple test options, organized into different sections.
