Now the company has announced it, I thought it would be useful to put together everything I know about Apple’s AR approach so far.
“We are high on AR for the long run, we think there’s great things for customers and a great commercial opportunity.” Apple CEO, Tim Cook, July 2016 .
As I expected, Apple has introduced a powerful partnership and platform play to kick-start AR experience creation and distribution across its systems.
The company shared many of the details of its approach at WWDC 2017. This will have a big impact across consumer, creative, and enterprise markets so I thought it would be useful to put together what we know so far.
Apple has leveraged multiple technologies to create an AR ecosystem. When it comes to mobile devices, Apple has introduced ARKit, its new developer framework for the creation of engaging AR experiences.
The idea here is that consumers will use the cameras, processors and motion sensors already packed inside their iOS devices to access AR solutions. Your iPad or iPhone effectively becomes a window to an augmented world.
“By blending digital objects and information with the environment around you, ARKit takes apps beyond the screen, freeing them to interact with the real world in entirely new ways, ” Apple said.
Apple’s developer pages already provide a rich quantity of information around ARKit. They confirm (for example) , that ARKit uses Visual Inertial Odometry (VIO) to accurately track the world around it using camera sensor and CoreMotion data. As I quietly predicted in May 2015,Apple has been able to build its implementation around world leading technologies it quietly acquired with Mataio.
Apple says ARKit provides:
Developers can build AR solutions using Apple’s state-of-the-art computer vision technologies.
Apple specifically suggests they will use ARKit to build games, immersive shopping experiences, for industrial design and more.
Here are a few examples of currently available VR solutions, but there is also big potential for AR deployments in logistics and distribution, emergency response, military, education, medical, and beyond.
One of the more critical supporting technologies for Apple’s AR play is “Metal for VR” within macOS High Sierra. A component of the heavily improved Metal 2 system, Metal for VR offers a range of VR-focused features, including a VR-optimized display pipeline, viewport arrays and stereoscopic visualization frame debuggers.
Metal 2 also builds in support for external graphics processing units (GPUs) . This means you can connect your MacBook or iMac to an external GPU system in order to make use of more powerful graphics processing, which is ideal for AR developers.
“Apps that use Metal, OpenCL, and OpenGL can now take advantage of the increased performance that external graphics processors can bring, ” says Apple.
Don’ t neglect that early versions of the Oculus Rift dev kit ran on Macs, but that effort failed as Macs couldn’ t deliver the graphics support required. There was no news on Oculus at WWDC, but other partners were named.
I’ ve said this before and I’ ll say it again: ignore what you think you know and accept that Apple does partnerships. Its AR partners include Valve in order to provide SteamVR support on Macs. It is also working with Unity and Epic Games’ Unreal engine.
The Valve support means Apple has bought the HTC Vive into the fold – the company demonstrated some of the AR features using an HTV Vive headset.
“With partners like Apple, Google, Intel and dozens more, we will continue to make Vive the best VR platform for consumers, developers and enterprises alike, ” s aid Vive .
It is in development, but right now Unity promises its plugin lets developers use ARKit features like world-tracking, live video rendering, plane estimation and updates, hit-testing API, ambient light estimation, and raw point cloud data.
“Radeon Pro 500 series graphics is recommended for 3D VR content, ” said Unity .
Much more detail for developers here .
SteamVR for the Mac is now available in beta. It provides the same 360-degree, room-scale tracking as the Windows and Linux variants.
“On the development side, we have worked closely with Epic and Unity to make Mac extensions of content built on those engine technologies as simple as possible. Extension tools for those engines, and others, are available as part of this beta, ” the company said.
Unreal Engine 4’s early access support for ARKit is available now on GitHub. Binary support is coming with the Unreal Engine 4.17 Preview in July followed by launch in early August.
Blink and you’ ll have missed it, but WebVR will be supported in Safari on iOS and macOS. That’s significant as that technology should support Web-based VR experiences. Steam confirms it has been working with Mozilla to help enable WebVR support on Firefox, so macOS-based web developers can start using VR.
Developers running macOS High Sierra can make an immediate start on building VR apps using solutions from Unity and Valve. You can already download Unity’s Experimental macOS VR build here. You will also need Steam VR from Valve to use this, which is available here.
Apple knows the core Mac market.
At one time this market was defined by graphic designers, then musicians, web designers and video creatives joined the fold. The Mac felt the impact of mobile, while the next big tech wave, social, was fairly non-platform specific.
This changes with AR – Apple’s move to deliver high-end Mac kit, such as the utterly droolworthy iMac Pro and (promised) future Mac Pro shows the company’s late realization that in order to create AR content it would need to also deliver an ecosystem which can be used to help create it.
Apple now wants to provide the world’s best platform for AR content creatives, from content creation to consumer use.
With this in mind it’s reasonable to expect a range of additional content creation messages from the company in the weeks ahead, look to iMovie, Final Cut X and discussion about new VR creation workflows around Apple solutions.
We now know why Apple has made such a monumental, multi-year effort to ensure tens of millions of iPhone and iPad users are running devices equipped with seriously powerful processors.
That move to 64-bit everything was never trivial, it was a deliberate plan to ensure the company could deliver complex technologies, such as AR. Any mobile device with an A9 processor running iOS 11 will be able to take advantage of ARKit, which means any iPhone from iPhone 6s or later.
“With iOS 11, we’ re delivering the biggest AR platform in the world, and it’s available today for developers to begin building AR experiences using ARKit for hundreds of millions of iPhone and iPad users, ” said Craig Federighi, Apple’s senior vice president of Software Engineering.
This means developers working to create AR solutions will address a market of millions of potential users – and there will be big rewards available to developers who can create something viral.
Just like UK Labour leader, Jeremy Corbyn, Apple is frequently underestimated by critics and enemies self-condemned to be forgotten by history.
