What’s cooking at WWDC 2021 on AR/VR

Right, when the rumors of the apple glasses started surfacing we could see how the ecosystem is getting ready for this new medium. Adding to the observations from last year this year we could see it getting stronger.

Is the apple ecosystem getting ready for the glass?My observations at the WWDC20siddarth.design

Object Capture

Object Capture was one of the significant announcements in the WWDC 2021. One of the main challenges in AR is the content creation pipeline which takes more resources and time. By using Object Capture we can generate the USDZ files optimized for AR Quick Look.

Capturing Photographs for RealityKit Object Capture

ARKit 5

ARKit 5 brings features improvements to Motion Tracking and support for Face Tracking in the Ultra-Wide camera on iPad Pro.

RealityKit 2

RealityKit leverages information provided by the ARKit framework to seamlessly integrate virtual objects into the real world.

Widget Suggestions API

Contextual Widgets

Widgets will play an important role in the augmented space. The new Widget Suggestions API with Smart Rotate will help to create experiences that give timely information or learn common patterns.

Use the camera for keyboard input in your app

Though this tech is not something new but how apple is contextualizing it with in the ecosystem makes me wonder if this would work when the AR glasses comes out. One of the main problem in AR is text input.

Content Recognition

Improvements to Spatial Audio

New features like adaptive spatial sounds making this new dimension of audio work in all network connectivity.

Spatial Audio

Nearby Interaction with third-party accessories

Expansion of Nearby Interaction in iOS 15 and watchOS 8 and later, U1-enabled devices can interact with third-party accessories would be a game changer. Imagine controlling accessories in the augmented control.

Nearby

Built-in sound classification in SoundAnalysis

One of the main concerns in AR is the reliance of the cameras to understand the environment. Sound Classification and the Sound Analysis framework, might come in handy to understand the context and to trigger realvant actions. That too the built-in part of it addresses the privacy concerns. Already Apple LiDAR sensors are said to be game changers in this area. This advances in sound detection might add value to it.

Sound Classification

Core Haptics

Haptics are very important in a multisensory experiences and also in a multimodal interactions they play a significant role. The new developer options to customize and design haptic experiences would come in handy in future.

Apple Haptic and Audio Pattern (AHAP)

Few more updates which are relatable,

Updates to Vision framework to detect people, faces, and poses.

SharePlay experiences for social experiences.

Will keep this post updated as I come across other cool updates which I think realtes to the future of AR.