Apple Spatial Audio – Which device or tool gives you the 3D effect?

Recently, Apple has been on the verge of establishing spatial audio. With Apple Music and the support of Dolby Atmos, the company is taking the lead in terms of commercially available 3D audio products and applications. Also, thanks to Apple, the interest in spatial audio is currently increasing strongly in general. Everyone is acting like they know what it means. But for real: what the heck does it mean for existing Apple hardware and software?

By now, it feels like every Apple product is equipped with spatial audio features. Note on the side: While in English they talk about “Spatial Audio”, in German they use “3D Audio”. So many terms for the same technology? Not quite, because although spatial hearing is used, sound has to be thought of differently depending on the application.

Therefore, in the following an overview should be created, what are the added values for us consumers and content creators? So that one knows again where and how Apple’s spatial audio currently works.

Updates

• 22/12: There have always been rumors of Apple joining into Virtual Reality, Augmented Reality – or the metaverse as some say. I’m not a fan of spreading guesswork, but I added a section for insider news.
• 23/01: The second generation Apple HomePod supports 3D audio with Dolby Atmos for audio and video with both one speaker and two HomePods combined into a stereo pair.
• 23/06: Apple Vision Pro with VisionOS and the HMD itself added. The VR headset from the giant is finally here, learn more about the first spatial computer below
• 23/06: RealityKit and spatial audio capabilities added.

What is Apple Spatial Audio?

You could divide Apple’s spatial audio offering into an application and a product component that are more or less dependent on each other. That is special software and hardware, sometimes a combination. So there will be a lot more to come in the future. For example with the Airpods Pro 2, but what is already available today?

Apple Spatial Audio Apps

By this, I mean all those apps that are supported by Apple products in the 3D audio context. First and foremost is Apple Music, the company’s own music streaming platform. Dolby Atmos has recently been supported here, so the streaming service now offers thousands of songs mixed in this format, in addition to the 75 million ‘normally’ produced in stereo. You can find out why Dolby Atmos music isn’t quite convincing yet, though here.

Of course, Apple spatial audio is not limited to music. In the meantime, a large selection of corresponding content can be found in the video streaming area. I would even say that this is where the strengths of Apple products currently lie. You can read about which streaming apps are supported by Apple in this overview. Here, the so-called head tracking is used.

So you can watch a movie, for example, and it sounds like in the cinema. You move your head and the sound follows the direction of your gaze. With music, for example, this was done without and the sound remains unaffected by the head rotation. This makes it clear: even with identical hardware use, the content decides how Apple handles spatial audio. But which devices use Apple head tracking?

Apple Spatial Audio products

This includes Apple’s hardware offering that supports spatial audio. This now ranges from headphones like the Airpods Pro 2, to media players like the Apple TV 4K, to smart speakers like the Homepod. Of course, the iPhone and iPad also fall into this category.

The handheld smart devices are mostly used in combination with Apple’s headphones. However, Dolby Atmos Music advertises that the built-in speaker can be used without headphones. Hopefully, it is clear that you should not expect too much from that. Even if cell phones and tablets get better and better soundwise, they are not boomboxes.

What these products can do and how they can be used differently in terms of spatial audio will be explained in a moment.

AirPods Pro / Airpods Max headphones

As described in a previous blogpost, Apple’s spatial audio-capable headphones work really well. The headphones come in in-ear Airpods Pro 2 and over-ear AirPods Max versions. They are connected wirelessly via Bluetooth, especially since the Silicon Valley company has banned the good old stereo connection from iPhone and iPad.

Head tracking is made possible by built-in gyroscopes and accelerometers, i.e. motion sensors. This makes sound experiences much more immersive. The headphones are connected to the iPhone or iPad and use it as a reference point. So when you move your head, the device detects this, and the audio is counter-moved to the head rotation. Thus, the sound scene remains subjectively stable and does not rotate. This makes it much easier for us to immerse ourselves in a sound scene. However, only time will tell if this is desirable for every type of content.

In order to use Apple spatial audio with the headphones, you need the appropriate content. This is now quite easy to find since most of the common streaming providers are supported by Apple. Among them are Apple Music, Tidal, Netflix, and Hulu, to name a few.

In terms of audio formats, 5.1 and 7.1 surround sound, as well as Dolby Atmos, are supported.

It should also be mentioned that the headphones have an active noise-canceling function. This doesn’t directly have anything to do with Apple spatial audio, but it definitely helps with enjoyment when you’re in a noisy environment, for example.

iPhone / iPad

In the context of Apple spatial audio, mobile devices such as iPhone and iPad serve as playback devices and, in the case of head tracking applications, as reference objects. However, not all iPhones and iPads can play Apple spatial audio. This is possible with devices from the iPhone 7 and the newer iPads, the most important thing here is the operating system: iOS 14 and newer support corresponding spatial audio apps.

The new Lidar sensors, which are installed in the new Apple devices such as the iPhone 12 and iPad Pro, could also prove to be exciting. Lidar – short for light detection and ranging – is used to measure rooms and is therefore very interesting for AR applications. And we have already seen that audio is important for AR here.

It is made possible with the ARkit, the Augmented Reality Apple Developer. With this, an audio source can be played from any direction in the scene. This works via the so-called SCNaudiosource to which the audio is attached. If you also create the SCNAudioPlayer, the 3d sound becomes audible. This is how simple Apple spatial audio in AR is.

Apple TV 4k

The Apple TV system works via so-called set-top boxes (STB), i.e. media players that are connected to the TV and have access to various streaming platforms and apps. Apple has now reached the sixth generation of these STBs. Since the fourth generation, these boxes have been working with the company’s tvOS operating system, which is very similar to the iOS operating system for mobile devices.

Apple TV with Dolby Atmos sound system

The introduction of this operating system enabled Dolby Atmos support. So you still need a compatible Dolby Atmos sound system. Of course, movie content must also be available in 5.1 or 7.1. Not all of them are, as it says in the fine print, but most of them are.

Not everyone has several speakers or a soundbar next to the TV. Therefore, there is also the possibility to pair Apple TV directly with the company’s headphones. Here's a guide.

Update tvOS 15

With the introduction of tvOS 15, head tracking will also be supported starting next fall. This will allow Airpods Pro 2 and AirPods Max to be connected to the STB. As with the iPhone or iPad, the box then acts as a reference point, which is supposed to create an immersive listening experience that makes you forget you have headphones on.

Since the fourth generation of Apple’s set-top boxes, Dolby Atmos playback-capable systems can thus be used: From speaker set-ups to soundbars to smart speakers. The aforementioned introduction of the new operating system tvOS 15 additionally enables the playback of spatial audio without speakers – one neighbor will be grateful.

Regarding the available content, there is already quite a bit to choose from, especially in the movie and video area. However, Apple Music is also supported here, so there is also a lot to discover in the musical field. However, it remains to be seen whether Apple Music support offers real added value since you are more or less tied to the couch when listening to music. Well, the feature does not hurt in any case.

Homepod Smartspeaker

Apple’s smart speaker flagship is the Homepod. Equipped with 7 directional speakers and a woofer, the speaker can create a 360° sound experience via beamforming. Sounds like a very exciting technology. If you want to learn more about 3D sound in soundbars and smart speakers, here can find the magic.

Since the latest Apple Music update, namely the aforementioned Dolby Atmos support, spatial audio playback is also possible via the in-house music streaming platform. However, in order to play audio via the smart speaker, it has to be connected to an iPhone, iPad, or an Apple TV STB.

The fact that Homepod does not support the latest Apple Music update, namely Apple Lossless Audio (ALAC). This is not surprising, because unfortunately this will not be continued in the future. Instead, they want to focus more on the more popular Homepod mini in the future.

Update 23/01: The second generation Apple HomePod supports 3D audio with Dolby Atmos for audio and video with both one speaker and two HomePods combined into a stereo pair.

Apple Facetime Chat

In the course of the new iOS 15 update, the video telephony messenger Facetime also receives spatial audio support. This should make video meetings much more natural and pleasant in the future, keyword Zoom Fatigue.

But not only Apple uses 3D audio for video calls, but more and more companies also recognize the added value behind it and I have already dedicated myself to this topic. Nevertheless, the full potential of spatial audio is far from being exhausted. Here you can learn more about it

Logic Pro Apple Music

Soon Apple’s in-house DAW (Digital Audio Workstation) will be equipped with tools for spatial audio mixing. Currently, the software supports Quadro, 5.1 Surround (ITU 775) up to 7.1 SDDS (Sony Dynamic Digital Sound) in addition to stereo. This means that a maximum channel count of 8 is currently supported.

Comparable audio software for music production like ProTools, Nuendo or Reaper also supports Higher Order Ambisonics with 16 channels. Reaper even supports up to 64 channels per track. So Logic Pro currently has some catching up to do.

Now the question: What does this mean for Apple Spatial Audio that everyone is talking about? More than announced it was not yet, but as an audio engineer, we can speculate a bit here. After all, audio programs already mentioned can do this. So is Apple following suit with Logic Pro or introducing something new?

Speculation 1: Dolby Atmos Music Interface

Apple naturally wants to fire up its own platform, Apple Music. This is technically implemented with Dolby Atmos Music. Thus, it is obvious that now Dolby will also get a special panning in Logic Pro. Avid Pro Tools Ultimate has received its own object audio track thanks to the cooperation.

This can be used to control the Dolby Atmos renderer, which runs separately to ProTools or Steinberg Nuendo. This also bypasses the channel limitations that exist in the software. The actual spatialization is done by the Dolby software. The DAW only sends the mono objects to the renderer and this can play back the audio mix as binaural audio or also multi-channel on speakers.

Since Dolby Atmos usually has a so-called bed track, it may be that Logic Pro is retrofitted here. ProTools has received a 7.1.2 track, which is added to the audio objects. It is used, for example, for spatial reverb, Atmos or multi-channel recordings.

Speculation 2: Spatializer plugins

Rather unlikely for me, but maybe Apple will release its own AU plugins. I doubt it’s interesting for Apple to implement multichannel tracks beyond 7.1. That’s probably irritating for users. In fact, only stereo is played back on Apple Music currently anyway.

So it could be that Apple is releasing a spatializer that is similar to DearVR Micro. Here, a mono sound is simply put into a virtual 3D reverb and directly binauralized. This makes a sound spatial, as known from 8d music.

Far thought it would not be, however, that all mixing engineers then deliver only a stereo. It would be conceivable that an ALAC is played out during the export, which picks up the 3D metadata from the Logic Pro session. So you have a binaural, 3D stereo during playback. But in the container, you would still have the possibility to get more information. Especially if personalized HRTF and head tracking become an issue in the future.

Caution with 3D music

With this, Apple wants to give even bedroom producers access to 3D audio as well. I think that’s great at first because I can’t do all the work on my own anyway. Nevertheless, caution is advised here, because many techniques known from stereo workflow do not work in 3D. That’s one of the reasons why there was some initial disillusionment with Apple Music, as described here.

How to get around it when it comes to 3D audio technology? I’m happy to help with that!

Apple VisionPro

Yet unknown if this will be a thing, but decided for yourself: https://appleinsider-com.cdn.ampproject.org/c/s/appleinsider.com/articles/22/12/01/apple-ar-vr-operating-system-now-called-xros/amp/

Update: Now we know, the Apple VisionPro AR/VR headset is here. Read everything you need to know about the Apple VR headset. It included features such as:

• Facetime Spatial Audio, as described above but with extra immersion
• Capturing Spatial Audio with its six microphones
• Audio pods that give you a personalized spatial audio experience with audio raytracing
• Unity partnership to get all the VR apps to the app store and VisionOS for developers

RealityKit Pro Spatial Audio

Discover the three types of Audio in RealityKit: Spatial, Ambient, and Channel. Let’s have a look in detail:

• RealityKit sounds are spatial by default so audio sources sound like they actually exist in your surroundings. Spatial audio component lets you customize how your objects emits sound into your space. To make them even more realistic or more artistic. Use directivity to omit sounds in all directions or projects sound in a specific direction.
• Ambient audio component is great for multi channel files, which capture the sound of an environment no additional reverb is added to ambient‘s tracks. Each channel of the ambience is played from a fixed direction, suitable for 5.1 recordings or Ambisonics
• channel audio component: it sends the audio files channels directly to the speakers without any special effects. This is ideal for background music, which is not connected to any visual element. So non-diegetic elements also like a narrator voice.

You can add audio to your scene in reality, composer, pro and interface with it using RealityKit, or you can hook up audio and code. Find out more about spatial audio file formats in a dedicated article.

Conclusion

You can see, as usual for Apple, that everything is somehow connected or dependent on each other. In the case of Apple Spatial Audio, a host device such as iPhone, iPad, or Apple TV STB is required: First, the connection to corresponding content in the form of compatible apps like Apple Music is established here, and second, these devices serve as reference points for accurate stabilization of the sound scene in the case of using headtracking-enabled headphones. Furthermore, corresponding playback devices are needed, which would be the Airpods Pro 2 and AirPods Max headphones from Apple, as well as the Homepod smart speaker.

This interconnectivity is, of course, practical in theory, but on the one hand, it means that you are very brand-dependent, which you are already used to as an Apple user, but on the other hand, it also means that some content works better and some worse. But which one that is and how you can take advantage of it, I can explain to you in more detail in a non-binding conversation.