The magic behind Netflix and AMBEO Spatial Audio explained

So there it is, Netflix is following up with spatial audio support from Ambeo… or ahead? Because there was already the news over a year ago. Netflix supposedly supported Spatial Audio with Airpods Pro.

But in fact, it was a hoax. At least for a while. Today, Netflix supports Dolby Atmos on streaming apps just like Disney+, AppleTV, and Co, Netflix stayed in stereo. By the way, here are all apps that support Spatial Audio movies.

Confused? No worries. I had the opportunity to gain some insights from the streaming service. So thanks to Scott from Netflix for sharing his insights and getting in touch.

Update 23/01

Netflix Premium plan subscribers can already enjoy an immersive audio experience with no additional equipment and over 700 titles – find the list below, it was also updated. Enjoy your content with new dimensionality – Netflix Spatial Audio will automatically activate on stereo streams, indicated by the spatial audio icon. Here is a new video showcasing the audio feature, and a comparison of Netflix’s spatial audio to stereo:

Update 23/04

Sennheiser introduces Ambeo 2-Channel Spatial Audio for live productions. The technology was prototyped at the NAB show and is aimed at broadcasters and sports productions. How it all works is in the text below.

Why Netflix decided to make spatial audio accessible to everyone?

This can be answered quickly. Just like myself, Netflix strives to enable everybody with the best possible immersive audio experience. Before partnering with Sennheiser, they have been working with HDR video and Dolby Atmos (which I will cover later in this article). So what we don’t know as consumers is research and experimenting happening behind the curtains.

Previously, you needed dedicated hardware and software to listen to spatial audio in cinema-like sound. While I know that it can actually be possible to experience at least binaural spatial audio at home without having surround sound speakers or home theater euipment. All you need is headphones. So Netflix could have just enabled the Dolby binaural renderer, right?

Well, but what if somebody listens with his or her TV, or has a stereo loudspeaker setup? Which is still a huge percentage of how people watch Netflix. They would need to change the audio setting in the Netflix app is vary depending on what hardware they are using. For consumers, you want to make technology as easy as possible.

So all in all Netflix wants to make it easy for all of their more than 200 million subscribers to get a sense of what immersion audio could be like. No matter where they sit or what their budget for audio hardware is. And guess what, they choose an interesting approach to make it work that we all can learn from.

How spatial audio is usually delivered on headphones or loudspeakers

Right now every techy news portal is banging out the same announcement. That Netflix is now more about immersive sound, via speakers and headphones. Until now, spatial audio was reserved for people who either owned Apple headphones like the Airpods Pro and Airpods Max. Or speaker enthusiasts who own a soundbar with Dolby Surround or 5.1 function. Now everybody can have a listen. No matter what 2.0 system you use. Newsflash over.

But from an audio perspective, this is very questionable. Because binaural works best – or rather exclusively over headphones. Binaural audio played back on stereo loudspeakers sound weird. It can be described as canny, roomy, and dull.

What if you don’t have more than two speakers? We need a different approach. With speakers that don’t have multi-channel capability, cross-talk cancellation is common. I’ve already reported on this in more detail at 3D Audio Soundbars.

Sennheiser has released a pretty impressive press release here. But for non-audio folks, there are some waveforms and it’s hard to understand what is happening besides some “magic processing”. So let’s find out what they did.

How does Spatial Audio work on Netflix with Ambeo?

The spatial audio community was very confused and discussed a lot. How can it be possible to have good immersive surround sound both on headphones and normal 2.0 stereo speakers simultaneously?

Technically it is not that easy to achieve this. Actually, it is a contradiction. Therefore I saw a lot of marketing hype on news platforms. Using words like Spatial Audio, Dolby Atmos, etc. in the wrong context. But it indeed looks like the audio engineers achieved something that seemed impossible.

Consumers may have heard of the terms SRS 3D, Dolby Atmos, or even DearVR. These audio techs do work for headphones, laptops and tablets and chances are good people had first touchpoints. But all the magic algorithms happen on a certain device. This adds an additional boundary to making “more immersive audio” accessible.

So let’s start with what Netflix already achieved in the field of spatial audio before collaborating with Sennheiser:

Does Netflix support spatial audio with Dolby Atmos?

The short answer is yes. Of course, you need a subscription to gain access to Netflix’s content and find a movie that was mixed with Dolby Atmos. You can use the Netflix search bar on your streaming device by typing in ‘Dolby Atmos’ to find suitable titles. Currently, it is supported as a 5.1 audio stream for loudspeakers and soundbars. Or when played back on an Apple device with Apple’s headphones you have multiple options to choose from. This I will explain later, but let’s have a look of how Netflix actually enables spatial audio for everybody:

How to listen to Dolby Atmos on Netflix

When I visited Dolby over five years ago, I remember the following: An employee was uploading one Dolby Atmos master file after another. These files can be up to 100GB big – just for the 3D sound without visuals. So all the information for spatial audio is already laying somewhere on Netflix’s servers.

But as you can guess, streaming or downloading for offline viewing that much data at home could be an annoying buffering nightmare. So there is some encoding and downmixing involved. At the moment, with the appropriate app on the smart TV, you can at least enjoy most of the tv shows and movies using streaming services like Netflix, in 5.1 Dolby Surround. Better suited for streaming on mobile devices and smart TVs. This is how it usually.

It doesn’t mean you can only access 5.1. surround sound speakers. For Dolby Atmos, you need the right hardware such as an AV receiver or 3d soundbar. Since it’s object-based encoded, the playback is delivered optimized for sound speakers in your loudspeaker setup. It is a bit hard to find out and regularly updated, but chances are it’s enabled automatically a so-called EAC3-JOC.

How to listen to Dolby Atmos with Apple Airpods

All you need is a supported Apple device and the Apple Airpods. The newest generation hardware is all capable of that. The headphones are connected to your device. And the device detects that there is a movie with Dolby Atmos content. This information is transmitted via metadata embedded in the movie file.

While running the Netflix app, you’ll see the Netflix spatial audio icon in your headphone settings. There is a little text saying “Dolby Atmos” or “multichannel audio accessible”. Then you just need to press the “3D audio / spatial audio” button on the button right. When it’s blue, you’re good to go to enjoy spatial sound!

You can even enable your disable dynamic head-tracking. I strongly recommend head-tracking. It solves the so-called front-back confusion. Also it sounds even more like a cinema sound experience. A bit like VR sound, but without the HMD.

On an iPhone, it looks like this. The left image shows a movie that supports spatial audio in the spatial audio setting. In the middle, 3d audio is activated. And on the right, it shows spatial audio setting a movie that was just mixed in stereo. Here you can at least enable the “spatialize stereo” function by Apple.

Something that gets close to what Netflix is doing with Ambeo with an important difference: Apple Spatialize stereo is using stereo as a source. While Netflix movies make use of the multichannel input for Sennheiser AMBEO 2-Channel Spatial Audio.

Will Android support spatial audio?

Probably yes. At the moment Android doesn’t have such a neat implementation and pipeline of spatial audio as Apple. From the hardware with smart devices and headphones to the software with its operating system and apps. Android users need to be patient. Currently, every spatial audio tech that Google aka Alphabet is supporting can be found ”here”.

But the next generation of Pixel Buds has been announced. The Galaxy buds pro had some light version of head tracking. And the new Android version is supposed to have 3d audio features. So long story short: Netflix will continue to add spatial audio technologies that create a better immersive experience.

Let’s finally have a deep dive into the technology used by everybody. We are now on the same page, so let’s find out how they made something possible that seemed impossible.

What is Netflix’s Spatial Audio Sennheiser AMBEO 2-Channel?

Netflix has teamed up with Ambeo. This Sennheiser brand is known for innovative ideas in 3D audio. Turns out, the two parties have been working on this for a few years. They really wanted to make sure that the final version doesn’t just become a mixer. Or a narrowed-down rendering just like it is happening for music streaming.

They wanted to have audio mixing engineers and consumers likewise on board. This is pretty special and requires a lot of work. What is the process of delivering this audio format to the streaming service? A brief overview is:

• taking a multichannel ADM file from the original soundtrack
• applying the AMBEO processing to receive a stereo file
• getting approval from everybody involved internally
• Replacing the original stereo-mix with the “more immersive” mix

How is multichannel audio used?

As you probably know, most of the “big” movies today are mixed with Dolby production suite. I already described this hybrid object-based format for 3d audio in my article on Dolby Atmos Music. Some films used 5.1. surround or stereo as their main soundtrack. As much as I criticize Dolby, I think the infrastructure works great for films.

Unfortunately, to access this cinema experience, you need dedicated hardware. Like soundbars, receivers, or Apple headphones. This could be the reason why Dolby didn’t come up with the idea that Netflix had with Ambeo: Delivering spatial audio non-exclusively to everybody. Enough guesswork.

So Netflix takes an ADM wav-file – before gets processed by Dolby’s encoding – and applies its own processing to it. The processing can be seen in the screenshots, but to set things straight:

• The algorithm doesn’t apply any processing to the front channels LCR (Left, Center, Right). This maintains dialog integrity and a solid mix
• But it adds even bigger spatialization to the surround channels. This is pushing the enveloping immersive sound experience even more.
• No additional reverb or room is added, which other binauralizers do and consumers sometimes don’t like
• it is as we thought a mixture of binaural audio (for headphones) and cross-talk-cancellation (for loudspeakers) called transaural

Hold on a second: didn’t I just say there can’t be binaural and transaural at the same time in one file? Yes! But since we are targeting consumers here and not audio engineers I dig the approach that Netflix did with Ambeo:

What is the magic behind Netflix Ambeo Spatial Audio?

So you just read that there is some processing happening that is actually super complicated mathematics. Especially for transaural audio, you’d need to know where the listener is sitting, what loudspeakers are used, etc. What you can do is make assumptions. Netflix knows a lot about how consumers use the streaming service and therefore can guess what works for the majority of people.

In the end, what the research team did is many listening tests. This is the reason they just started with a bit over 20 films so far – over a period of two years. Doing tuning and making the settings right. They have been working back and forth.

I’m told that the processing they developed works for most titles. There is not really a need to do more finetuning since:

why would you change a method that works for most of the films?

As you can guess from the screenshot above you can adjust the level of sound speakers, and the amount of spatialization. Also, they won’t set any keyframes for scenes of the film. The algorithm works in one pass.

Therefore what differentiates this approach from the competition is that this method was not only tested with consumers. The original sound mixers – the person who mixed the actual movie – were involved in the process. And only if the audio professional – that knew the audio content better than anybody else – approved the Ambeo processing, this is when it got added to the Netflix streaming catalog.

More questions from engineers on Netflix Spatial Audio

What HRTF is Netflix using?

What HRTF is Netflix using?

The good thing about binaural audio is that the methods are pretty straightforward. You can easily create your own models. For instance, the video platform YouTubeVR uses a different sounding profile than Facebook360. Netflix decided to create their own profile that works best combined with transaural audio.

With Apple, there is a trend for personalized HRTF. Since everybody has a different pair of ear shapes, the spatial audio rendering should ideally be adapted. I wrote a dedicated article on that right here. So chances are good Netflix will also update from a generic head model to a unique one as soon as the mass infrastructure is there. The current workflow wouldn’t allow it because the HRTF is pre-encoded. But this can be updated of course.

Regarding cross-talk-cancellation, it seems to be possible to also find a setting that works for the majority of people. Most consumers sit on a couch or bed, while the tv is a few meters away. The room model is comparable. Of course, it makes a huge difference if a stereo soundbar or internal monitor speaker is used. But still, there seems to be a satisfying way of making the surround sound much more immersive sound than a stereo soundtrack. I made similar experiences in my projects.

Why was the original stereo mix replaced?

Speaking of the original stereo version. Usually, a film comes in a surround mix and a stereo mix. This is one of the reasons for Dolby’s success. In a perfect world, is no need for a dedicated downmix with object-based audio. The content gets played back in the best possible way. You are no more bound to a channel layout.

Historically, most films still have a dedicated stereo version for multiple reasons. TV shows and movie stations for example have different loudness requirements than a cinema. Such a stereo version works great on pretty much every device and platform. No matter if loudspeaker, headphones, smart tv, etc. But it’s not an immersive sound or system because all the spatial sound information gets removed.

Netflix decided to replace this stereo version with the Spatial Audio Sennheiser AMBEO 2-Channel system. As a content creator, I would have loved to compare and test this new immersive mix with the classic downmix. But Netflix has two good reasons to do that.

Firstly, the team is confident that the immersive version will always sound better than the one-dimensional stereo version

Secondly, Netflix already provides more than 100 synced audio streams per film. This is due to several bitrates and qualities that need to be streamed without any delay or dropouts – all in sync. Additionally, you have multiple language versions and even audio descriptions. Still, we have the additional 5.1 surround sound by Dolby. Imagine having multiple versions of immersive audio. Not really consumer friendly, right?

Are more updates and films coming up?

Yes, there will be more films on Netflix with that feature. As you now know, this testing takes a while. Netflix wants the film producers to feel good about it. Trying the maintain the original intent but still adding a new flavor. Pretty conservative as you wish, but precise.

If a film doesn’t have this audio feature yet, it doesn’t imply that the people didn’t like it. Netflix doesn’t necessarily own the movies. Certainly, they want the final result to be clean and the audio mixer to be OK with it. And as an audio engineer myself I really appreciate the valuation.

Which Netflix spatial movies are available for streaming in Sennheiser Ambeo?

But to give this article some substance, I took the “trouble” to type out all currently available movies from Netflix. To do so, simply use the Netflix search bar and type in Spatial Audio. Which movies are available and which titles can be streamed immediately can be found here:

• Red Notice
• Stranger Things
• Adam Project
• The Witcher
• Interceptor
• Kate
• Raising Dion
• Sie lieb – Bete und Gehorche
• Castlevania
• Locke & Key
• Jennifer Lopez: Halbzeit
• The Unforgivable
• The Order
• Fear Streat Teil 1 1994
• Fear Streat Teil 2 1978
• Fear Streat Teil 3 1666
• Spuk in Bly Manor
• Warrior Nun
• Black Summer
• Archive 81
• Selena Die Serie
• True Story
• Lulli
• Seoul Vibe
• The Good Nurse
• Schlummerland
• Resident Evil
• You
• Your Place or Mine
• Luther: The Fallen Sun
• Tour de France

These are just a few of the Dolby Atmos movies on Netflix. Netflix just enabled over 200 million with accessible spatial audio technology in 2023. Find out today how 3d sound can be the perfect solution for your immersive project.

Sennheiser Ambeo 2-Channel Spatial Audio for Live-Streaming

Sennheiser introduces Ambeo 2-Channel Spatial Audio for live productions, targeting broadcasters and studios, particularly for sports events. This technology aims to provide an immersive experience for stereo headphones and speakers. Translating the surround mix into into an enhanced 2-channel experience.

Sennheiser showcased a Live Renderer prototype for Ambeo 2-Channel Spatial Audio at the NAB Show. Controlled by the Anubis software in the broadcast industry. The Live Renderer converts multi-channel surround and immersive formats into two audio channels in real-time, with controls to fine-tune room settings for mix integrity and dialogue clarity. Basically what Netflix Spatial Audio does, but in Realtime.