How Spatial Sound justifies the Audio Metaverse Hype

This article aims to show the possibilities sound effects that spatial sound will have in the audio metaverse. Therefore, this text is also about user experience, communication, concerts in the metaverse, and audio branding.

The text is based on my little talk about the Audio Metaverse below. But I would like to add a few things. I also want to give especially non-German speaking people the possibility to understand the potential.

To be honest I’m a bit sick and tired of all the hype around the word metaverse. And now it leads to a bashing on Meta why it’s not as successful and Zuckerberg is losing billions. But the fact is for me that he is building incredible technology that will be a part of our lives in the future. And spatial 3d audio will definitely be a part of it.

UX / UI in the audio area

I always like to say: “in immersive media like VR, we can like to miss things, but not be overheard.” Because even in virtual reality, you can’t see what’s behind you, but you can hear it. This factor alone should really make you think when working with immersive media.

So how can I design my UI/ my UX better without having to paint arrows into my VR world or other curious ideas? I keep seeing experiences that ruin their beautifully designed 3D environment with graphics like this. So how good to be able to use visuals and sound to intuitively steer people through an experience!

Apple and head tracking

This is one of the many things you can do with 3D audio, and it’s actually accessible with just regular headphones. Despite the marketing of manufacturers trying to sell you a new pair of earbuds, or whatnot. Binaural 3D Audio is accessible on normal stereo headphones. But there is a new evolution of sound technology coming in called dynamic head-tracking. Read more on this tech here.

Apple has a lot of things happening in the background right now that are probably laying the groundwork for their upcoming XR devices. This device which coming apparently, but is postponed by a year every year it feels. Nevertheless, the the sound design will also be important here and they know it. Because Apple is also working on its own Spatial Audio technology. To what extent the Metaverse Audio will be used by Apple remains to be seen yet. But the infrastructure is building up.

The fact is that Apple goes one step further with its headphones, the Airpods. Their spatial audio has integrated dynamic head tracking. This is exactly what we already have in VR. That the sound is adjusted in the real time based in-time to the viewing direction. This can significantly improve the user experience. Above all, augmented reality applications are increasingly interesting for Apple headphones.

Why do only a few people know about head tracking?

Hard to tell because it’s already being actively advertised now. Apple calls it “Spatial Audio” in the USA and in Germany they call it “3D Audio”. Which is funny, apparently the Germans just like more numbers, data, and facts. But it looks like the average consumers aren’t really aware of this audio technology on the rise. Because they don’t know what they can even listen to?

So what is there for 3D audio content? You can listen to Dolby Atmos music, where the music kind of happens spatially in virtual space around you. Or watch movies with it, which is a bit more fun with it. If you want to know what other examples there are for 3D audio content beyond music and movies, you can find them here: Spatial Audio. As a content creator, I realized I have to rethink how audio content is experienced with this new freedom. Aka 3DoF (three degrees of freedom), the rotation around X, Y, Z axis.

This has already enabled me to explore and identify exciting use cases. I don’t think Apple itself knows exactly what they want to do with the technology yet. But they’re building it in already, they can claim already claim that they’ve created something new. And when the content is ready, it can simply be unlocked via software update.

Just like they did recently with their Personalized Spatial Audio. However, this doesn’t really give much added value so far. That’s also the point I want to get across to people! Not only is the technology cool and fun, but it can actually solve problems.

The added value of 3D audio is not Dolby Atmos

You guys are probably familiar with Dolby Atmos, too? It’s a lot of fun in the movie theater when you feel like you’re sitting in the middle of the movie. Everyone advertises it as being more immersive. But even I wonder what that means exactly?! The 3D sound gives more depth and it’s a nice experience when, for example, the plane flies overhead.

However, I actually see the benefits in a completely different place. If we look at how the current zoom calls work, we see that they seem quite monotonous.

After all, people don’t all sit on top of each other in a real round of talks. But that’s exactly what the tone of video calls like Zoom, Teams and Co. suggests.

Instead, the people are spread out, they represent a conversational life situation, for example, when they sit next to each other at a table. That’s exactly what Apple is building into Facetime. With it, you can really hear where people are sitting through the sound. This creates a conversational environment that is much more natural for our brains.

That’s going to solve the zoom fatigue that we all know from all the calls. So it’s easier for our brain to process information when it has a spatial tone.

Metaverse Navigation

And now, of course, you can take it a step further and ask yourself what the future holds for audio and metaverse. Now, before we talk about whacky applications like artistic concerts, here comes a practical example. And that’s for navigation.

It feels much more natural when I can actually hear, say, people in the virtual room in front of me. Then when you walk into the room, you could actually hear what’s going on. It definitely creates much more of a sense of presence to have voices around you spatially. Afterward, you also really have the impression that you’ve been talking to people – your friends, not just with your laptop.

Audio in metaverse can break through the boundaries of reality

You can go one step further. In such social VR worlds, you can overcome the physical hurdles of sound. You then build your own listening zones in immersive environments in a way that you couldn’t do in reality. As a user, you go in and people don’t hear what’s happening inside, and you don’t hear inside what’s happening outside. So you have a private space – a kind of sound bubble.

There are many more added values of audio in the metaverse. Here’s my article on Communication in Virtual Worlds.

More than entertainment with concerts

With virtual concerts, provided you do it right, the wow feeling is there. Because the crowd is there, singing along, cheering, or simply being present. That’s exactly what would make such virtual environments a concert. But currently, most of the experiments in VR seem quite sterile. Most of them just use stereo sound and pretend to be concert experiences. But more on what they are missing in terms of 3d sound later.

So you can still think the sound a bit further. And I think that’s where the potential of VR becomes apparent, that we don’t say we’re recreating reality. Because the sound in reality often doesn’t sound that great, because the window is open, for example.

There’s still an insane amount of potential there.

3D Audio is not equal to Metaverse

Some have probably read the news that Jean-Michel Jarre is producing “something in 3D audio.” And quickly the term metaverse comes up in several posts and articles. He said something clever about immersive audio, but it fell in a different context:

“For humans, the feeling of immersion is primarily audio before it is visual. The visual field encompasses 140 degrees. The audio field encompasses 360 degrees. So for the evolution of the metaverse, the evolution of XR and VR, the sound is absolutely critical. And I think musicians have a lot to say and do in virtual worlds.”

First, I think it’s great when people experiment with the new possibilities of spatialaudio. And he said it right, that you should already think about 3daudio when composing. I said exactly that years ago in my article about the good and bad Dolby Atmos Music.

So here’s an overview of why his work is not all that relevant to whatever the metaverse will be:

• The Metaverse is a more interactive format, but his experience is simply linear
• The metaverse will at least require head-tracking for spatial audio, something #Dolby doesn’t like yet
• the Metaversum has 6 degrees of freedom, so the sound should adapt to movement in space, something the audio formats it uses can’t even do.

Here is the misunderstanding, Jean-Michel did not even claim to have pioneered the metaverse. He states that there is a lot of potentials, and I have to agree with him. All he says in the article is “3D will be the new stereo,” so let’s be a little careful to separate immersive technologies.

Therefore we have to be way more careful. Just because it says 3D audio somewhere doesn’t mean it has 3D audio in it. Especially on other platforms like the Metaverse, different laws apply to audio content. I’ve literally talked to audio engineers who think they’re helping to build the Metaverse – by working with Dolby Atmos Music. But it’s just not that simple.

Seems obvious, but what does it sound like? Just try the demo from the colleagues of Atmoky. It is not a live concert in the real sense. But it shows very nicely how interactive 3D audio with 6 degrees of freedom sounds. All this in the browser, but I have rarely heard in alleged “Metaverse concerts”. Try for here for yourself!

Audio branding in the metaverse.

There’s a question that’s asked far too infrequently in connection with the Metaverse: Yes, what are people actually doing there? What are they seeing and, more importantly, what are they hearing?

One example is the ability for brands to create their own iconic sounds that they can then offer to users for promotional purposes. A classic example is advertising jingles. Everyone is probably familiar with McDonalds’ “Ba dab dab da.” But that’s very one-dimensional thinking; in the metaverse, there are more possibilities with audio:

It’s likely we’ll see more custom branding options in the future. Such as where users can unlock celebrity voices for their virtual avatars – for money. Chatbots can also be equipped with real-time text-to-speech and make you feel much more like you’re interacting with a brand. Then just with artificial intelligence.

Brands are an integral part of our daily lives, whether we realize it or not. Whether we encounter them on social media or in a virtual gaming environment, brands need to be able to connect with consumers in a natural and meaningful way. Ideally, brands should find ways to seamlessly engage in interaction moments to build a stronger relationships with their customers. Audio is key in this regard for recognition and to create emotion.

People need to experience it to understand it! Explaining is good, but experiencing is better.

So if you want to experience the Audio Metaverse, just drop me a mail!