Why immersive storytelling works best with 3D sound design

What is immersive storytelling?

“Storytelling” is one of those terms you hear again and again – especially in new media production. Most creative people can probably roughly imagine something under this term: the classic hero’s journey idea, protagonists/antagonists, suspense arc…

But what does storytelling actually mean for content creators? And how does storytelling work in practice? And even more important: What role does sound and data play in this more traditional storytelling?

This topic was a collaborative work with Adam Audio, here you can find the first part.

Narrative sound design

One of the probably oldest sound clichés goes with this: the greatest compliment a sound person can receive is ignorance. If no one complains, the sound was probably good.

“Good sound is neither noticed, nor it is missed!”

Of course, that’s nonsense – good sound should be praised just as much as a good picture. Although it highlights the problem: Sound as art tends to be noticed subconsciously. Listeners are rarely aware that sounds serve a variety of functions.

Without the use of a narrative soundscape, certain expressions or narrative forms would not be possible.

What is sound design?

Sound design is the creation of the sound layer, and thus an important part of the artistic design of a film (or other medium). Sound is a tool to create dramaturgy (emotionality, fear, tension, reality, etc.) and to subconsciously convey information about time and place.

Here you can find an external article about storytelling in sound design.

We will now look at different ways to explore, tell and create stories through sound.

How can sound be used for immersive storytelling?

Now that the terms storytelling and sound design have been clarified, we go straight to the next buzzword: Immersion. Especially in immersive storytelling, the different senses have to support each other.

In film or VR (virtual reality), the eye is supposedly the first priority. However, in classic moving images, sound usually has the function of underpinning what is seen. We will discuss actively chosen clichés, devices and stylistic breaks in more detail later.

But with virtual and augmented reality it goes one step further.

The ears show the eyes where to look.

Because unlike in the theatre, the cinema and on TV, the image is not only in front. The image is spherical, so to say 360 degrees around us. Even though surround sound with Dolby can also reproduce sound from behind or above: It is not desirable for the listener to be irritated by sounds that do not come from the front.

However, in the world of virtual reality, this is precisely what can be made us of. Immersive storytelling in virtual spaces is calling for well-designed 3D sound.

But first things first. Before we open up a new dimension with surround sound, let’s stick to classic stereo for now.

Creating moods in (virtual) reality

One of the most important features of the “sound” component is the creation of moods and emotions. Be it with classical film music, pure sound design, or a combination of both. In this way, it is possible to make clear what is happening. A suspenseful, exciting scene only becomes really exciting with music and sounds. Both components are crucial in order to create a immersive story.

Without the right soundscape, it can be difficult in both games and films to build up an emotional connection with the characters or to recognize significant phases in the action.

Representation of environments

Music and sounds can also be used to make the era and setting of the narrative easier for the audience to identify. Stories set in a particular time or place need a sound design that matches it.

For example, the sound of old radio and record player recordings from the 1940s can be very effective in creating a mood of that time. So it’s also about using sound design to meet the expectations of audiences of the audience. I like to say:

It’s not about reproducing reality, it’s about creating a believable soundscape.

Creating immersion

Another important point is the creation of immersion, another absolute “buzzword”. But as often as you hear this term, and as generalising and non-concrete as it is, immersion plays a huge role, especially in 3D audio.

In video games, VR experiences and 3D audio in general, one of the most important goals is to “tie” the listener to the action and thus immersively involve him or her. Sounds and music play a major role in order to create immersive experiences and immersive stories even in video games.

That’s what everyone who talks about immersive storytelling says. But on this blog I aim to go further into detail, and uncover the secrets of sound….

As explained previously, emotions can be strongly influenced and controlled by music and sound effects. Breaks or gaps in sound can destroy immersion, and pull listeners out of the story.

Music and sound can also be used to interact and create links between scenes and to transition from one level of action to another.

Providing cues and information with an immersive story

One of the most important functions of the sound level is that of conveying information. A horror film that runs without sounds and music clearly loses tension and horror. Especially in this genre, sounds are often used to announce things (a door squeaking in the background), music that suddenly becomes more suspenseful, or even the sudden absence of any sound.

This level of auditory and musical information cannot be conveyed purely visually, but is elementally important for immersive interactive experiences and storytelling. It blurs the boundaries between diegetic sounds like those mentioned above (door, atmospheres, etc.) and non-diegetic sound effects like drones, PADs, whooshes and impacts.

I should probably go into more depth on this in another article….

Especially in music, the term “leitmotif” is worth mentioning here. Leitmotifs serve to associate specific musical motifs with characters. It is through the use of leitmotifs that important clues can often be given for the understanding of the plot.

You can see a nice video that deals with storytelling with sound here:

Higher Order Semantics

with the kind support of Marco Dahl

Higher order semantics is the term used to describe sound objects with a simple meaning that gain importance through a wider frame of reference. Sounds a bit scientific, it is. So look forward to concrete examples that you will certainly know from the immersive experience of storytelling.

Signals

These include digital signals, for example, which have a fixed communication content in society and can be described as sound objects in sound design.

An example of this is bells: depending on the sound and cultural/social context, they can be perceived as pleasant or also as warning signals. So low bells are more associated with churches and something serious. High bells, on the other hand, tend to have something “lighter, magical” about them.

Symbols

Another sound object of higher semantics is the symbol. In contrast to signals, symbols do not have a fixed communication content, but must be interpreted in order to be understood.

The interpretation space for symbols can be established by a divergence between image and sound. An example of this could be a door: one sees a street with a large door. In front of it a brawl is taking place, while behind it people are partying. This visual separation also separates the narrative in this case, with “in front of” and “behind” the door also differing sonically.

Stereotypes

Stereotypes are visual and acoustic characteristics that have been formed through countless repetitions in film culture. The learned features thus create a certain expectation of the acoustic and visual stimuli.

Since these stimuli are perceived only incidentally through learning, we speak of an “economisation of perception”. This creates space for other information and can give the viewer a feeling of “order and security”.

Narrative functions

Diegesis

In film, diegesis refers to everything that takes place within the fictional reality. For example, two people talking to each other within the story being told, or being heard by others within the story. Or like the examples of doors and steps already mentioned.

Nondiegetic refers to auditory sources that are not part of the fictional mixed reality show. These include, for example, voice-over voices or film music. They are not part of the scene. These elements would not be seen in the film or in VR.

With diegetic sound, a distinction can be made between active and passive off-sound. Active off-sounds have the characteristic that they stimulate the viewer’s involvement and are clearly assigned to localisable objects.

Passive off-tones, on the other hand, are intended to give the viewer an auditory constant and not to draw the audience’s attention back to themselves, i.e. they are sound objects without direct reference to action.

Dichotomisation

Dichotomization means a contrasting of different facts, which should make the difference between them clear. In the film “Raging Bull”, for example, the social advancement of the protagonist is made clear through the juxtaposition of two completely different sound environments (first loud, aggressive, then later muffled and quiet).

In the film “A Quiet Place 2”, strong tonal contrasts are also used: The part where the monsters live, triggered by sounds, is shown as absolutely silent. In contrast, the island environment where there are no monsters yet is filled with life and loud sounds.

Disorientation

In disorientation, the image in the film is shown in such a way that only partial sections are seen or the image is made indistinct by blurring. The indefinability of the space can be supported by the auditory level with a “diffuse atmosphere”.

The lack of information on the visual level draws attention to the sense of hearing, which can play an important role in this case. The disorientation is intended to affect the viewer’s perception in various ways.

This can be brought about aurally, for example, by a sensory overload, i.e. a dense accumulation of sounds around the viewer. This is intended to impair perception and make orientation impossible.

Alienation

Alienation is an effect in which the viewer’s expectations are broken with the use of strange sounds. In stereotypical environments such as a beach, one has a certain expectation of the soundscape (seagull cries, wave sounds, wind), when this expectation is broken, the scenery can seem strange and unfamiliar to the viewer.

Conclusion

So, as we can see, immersive storytelling is a large field on which an article 10 times the length could certainly be written. But what can be seen even in an article of this length is that storytelling offers many opportunities as students learn how to make narratives more engaging through the deliberate use of certain techniques and skills.

Of course, all of the above points apply to stereo sound just as much as they do to 3D audio. However, 3D audio technology can benefit storytelling in particular: Because 3D audio opens up a plane around and also above the viewer, this spatiality can be perfectly used to tell information and moods via the auditory level.

Especially with regard to virtual reality content, a three-dimensional image is thus complemented by three-dimensional sound. Through cleverly used storytelling and immersive technologies together, a story can be told that can be both highly realistic and totally “whacky”. The most important thing in both cases – and here we are again – is a successful immersion.

And this can be used in a new technologies very enriching way through clever storytelling + 3D audio. How exactly can this be done? Just ask me without obligation.