Virtual Reality Music for cinematic 360° experiences

Thanks to VR Videos, it is now necessary to ask ourselves things, that seemed to be clear for ages – using music for example in interactive experiences. Imagine attending a live concert from the front row, all from the comfort of your home, thanks to VR headsets and virtual reality technology that replicate the concert experience in stunning detail.

The problem: If you hear music in a 360° space, you should see musicians somewhere in the frame, right? So if you use it anyways immersive music videos, is the music supposed to rotate with the viewer’s line of vision with the VR headset, or should it stay as a static stereo?

These questions can not be answered that easy because they depend on a lot on the subject. That’s why I want to discuss the typical scenarios.

Note: This article is based on linear content only. Music for interactive VR content is a completely new topic that I will have a look at sometime.

spatial or non-spatial – that is the question

New tools to inspire audiences with virtual reality glasses

Technology has developed rapidly in recent years, particularly in the field of virtual reality (VR). This development not only brings new possibilities for gaming and streaming, but also opens up completely new worlds in the field of music production and the musical experience. The combination of advanced technology and creative application creates a unique interactive experience that is of interest to both the classical music lover and the modern technology enthusiast.

The right content is more than a game

It’s fascinating to see how VR combined with music can transform the living room into a concert hall, offering a whole new level of immersion audience engagement and interaction. This is not just a technical revolution, but also a creative one that could fundamentally change the way we experience music.

It’s not just about new musical instruments that can appear out of nowhere in a VR title like this. Here I would rather look from the direction of presenting music in the movie, how the songs should give the visitors the right information about the experience – whether Beethoven would be enthusiastic about everything with a headset? Decide for yourself with your project.

Live Music Performance in 360°

The simplest case is probably the virtual environment recording of music creation from a live performance. You can see musicians and artists, usually on a stage, so it is pretty clear, from which direction the virtual reality music is supposed to come from, while the applause is coming from the audience and fans for instance.

However, be careful with realism: In reality with live music, the sound source would be the loudspeakers. Although here, it would make even more sense, if the sound of a guitar, for example, can be located from the actual position of the guitarist as with live music.

VR360 Music Videos

Music videos are like icing on the cake and are constantly popular among listeners. The new trend is – who would have thought – 360° Videos. Out of personal experience, I can say: for good reason! With a VR headset in front of your eyes and music on your ears, you can really dive even deeper understanding more into music. It is a bit like you close your eyes and get lost in the world of music. However this time, imagination gets beaten by a music video that shows where the visual journey goes.

Here it is hard to tell if a static stereo-mix or an immersive mix would be the right way to go for virtual reality music. So for VR music video, you have to decide if allowing viewers to become active participants is what the VR technology should enhance your work.

Stereo-Mix

It is obvious to go for the mix that is also running on the radio. For me, this is a unique way and one of the few occasions, where virtual reality videos may resign spatial sound. The advantage is, that the musical instrument or piece will always sound like as it was supposed from the producer as there is only one version of the track that is supported by every platform and device.

3D Audio Music

This immersive version of playing the song requires an extra mix, which increases the cost and effort of the production. I just realize that I would have to do an entire blog post on that subject to really highlight all (dis-)advantages. But for now, you can form your own opinion with the following example:

Great Case: EDEN – drugs (VR Experience)

For this song, the effort was taken to create an individual mix for all kinds of users of VR-platforms. To be exact, YouTube with Ambisonics, and JauntVR with Dolby Atmos VR. But of course, there is also the original Stereo-Mix on YouTube.

So, do your very own music comparison, with JauntVR on GearVR and the YouTube-App on Smartphones or Chrome-Browser on desktop devices.

Personally, I like the sense of depth and space you get from a spatial sound mix, because the rotation of the sound supports the free movement of the pictures and therefore improves immersion. But I also enjoy the static stereo version. It feels somehow closer in your head, just like a piano when you know it from your walkman.

Music with Storytelling in Cinematic VR

There is a lot that can be said about this topic, but I want to break it down into core facts. Using film scores in 360° videos must be re-thought from the base because special effects and virtual and augmented reality music can rarely be reflected by a visible sound source (musicians, loudspeaker, monitors, etc.) Although you can take the easy way out, and use film scores just like it is being done for non-VR video. The audience has learned from the cinema, to be emotionally guided by the music and does not think “I hear an orchestra, so where is Hans Zimmer?!”.

For me, I mostly decide to be spare on using music in virtual reality headset, but it really makes sense if you have a couple of 360° shots in a row where music will not overload the mental impressions of the viewer. For this, I stay with plain stereo. As a composer, I usually don’t want the listener to have any influence on virtual reality music. But there is also the possibility to direct the attention with music in excess of sound. For instance with individual virtual instruments that are positioned in 3D space and can, therefore, be located.

Surround Sound music without Virtual Reality

There is a last determination I want to make. Mixing music in surround (5.1, Dolby Atmos, Auro 3D etc.) has been common for live concerts for quite some time. Although the approach to mixing cannot easily be adapted to VR content. This is the reason why: For surround mixes, you usually have only one fixed line of sight because the audience usually sits in the middle of a surround loudspeaker setup with a screen in front of it. So it is common to please the vocals or snare drum for example in the center so it sounds as if it is in front of the listener.

However if you carry this mix over to 360° playback on headphones, it would lead to the case that e.g. the vocals can only be heard on one ear as soon as you rotate the head by 90°. To simplify the consequences it can be said that for virtual reality music the individual musical instruments themselves should be placed more carefully in 3D because they can easily start to stick out as soon as head-tracking is used. In this blog post you can read more about VR concerts.

Conclusion

Virtual reality music offers an exciting and revolutionary opportunity to experience music in a completely new way. The combination of visual and auditory experiences in a virtual space opens doors to previously unimagined creative possibilities. Despite the challenges posed by the technology and production, it offers enormous potential for immersive experiences that go far beyond what traditional audio or video formats can offer. While we are still in the early days of this new era, it is exciting to see how content producers, musicians and technologists will use this technology to take music listening to the next dimension.

There are no general answers. They really depend a lot on the scenario. VR music videos on platforms like YouTube and Apple Music are changing the way music is consumed, offering a visual and immersive experience that goes beyond traditional music videos and songs. With the potential to revolutionize live streaming and interactive concerts, the realm of VR music holds the key to an exciting, tech-driven future with new avenues for the music industry and space for creativity.

Personally, I prefer music in static stereo because it keeps the music production classic and does not introduce a lot of surprises. However, there is still a lot of unexplored space and for myself decided to play and experiment more with virtual reality music and I’m excited what follow content creators will come up with the new possibilities.