Interactive Music for Immersive XR games

How can interactive XR music be used in immersive media?

This ByteFM Show is presented by Sophia Fischer. Interviewees:

• Christoph Spinger, VR-Nerds
• Martin Rieger, VRTonung
• Michaela Pňačeková, Symphony of Noice

Interactive, animated music videos

In part 1 we have already given a short insight, but let’s briefly summarize the three pillars again: The first one is the interactive, animated interactive XR music videos. For example, there is one that was totally celebrated in the press. Someone took this driving song “Old Friend” from Future Islands and created a colorful firework of music videos. If you have your magic glasses on, you can experience a truly absurd trip. With funny, dancing figures in changing colors with landscapes sprinkled under confetti. Can’t you imagine yourself? It’s also hard to explain. The LA Times commented:

“If ever there was a piece of content that you had to experience to understand, this is it. You can’t describe it either, you have to experience it.”

Volumetric Capturing in VR

The second version is the one in which musicians record their songs in front of green screens and you can stand next to them in the music video or on stage with the appropriate technology. Icelandic artist Björk, among others, has done this very impressively. The lady whose universe is much bigger than just interactive XR music. With her art, she constantly tries to fathom the limits of the possible and, also, the technically possible. For her single “Notget” from her 2015 album “Volnicura”, she has created a crazy spectacle. You can watch Björk, or rather her avatar, transform into truly fabulous beings in a dark, surreal setting. Flowing through elfish light, literally fantastic.

360° Music Videos

And then there is also the 360-degree variation, which you have to imagine that you are standing rigidly in a setting, but can turn around 360 degrees and whatever is shown there will be experienced. Maybe a little less exciting because it’s not that active, but for the sound, it’s still pretty cool. Because unlike in real life, with 360° sound you can also hear where Musicians are standing. In real life, you only hear the music coming from the stage, more precisely from the speakers. So the band and yourself have a completely new presence in virtual space.
The only thing you won’t hear anymore are for example people standing behind you and drooling all the time. And if you think about it now that you don’t have to stand in line anymore to make sure that you don’t see your heroes only in pin size on stage from a hundred kilometers away. Or that you can dance in complete exuberance and don’t have to stand squeezed in a mop like a sardine.

Diving into the music

The mental switching off workes perhaps even better for some of them. If you can be driven out of reality not only auditory but also visually. And apart from that, VR is also not a pure observation technology. Co-design is another exciting field. Of course, this is already possible in the classic gaming sector. For example, you can play virtual drums if you already have VR glasses and two sticks. Or compose music just by being present in virtual space:

Symphony of Noise VR

You can also make interactive XR music with it, for example in the VR project “Symphony of Noise”, which was presented at the Reeperbahn Festival last year. What exactly happens there, is what the expert herself can explain best. Michaela Pnacekova, Creative Director of the project, describes the whole thing as follows:

“You dive into, we call them, soundscapes and you discover and co-create sounds. That means you work a lot with your body, with your breathing, with your voice and you move in the space. But you also discover sounds that are already there and that you collect in different spaces. Then you combine all that together and conduct.”

Man becomes music. Everything becomes music. An immersive journey. By the way, immersive is one of the favorite words in VR and simply describes how much one forgets reality and dives into a new one.
So you put on your HMD, hear your heartbeat, footsteps, maybe pop a champagne cork, drop glasses on the floor and your breath. Unfortunately, I did not personally witness it, but I heard that it was also claimed that the ears were tuned like new afterward. I can already imagine that if you are pointed out that even in the everyday noises, which constantly surround us, a breath of interactive XR music is inherent.

Conclusion interactive XR Music

Well, I sense there’s more crazy stuff to come. We’ll see and we’ll literally hear. With this, I’d like to finish this edition. I would like to point out that you can subscribe to this podcast if you like it or not. The pleasure would be the same for me as it would be for you in the next issue on the other side of the Matrix. Thank you very much for your appreciated attention. I am Leonie Möhring.