360° video and 360° sound in one device?

More and more 360-degree cameras have built in more than just one integrated microphone and thus seem to support spatial audio. In principle, I find the development exciting, because currently additional surround microphones and recorders are needed, which have to be installed underneath the camera, which is not always possible without further ado.

So it could be an interesting option in the future. The Samsung Galaxy Buds 2 Pro, for example, offer 360 audio recording, allowing users to record surround sound videos by leveraging the built-in microphones of the earbuds. This feature is tied to specific Samsung Galaxy phone models and is a part of the company’s efforts to provide a more immersive audio experience during video recording.

But what is the current state of the art? Here is a review of Insta360 Pro, its capabilities and workflow on 360° sound.

Note: I have nothing to do with Insta. Due to a 360° project, I got my hands on the camera, but some points also apply to comparable 360-degree cameras like the Nokia OZO, Kandao Obsidian, GoPro Fusion or Vuze Camera.

Production and Audio Recording

The nice thing about it is that it works very easily. You only have to activate the Spatial Audio mode in the camera app and the camera records the 4 microphones in addition to the 6 camera pictures, as shown here on a moving horse-drawn sleigh for example. You can also adjust the camera settings for better audio recording. How this works from a technical point of view, read more about this later.

When you take your first shots, you’ll realize the Insta360 Pro isn’t really quiet. The built-in fan makes a lot of noise but can be set to be turned off as soon as the camera goes into record mode. However, this can become a serious problem, especially at high environmental temperatures.

Compared to the Nokia Ozo, it’s possible to adjust the microphone gain using the “Insta360 Pro Camera Control App”, unfortunately only on an inaccurate level from 0 to 100, so you never know how many decibels you’re on.

A similarly compact 360-degree camera is the Kandao Obsidian. This has a hot shoe adapter with a 3/8 inch thread. This means that a Gear360 can be installed on the camera as a preview camera or a Zoom H2n can be used as a backup recorder. Unfortunately, this is missing in the Insta360 Pro, as well as the remote function of the Kandao, to connect a Zoom H2n via a mini-jack cable to the camera, so that it will record as soon as the camera records. This can be very handy under adverse conditions and looks like the picture below, which was taken during my shooting in Bangladesh.

There is also no windscreen for the microphones. So, if any noise on just one of the 4 capsules should land, the entire Ambisonic sound field will be destroyed.

Post-Production and Editing

The 360-degree camera provides 6 video clips of each lens. The first two videos contain one stereo track each, i.e. a total of 4 audio tracks, which provides the signals of the 4 integrated microphones. Problem: The sound is already encoded into the mp4-files and the whole thing is encoded with the AAC-codec and a bitrate of only 128kbit/sec. This means that the sound loses a lot of quality already during recording and is almost a KO criterion.

The new Bluetooth audio standard, LE Audio, offers better audio quality, improved latency, and longer battery life. The 360 Audio Recording feature in the Galaxy Buds 2 Pro is the first to leverage Bluetooth LE Audio, providing a more immersive and realistic sound experience.

Insta has its own software to assemble the 6 lenses into a 360° image, the Insta360 Pro STITCHER. It now offers the possibility to assemble the raw data of the 4 micros to an Ambisonic sound field. The result is a file similar to the Zoom H2n. Due to the fact that the 4 microphone capsules are in one level, there is no height information for above and below. So you get a 4-channel wav-file in Ambix-format (channels 1: W, 2: Y, 3: Z, 4: X), where the third channel is silent, as you can see in the screenshot of Reaper below.

Checking the Spectrum with Spatial Audio

Here is a frequency analysis of the 360-degree camera with iZotope RX, using binaural microphones to analyze the audio spectrum, see below. (Blue is the waveform. The lighter the yellow/orange color, the bigger the intensity of the frequencies; black means that there is no sound.) You can clearly see that all frequencies above 12kHz are missing.

In addition, clear holes in the spectrum above 10kHz have been torn into the spectrum, which can be credited to data reduction. In addition, the picture shows periodic signals below 100Hz whose origin was not answered by Insta-Support. It is not audible but visible in the waveform and makes the low-cut filter mandatory.

VR audio is particularly relevant for 360 audio recording, as it often involves using channel-based microphones to capture audio from specific directions, enhancing the immersive experience.

This is what the spectrum should look like; reference recording with my Ambisonic microphone.

So how does it sound? An Immersive Listening Experience

To be honest, I could have avoided the detailed description here and just let the cat out of the bag: The camera just doesn’t sound good and fails to deliver an immersive listening experience. Who would have thought – it’s a 360-degree camera.

In my opinion, the built-in microphone capsules are the biggest problem that sounds more like plastic, metal and GoPro rip-off. Some time ago I was able to test the VUZE 3D-360 Grad-4K camera, which also has four micros in a similar way. However, the sound is so poor in quality that it is virtually unusable and the capabilities with regard to spatial audio are therefore irrelevant. I tried it anyway, and well, I could have skipped it if I hadn’t enjoyed it. Back to Insta.

Nevertheless, a single production error may well have played a role here, a camera colleague told me about a blurred camera lens. Of course, I didn’t miss the opportunity to test the 4 microphone signals individually in addition to the Ambisonic signal.

Surprisingly, I noticed that one of the capsules sounded OK, one was mediocre and two had a strong reduction at about 5 to 10 kHz. Equalizers come to help, but the problem is that the stitcher can only work with the original files, which causes the defect to be converted to ambisonic without hesitation, causing irreparable damage to the 3D sound. Capsule 1 looks actually quite good. Capsule 2 has a significant reduction in the upper frequency range.

While capsules 3 and 4 do not actually deliver usable sounds above 5kHz…

Audio samples would be appropriate at this point, but since this blog entry has become much longer than expected, they have to be left out for the time being. If you are interested, please write an e-mail, but who wants to hear bad sound?

Conclusion:

Actually, I wanted to take up the most important points again, but I had a lot more points on the contra than on the pro side. That’s why I would like to stress that it is a good idea in general, which is still in its infancy. Although the overall result was sobering, I think it is a step in the right direction. Detailed questions can be answered by Insta’s support on Facebook for its 360-degree camera.

The advanced video options in the camera settings of Galaxy phones are crucial for enabling features like 360 audio recording. Users can tap advanced video options to toggle on this feature, enhancing their video recording experience.

Galaxy Buds 2 Pro are compatible with 360 audio recording, allowing for spatial audio capture. This feature works seamlessly with Galaxy phones, providing an immersive and dynamic audio experience.

Various Galaxy smartphone models, such as the Galaxy Z Flip 4 and Z Fold 4, support 360 audio recording. Samsung Galaxy Buds 2 Pro play a significant role in enhancing the immersive sound experience when paired with these devices.

To enable 360 audio recording on a Samsung phone, ensure both the phone and the Galaxy Buds 2 Pro have the latest firmware updates. Then, navigate to the Camera app settings, tap advanced video options, and toggle on the 360 audio recording feature.