What is immersive audio?
Immersive audio is audio that surrounds you. For me, it means being surrounded by the sound field — being able to hear that something is to the right or left, in front of or behind you, and whether it is above or below you as well.
That sounds simple, but our brains do a surprising amount of work to figure out where a sound is coming from. Our ears supply several different cues at the same time, and our brains put them together almost instantly.
How our ears locate sound
Let’s quickly break down how that works. Our ears have three basic parts: the outer ear, the middle ear, and the inner ear.
The outer ear is what you see and is called the pinna. Everyone’s is shaped a little differently. It directs sound into the ear canal, but its shape also EQs the incoming sound depending on where that sound is coming from. Move your head up and down and those frequency cues change. Interestingly, several species of owl have asymmetrical ear openings, which helps them locate prey in three dimensions. Ours are symmetrical, so we have to do this a little differently.
The middle ear is kind of like a biological impedance converter. It starts with the eardrum and three little bones called the ossicles. Together they help couple sound efficiently from the air into the fluid-filled inner ear.
The inner ear contains the cochlea, a fluid-filled structure containing the hair cells that convert vibration into nerve signals. Right beside it is the vestibular system, including three semicircular canals that sense rotation and help your brain keep track of the orientation and movement of your head. Hearing and balance information travel toward the brain through separate branches of the same vestibulocochlear nerve.
We also have two ears, spaced roughly 17 cm apart. That means a sound coming from one side usually reaches one ear slightly before the other.
Our brains use several cues to figure out where something is. One is interaural time difference, or ITD. The human auditory system is incredibly sensitive to timing. Under favorable conditions, we can detect timing differences on the order of 10 to 20 microseconds. We can also detect very small changes in source direction near the front of the listener — on the order of a degree or two.
A second cue is interaural level difference, or ILD. Your head gets in the way of sound traveling to the far ear, particularly as the wavelength gets shorter. That creates a level difference between the two ears.
Then there are the frequency-dependent cues created by the pinna. These are more subtle, but they become especially important for telling whether something is above or below us and for resolving front-versus-back ambiguities.
This is also much more interactive than it first appears. When a bird chirps in a tree across the street, you probably turn your head toward it. As you move, the timing, level, and pinna-related frequency cues all change. Your vestibular system tells your brain how your head is moving, and your eyes see the tree the bird is probably sitting in. All of that information arrives together, and somehow you hear exactly where that bird is.
Binaural playback
So, how do we reproduce that with recorded audio?
One approach is binaural playback. In theory, we can reproduce a three-dimensional sound field with only two audio channels feeding headphones. The challenge is that the filtering produced by your head and ears is unique to you. The combination of head shadowing, ear shape, and other anatomical effects is described by your Head-Related Transfer Function, or HRTF.
An HRTF can be measured, and that information can be used in a binaural decoder. The closer the HRTF is to your own, the better the localization can become.
There is another challenge: head tracking. In the real world, the sound field stays put when you turn your head. To create the same illusion with headphones, the decoder has to know that your head moved and rotate the virtual sound field in the opposite direction.
I have heard this work really well. At a recent AES event I heard a system that used multiple cameras and infrared tracking of the headphones and measured all six degrees of freedom. The illusion was very convincing.
Apple AirPods also use head tracking, and personalized HRTFs are becoming more practical. Even so, binaural playback is still inconsistent for me. I can often hear horizontal placement, but I personally get very little height information from many binaural renderers. Sometimes the vocal appears to move while the surrounding bed stays relatively static. So I think binaural is extremely promising, but I do not think it is yet a universal replacement for loudspeakers.
Immersive loudspeaker playback
The other approach is to put speakers around the listener.
A good immersive loudspeaker system can sound amazing. Once you hear a really good 7.1.4 playback system, stereo is just not quite the same.
Historically, we went from mono to stereo: one channel, then two. But people were experimenting with immersive playback surprisingly early. Disney’s Fantasia used a system called Fantasound, with left, center, and right playback plus additional auditorium speakers and control signals that could move sound around the theater. It was technically complicated, expensive, and installed in only a small number of theaters, so it was more of a roadshow system than a permanent standard.
Later came surround formats such as 5.1. The “5” is the number of main ear-level channels: Left, Center, Right, Left Surround, and Right Surround. The “.1” is the Low Frequency Effects, or LFE, channel. That channel is normally reproduced by a subwoofer, although it is not quite the same thing as bass management, which can also route low-frequency content from the main channels to a subwoofer.
With 7.1.4, we add two more ear-level surround channels and four height channels. So we have seven ear-level channels, one LFE channel, and four speakers above the listener — usually front left, front right, rear left, and rear right.
Those four height speakers change the experience dramatically. Now a sound can move not only around you but above you. Reverberation can occupy the room instead of simply coming from the sides. A pipe organ can feel like it is filling a large space. A train can move through the sound field instead of simply panning from one speaker to another.
That, to me, is when playback really becomes immersive.
Capturing the sound field
Of course, playback is only half of the problem. We also need to capture the original sound field in a way that gives us enough information to reproduce it.
That is where techniques such as Ambisonics, Mid/Side arrays, ORTF3D, and the microphones developed here at Sound Sleuth come in. The goal is not simply to create more channels. It is to capture enough information about the direction, level, and character of the original sound field that we can reconstruct a convincing sense of space later.
That is the part I find really interesting: building the microphones, recording with them, decoding the results, and then listening to see how close we can get to being back in the original space.