Wrapped in Sound: Approaching Immersive Audio in Today’s Technoscape

Imagine you’re watching a movie with a jungle scene. There is the steady hum of wildlife all around you, a monkey screech echoes off to the left, and there’s a rustle of the leaves in the canopy as something moves quickly above and behind you. Instead of hearing it all coming from the same place, spatial, immersive, or 3D audio allows you to hear it coming distinctly from different directions and positions. It makes the viewing experience feel as if you are actually part of the scene.

The most well-known immersive format is Dolby Atmos, initially developed for the film industry. I remember the first time I heard a Dolby Atmos demo. Listening to songs like Elton John’s ‘Rocket Man’ and other tracks re-mixed in Dolby Atmos is a very different and incredible experience. The clarity, positioning, and panning of the instruments and vocals in the virtual 3D space creates the feeling of being inside the mix, as opposed to being in front of it. It is the addition of height channels with immersive formats that provide a more spherical sound field.

At the time, I wished everyone could hear mixes this way. But fast forward a decade, and now, the tools for professionals and consumers alike have made that possible.

Compact, sophisticated sound bars from Bose, Sonos, Samsung, Sony, and others have brought immersive audio into living rooms without a roomful of speakers. And thanks to Dolby’s partnership with Apple, spatial audio can even be reproduced with an iPhone and AirPods Pro (2nd Gen) or AirPods Max.

It is the ability to experience immersive audio with headphones that enables us to share Dolby Atmos mixes online. We’ve done this with brands like NASCAR for the introduction of Next Gen cars. Instead of shots of drivers roaring by at 200 miles per hour, the spot features slow pans of the impressive cars, while the music and sound design take you on a sonic rollercoaster.

The low end impacts that start the spot utilise the entire Atmos soundscape, enveloping the listener while the engine ominously growls, serving as a launch pad into the music and mayhem. The sound design creates movement that propels the listener along a virtual racetrack. The cacophony ramps to a finish with detailed sounds and panning, complimenting the intricate graphics of the NASCAR logo.

While companies like Netflix helped get the ball rolling, there’s still a lot to look forward to. Updated broadcast specifications for live television are going into effect around the world, delivering 4K ultra-HD video, along with immersive audio.

I’m inspired by where it’s all going and grateful for the talented, dedicated teams helping us make our way forward. And of course, we couldn’t do it without our amazing clients - collaborators in the mission to create exceptional work while having fun doing it.

For immersive audio in 2023, it’s really about creating content - and a sound design - for platforms where people are already consuming media. VR is still the strongest use case when it comes to the application of 360 spatial audio, but while it’s had its ebbs and flows in popularity over the past number of years, it still hasn’t truly taken off in a meaningful way. Since Dolby Atmos mixes can be heard not only in theatrical settings, but in Atmos-enabled headphones and earbuds, this binaural audio application of immersive sound design has a much wider potential audience.

The main difference between the two is that with 360 audio, there is head tracking that creates the sound all around you, and with binaural audio, the viewer is in a fixed position, with the audio being manipulated to sound as if it is happening all around them. Even in web3 and metaverse experiences, most of the audio is crafted in a video game engine and the audio is attached to a specific object, character, or environment. You are only using 360 audio for a video that is made to be viewed that way.

In an Atmos-enabled theatre, the viewer is sitting looking at a screen and the various speakers in fixed positions. Binaural audio uses phase and delays to have the sound perceived to be coming from a specific location that gives the feeling that there are a bunch of speakers. One of the reasons I think there is much more of a use-case for binaural audio and Dolby Atmos is that the speakers and headphones are so accessible. We are constantly creating audio files optimised for phones, laptops and platforms that most consumers are using every day - all of which can help elevate the audio experience and heighten storytelling.

The latest installation at the Science Museum (‘Science Fiction: Voyage to the Edge of Imagination’), gave our immersive audio team here at Factory the opportunity to geek out over all things science and science fiction, whilst using audio technology to help maximise the impact, creativity and experience for visitors to this awe-inspiring new exhibition.

With the use of 360 degree soundscapes and interactive installations, we transported guests to an alien spaceship/world, and as part of the sound team, I worked across the design, mix and implementation for all of the audio within each gallery. The soundscapes that play throughout the exhibition are designed to enhance the unique character of each zone, all of which are filled with hidden treasures including authentic artefacts from iconic films, TV shows, and books. 

It was great fun imagining and developing the sound of the alien ship (called ‘The Azimuth’) where the experience takes place. We designed it in such a way so each room was interconnected with the preceding one through a base layer of evolving engine hums, serving as a cohesive audio element throughout the entire experience.

Bespoke speaker and installation configurations for each room kept us on our toes. Within the ‘Engineering Zone’, for example, we had six overhead speakers, two subwoofers, five media screens and two interactives to consider as we approached the design and mix. 

Throughout the various zones, a fun way to break up the soundscape loops and enrich the experience was to weave in audio easter eggs that panned to speakers nearby the relevant exhibit. For example, visitors will hear a NASA countdown when passing the Saturn V model, and a transporter-style sound near the USS Enterprise. During an off-world visit, guests also get the chance to interact with an alien species. This whole section is built within Epic’s Unreal Engine. Our sound design and music were created and integrated to evolve and react as the guests move within the space and begin to communicate with the new species.

The project required leveraging a combination of immersive audio design, mixing, and implementation tools and techniques. Whilst we finalised the mix on-site - using the museum’s AV show control system - we used our equipment and software at Factory to simulate the spaces and installations to make informed design and pre-mix decisions. By simulating each room either through Unreal Engine or using custom speaker outputs in our Dolby Atmos studio at Factory, we were able to dramatically cut down the time we spent on site.

'Hilda' has lived as graphic novels, a Netflix series and books, so we were excited to work with Audible on adapting the books into an audiobook series. It was the perfect opportunity to utilise binaural sound to create a fully-immersive experience for the listener. The stories feature a vast array of characters and creatures, so there was so much opportunity for creative and fun sound design. Aside from the main narrator, everything else was given the binaural treatment. This offered huge scope for creative ideas, and the movement within the 3D space really helped bring the stories to life. The binaural work was done using a number of spatialisation plugins. For the larger scenes with a lot of action, this really helped to convey a sense of scale, and I found the ability to control height particularly useful in fully utilising the 3D space.

Will> Having two meticulously lined up Dolby Atmos studios with 9.1.6 monitoring gives us a great deal of flexibility in terms of mixing for spatial audio through the Dolby Atmos Renderer - the central element of any Dolby Atmos mixing system. From mixing an orchestral score for a recent feature and premixing the sound design in surround, to creating immersive audio drama work for one of our biggest clients in Atmos and remixing Max Cooper’s latest album in Dolby Atmos, the studios allow us to make informed decisions for a broad range of immersive work.

When creating the immersive mixes for the film of Max Cooper’s album, we theorised about what the spatial structures could be that relate to the concepts within each piece, and continued to refine/adapt these concepts depending on what felt right in the room. We knew that a large proportion of people would be listening to the work on headphones, and some on the new Apple head tracking models, so we incorporated various binaural and immersive head tracking nuggets into the work that can be discovered when listening through this medium. On the chapter ’Awakening’, for example, we created alternative stereo fields in the mix depending on which axis your head is aligned with when listening to the music - so rotating your listening position with head-tracking headphones will unveil a different listening experience. 

Another idea we had was to detach the music entirely from the screen for one piece of music where that bare observational concept was useful as a tool - so the sound and vision are strikingly opposed in terms of where they are in the room. By utilising binaural settings effectively in the Dolby Renderer, this front to back separation still works in the binaural downmix.

Having top-notch studios with Dolby Atmos speaker arrays gives our team a great headstart for working on spatial projects, but ultimately, a large amount of material is for headphones. Understanding the end user’s experience and effectively utilising the Dolby Renderer’s settings is where our experience shines.

Lawrence> Dolby Atmos mixing is still a relatively fresh technology, and we’ve found it fundamental to question and develop the style of surround, immersive mixing, rather than just treating it as ‘surround sound version two’. Being surrounded by full-range speakers (rather than the limited, discrete five or seven speakers in surround) has huge implications for what is achievable in a mix. Developing the listening experience into something that truly puts the listener into a sonic space opens up some amazing creative possibilities, whether that’s in full Dolby Atmos, or a binaural mixdown.

We’ve also been experimenting with integrating our ambisonic and spatial microphones into our Atmos productions, and the results have been excellent. Being involved in the actual initial performance capture - knowing how we will utilise the recordings in the Atmos mix - has led to some incredibly immersive results. It means we can push the limits of the spatial side of the mix in a really efficient, but ultimately more natural and believable way.

We have noticed an increased interest from clients in relation to the request for 3D audio in particular. In audio-only productions, it has become quite a powerful tool in creating interesting and immersive soundscapes, particularly to audiences that are likely to be wearing headphones. 

Spotify is one of the platforms that has pushed for this content in its advertising deliverables, and we now find ourselves creating bespoke 3D mixes of existing creative pieces specifically for that platform. While the use of these tools can certainly amplify most creative briefs, the true power for creating amazing, immersive content is for the creative concept to be developed with that particular output in mind. 

Immersive audio thrives on space to be appreciated, so by working together with a sound house from the creative conception, agencies can ensure the best possible end result.

The ‘Spotlight’ film for Mastercard showcases their ‘Touch Card’, and highlights the importance of accessibility for blind and partially sighted individuals. The creative behind the film was unique, using audio description in the mix, rather than being added after the fact. 

The creative team at McCann wanted to take the opportunity to further heighten the immersive experience of the film through the sound design and mix. After doing some experimentation and research, even though there wasn’t any ability to ship the film in native Dolby Atmos for broadcast, we chose Dolby Atmos as a master mix format, and delivered 5.1 re-renders, and binaural stereo outputs for broadcast and traditional web deliveries. While the spatialisation in the binaural outputs and re-renders weren’t quite as precisely focused as the direct Dolby Atmos playback, working in the format and exporting still gave an immersive experience, while being able to derive several different output mixes from a master Dolby Atmos mix. It’s exciting to see Dolby Atmos and other spatial audio formats gathering more widespread support, and I’m looking forward to being able to deliver higher quality immersive mixes with less friction as the tools and platform adoption continue to grow.