Sonic Boom: The Incredible Potential of AR and VR with Torin Geller
Torin Geller is the technical and creative director of audio at One Thousand Birds LA. When he’s not cooking up new sounds for clients or endlessly tweaking the custom speakers he built over at OTB LA, you’ll find him searching the streets for LA’s best burrito.
AR + VR are mediums for storytelling that hold incredible potential. The level of immersion is simply unattainable in any other medium. In 2017, I was able to experience Disney's star wars VR experience, that now exists at the star wars sections of their theme parks. A four person immersive experience that features full body tracking and a completely wireless experience blew me away, as someone who already had experience working in VR and was highly sceptical of its future as a popular media platform. I remarked afterwards that I could finally relate to the people who ran out of the movie theatre during the first screening of L'arrivée d'un train en gare de La Ciotat in 1896. A brand new medium that has the power to evoke emotion in a completely natural, visceral way. Exhilarating, but also a little scary. Since then, AR+VR has largely faded from the public eye, remaining mainly a niche and bulky hobby. But that changed in a major way with Apple's unveiling of Apple Vision Pro. And while Apple might never use the term 'virtual reality' in their marketing around their new headset, it’s brought VR back into the public sphere in the biggest way since Meta acquired Oculus back in 2014. As a sound designer, this is exciting because first and foremost we are storytellers, contributing to a project’s overall vision through sound. AR+VR allows us to more deeply immerse the listeners in our craft, and it’s especially exciting to work with it in regards to specialisation, and interaction.
While Apple won’t call its new headset a 'VR headset', but rather a 'Spatial computer', it builds upon the many generations of headsets dating back to the 90s. Of course, it’s a very Apple take on the tech, spinning technologies that have existed for some time as a brand new, totally unique thing only Apple could create. But it does deliver some legitimate breakthroughs in the amount of sensors, cameras, and processing power being put to use to create a seamless experience for the user, especially in its AR capabilities. But really, its biggest “innovation” is simply the fact that Apple is releasing it all, all but ensuring another wave of interest in the mediums of AR + VR. Other companies will follow, and a new wave of AR+VR content is no doubt going to come upon its release next year.
VR has come a long way since the first modern boom in 2014. Oculus now offers an incredible stand alone headset, and Sony just released it’s latest headset earlier this year, putting VR in thousands of more homes. As sound creators, it’s exciting that this tech is becoming more popular, as it is the absolute best use case for spatial mixing and experiencing 360 sound tech like Dolby Atmos at home.
During VR’s first boom in the 21st century, from around 2014-2018, we worked on a lot of VR content, mainly in the form of short form 360 video pieces. So much so that we actually changed over our computers from macs to PCs so that we could mix with virtual reality headsets in real time, a feature that was unavailable on macs at the time. It was an exciting time, where film festivals were creating 'immersive film'mcategories, and every major brand was jumping at any chance to create new immersive experiences. Jumping into a new format was very fun from the sound perspective, and the level of detail needed to create convincing soundscapes in VR was an incredibly fun challenge for sound nerds like us. We now had the ability to place each sound in 360 space and then actually interact with the sounds in real time by moving around in real space. Each and every possible angle that the viewer could see needs to be taken into account, and the space to add little details was seemingly endless.
Honestly, the tech is not really the important part here. Simply the fact that Apple is releasing this tech is the important part, as it will generate a buzz that no other company could conjure up. Time and time again apple has released products that aren’t actually as 'revolutionary' as they say they are - mp3 players existed before the iPod, touch screen smart phones existed before the iPhone, smart watches before the apple watch. But they are able to execute versions of these technologies that generally work better than the competition, and are marketed very differently than the competition. Generally, they don’t actually market the specs or details of their technology - the user experience is the important part. This is a huge difference in the way VR is generally marketed as a niche, highly technical gaming device or professional design platform. All of this creates buzz around VR that it hasn’t had in quite some time - and while Apple Vision Pro’s price tag might mean it won’t become a household item for the average person, it no doubt will be a staple of creative media studios everywhere as a wave of new content and apps made for the platform start to appear.
I think there are two major challenges in regards to AR+VR: physicality + perspective. By physicality, I mean the physical realities of the technology. Wearing a VR headset is not a comfortable experience. Having glasses makes it even less comfortable. The tech is still incredibly bulky. The lag in processing time and head tracking commonly induces motion sickness. For high end experiences, being tethered to a high end PC is a necessity. Many times it requires physical space that many people don’t have in their homes. And it’s an incredibly solitary experience. Some of these issues can be addressed with better sensors, lighter and customisable headsets, and more processing power. But some of it is inherent to the medium, and must be considered from the start. While Apple touted watching full length films like Avatar on their headset, would most people prefer that viewing experience of likely sweating into a heavy headset for 3.5 hours that requires either a battery swap after two hours or requires you to be plugged into a power source vs. watching on the couch, potentially with a friend or two, free to move about unencumbered by wires.
This challenge relates to sound as well. Many headsets come with built in speakers that are lacklustre in terms of sound quality to say the least. Users generally can plug in their own headphones, which works, but adds even more bulk and wires to the scenario. Here is an area where Apple Vision Pro could make technological strides, as Apple has a knack for getting impressive audio fidelity from small speakers.
The other challenge for story tellers specifically, is perspective. Martin Scorsese’s quote defining cinema as 'a matter of what’s in the frame and what’s out' is particularly apt when thinking about story telling for immersive mediums. Taken literally, cinematic storytelling is of course what is shown on screen, and perhaps what is implied off screen. VR completely removes the frame from the equation, in a literal sense. The viewer is free to frame their own “shots” by simply turning their head. But more figuratively, it is also the filmmakers point of view, opinions, and choices that create a cinematic experience. By losing control of the camera, we must think differently about how we communicate our perspectives through stories. Simply taking a story and putting it into a 360 film can feel like a gimmick. This is where I think many filmmakers or brands could learn a lot from video games, a medium that’s been telling immersive stories for generations now. There are many examples of incredible stories that simply cannot be told with the same power in any other medium by giving control to the player to either dictate or discover the narrative on their own terms. The user's ability to explore and interact with even a 360 video should be taken into account when making new stories for these mediums. Sound fits into this issue nicely, as sound can be utilised within VR regardless of where the user is looking. Placing sounds smartly can pull the viewer's focus, to find where a sound is coming from, nudging the viewer to more deeply explore the scene.
It certainly remains to be seen if AR+VR tech can truly hit the mainstream, and Apple Vision Pro’s $3k price tag means we’re still likely years away from that reality. But Apple jumping into this world will certainly kickstart a new wave of content creation for the platform that will no doubt lead to innovation in storytelling.
