Gear Seven/Arc Studios/Shift
I Like Music
Contemplative Reptile
  • International Edition
  • USA Edition
  • UK Edition
  • Australian Edition
  • Canadian Edition
  • Irish Edition
  • German Edition
  • French Edition
  • Singapore Edition
  • Spanish edition
  • Polish edition
  • Indian Edition
  • Middle East edition
  • South African Edition

Behind the Work: James Frost on a Music Video Created Entirely in Unreal Engine



Director speaks to LBB’s Ben Conway about collaborating with RÜFÜS DU SOL on an awe-inspiring video that isn’t quite as it first appears

Behind the Work: James Frost on a Music Video Created Entirely in Unreal Engine

RÜFÜS DU SOL continue their legacy of creating immersive worlds for their fans to explore in the form of a new official music video for their new single, ‘Alive’. Created by renowned director James Frost, the film gives the audience a birds-eye view on a surreal virtual ‘flyover’ above stunning scenery that ranges from snowy peaks to rugged mesas and everything in between.

The big reveal, however, is that every inch of the vibrant, photorealistic landscapes have been constructed entirely using Epic Games’ Unreal Engine - a tool that gamers will be very familiar with and is beginning to be used more in the art of filmmaking. 

The director is no stranger to using emerging technologies and visual trickery - his ‘House of Cards’ video for Radiohead was the first music video to be premiered by Google and was the first to be shot without the use of any cameras, opting for 3D scanning systems and LIDAR laser technology.

Combining a longing for the outside world - during a pandemic that separated us from the natural landscapes represented in the video -  with a desire to push photorealism to its limit, James has produced a video that not only soars over the uncanny valley, but also flies convincingly to the liberating energy of the new single.

LBB’s Ben Conway spoke to James and discussed using the Unreal Engine, the leaps of faith required for music videos, and the trials of remote working.

LBB> What was the starting point for this video? Did RÜFÜS DU SOL already have a concept that just needed narrowing down, or did you all work together on the initial idea?

James> I had worked with the band on a film for their track ‘Underwater’, which screened at Coachella in 2019, and we just stayed in touch. Around March this year, I started chatting with their creative director, Katzki, about what types of visuals the band might want for a video. There was no music, but our chats grew organically - from throwing around ideas and techniques, to me sending a 35-page mood board. A lot of what I sent was based on digital manipulation of imagery. This quickly evolved into sending images and video clips back and forth, and really, this idea of ‘Analog vs. Digital’ kind of naturally evolved out of what the band was initially thinking.

LBB> Where did the idea of using photo-realistic environments come from? Was the ‘reveal’ inspired by your analogue vs. digital concept? 

James> I think the soaring landscape concept was narrowed down as we went along. Initially, we looked at all sorts of different ideas and concepts. They included structures such as bridges and cityscapes with human elements, and then the reality hit that there was no way financially or practically (considering we are still in the middle of a global pandemic) to really achieve this. We thought about hiring people in different countries to shoot certain things, but that would have been a logistical nightmare, with very little control over what we would get. Also, the VFX required on the back end would have been impossible to achieve in the given time frame. Everything started to point toward Unreal as being the only viable way to pull it off. 

Having studied landscape photography, my biggest fear was making sure it would look real. I needed convincing. I have an old friend who heads up Innovation at Epic Games in London, so I talked with him and picked his brains about it all. I was sent some stuff that convinced me we could pull it off. As we went along, it started to become more evident that by removing the human element and really focusing on this incredible landscape scenery, it would make the switch in the reveal that much more impactful. It’s an observation of our natural state and how glorious it is, and then how we can manipulate it and deconstruct it.

LBB> Was this your first major project using the Unreal Engine? What are its main benefits for creating films?

James> It was the first time. I enjoy learning about new technologies and how we can use them in storytelling. I believe that the Unreal Engine will become a powerful tool for filmmakers in the years to come, and wanted to get stuck in there and learn. 

As for why, there are two reasons. One was practical, which I mentioned earlier, and the other was that it allowed me to explore the digital asset side and strip back layers of reality to reveal what a beautiful world would look like as a wireframe, its bare basics. There is something strangely ethereal to me about that.

As for benefits… well, as the technology becomes more mainstream and more artists learn how to use it, I think it will give filmmakers unprecedented power to create scenes that would be budgetarily impossible. It literally provides the ability to create totally unrestrictive worlds in terms of how you present them to an audience. Right now, we are restricted by location, time, schedule, technical restrictions in terms of camera, or what’s possible in visual effects and so on. All of that could potentially be gone. Not to say suddenly everything will get magically easier - that would be foolish - but just the way we approach things, it becomes another tool in the toolbox.

LBB> The visuals and movement seem to embody the music itself. What did you first think when you listened to the track, and how did you see it translating into visuals?

James> When I was first sent the music, I understood the band's yearning for forward momentum and feeling free. Until this point, I had no music to reference. We'd been watching a lot of POV drone videos and footage with cameras attached to birds of prey, basically falling into that infamous rabbit hole on YouTube you tend to go down, and these majestic landscapes just fit perfectly with the music. Hearing the track allowed me to build a linear story based on visuals.

LBB> The ‘reveal’ of the environments’ computer-generated, geometric underbelly, and the virtual camera soaring through it visually reminds me of your ground-breaking House of Cards video and its LIDAR landscapes. What do you think about that comparison? And how has technology changed in that timeframe? 

James> The only comparison for me is the visual on the back end, as both represent very different things to me. ‘House of Cards’ was made using data to create an image. Everything we scanned existed, and we see a point cloud data version of that. This is the complete opposite, everything is data to represent reality, and then we reveal it for what it is. 

It's hard to compare the two technologies, as they serve different purposes. LIDAR is used in so many industries; obviously, we use it in production to aid with VFX. But it's been around since the 1960s and is used in many other ways, from autonomous cars, geometric mapping and topographic mapping, all the way to military applications. It’s now even on our smartphones, to help us create maps of our homes and whatnot. 

Unreal exists as a gaming engine, so its development is based on making the engine better and better over time, with the sole purpose of video gaming. It's now got to that point that the way we integrate it into film production will be exciting.

LBB> Was making people initially question whether what they were looking at was real or CGI a conscious part of the process? What did you want the audience reaction to be?

James> I think to a degree, yes. Certainly, as we got further and further into building scenes, I became more conscious of the quality of the asset and whether it felt ‘real’ or not. At the beginning of the process, it was about creating beautiful visuals that spoke to the feeling of flying and feeling free, and the notion of ‘look how beautiful the world is’ and feeling alive and energised. But to say to the audience what you've been watching is fake, although not in the ‘you just got duped’ way? Well... maybe a little.

LBB> Which do you find more rewarding, using CGI and creating a photorealistic virtual environment that fools people, or creating a practical phenomenon like the ‘This Too Shall Pass’ Rube Goldberg Machine that you worked on? 

James> I mean, they are so different. Both offer a great deal of anxiety, stress, sleep deprivation, satisfaction, relief, pride, and every emotion in between. But these types of projects are always a challenge because you're doing something that requires a lot of time and energy.  A certain leap of faith is needed by everyone involved, whether it's the band, the label, or an agency. They are very mentally taxing, and a lot of discovery goes into creating them - whether it’s creating assets in the computer, or designing and building set pieces that interact with each other or people. As long as everyone is on for the ride, then both are equally as rewarding.

LBB> How has the ability to create these virtual scenes in real-time - rather than having to wait to see your results - affected or improved the process? 

James> That’s a little bit of a myth, as the creation of the assets still takes time. Whether you’re downloading pre-existing assets or building from scratch, it takes a fair bit of time to map out and build a scene. Once those are made, this is where Unreal really comes into its own. It allows you to create variations of the scene in real-time, such as changing where the sun is, removing assets like rocks or moving trees, adding atmospherics. So in terms of improving the process, it means you can change things without having to make decisions that mean having to wait hours and hours for it to render.

LBB> Considering you made this entirely in the Unreal Engine, could you talk us through that process?

James> The process is really like building a house, I guess. You start with the blueprint - my treatment where I had actual time stamps and ideas like the 360º camera move with the sun setting and rising. My editor Nicholas Wayman Harris and I sat for a few days building an edit that would serve as a timing template for the entire video. So, we spent a day pulling references of all different landscapes, scouring the Megapixel site for the existing landscapes that Epic has made available, using existing video footage, and slotting them into a timeline. Ideas were discussed and a lot of changes were made during this process. Working closely with Nicholas, working things out, and sounding ideas was crucial. 

Essentially, we presented an edit to the band that consisted of just still images and written descriptions of what would happen in that shot. From that point, the process involved Unreal artists finding and building assets into the timeline. We worked with Joey Hunter and Claudia Brand, who painstakingly pieced together the assets. Then, once we had filled the timeline, it became about establishing the camera and what the camera was doing. 

We knew early on that we wanted the movement to be a crucial part of the video, so we had to match the horizon lines at the end and beginning of each shot, so it felt like a continuous shot. Then it was lighting and atmospherics, time of day, weather, and all the fun stuff to build the final image. 

It was quite the organizational exercise, as every time a new scene was created, the team in Sydney (Joey & Claudia) would post the raw FBX files from Unreal along with a QT reference file. We would drop the QT into the off-line, and the FBX would go to the VFX to build the back end. We worked with Jane Studios here in Los Angeles for the VFX, spending an intense few days messing with the wireframes and putting the final minute of the video together.

LBB> What were some of the challenges with creating the video? How did you overcome them?

James> Time. When you start something like this, you think the timeline makes sense, and you can spend time researching and experimenting. But there is never enough time. 

We had some technical hiccups, and the back end got time-crunched a little more than I would have liked. Still, as with any project, you realise you have this small team of people who have a lot invested. So as much as there is tension or urgency, and all the other emotions that come into play when people are up against a deadline, there is genuine camaraderie amongst the team to push it an extra step to get it done. 

Obviously, working in different time zones and remotely was a challenge. We had our Unreal guys in Australia, our visual effects in Los Angeles, and technical support in London. I was in Croatia at one point, so that made it really interesting there for a week or so - with Zoom calls with everyone in different places. Let's just say I got used to my working schedule, starting around 10 a.m. each day with the previous day’s adds to the off-line edit, and then Australia would wake up around five p.m. my time, so I’d often finish up around two or three a.m.

LBB> What was the inspiration and purpose behind each ‘region’ that the video shows? Is it to show off the capabilities of the Unreal Engine? Or were they more thematic decisions?

James> Most definitely the latter. At the beginning of the process, you randomly throw things into the timeline, but I had an idea of what I wanted to do and where. But you see that certain things don’t complement each other, and you say to yourself, ‘What if it was mountains instead of the desert?’ So you change things around until it feels right. An example of this was that I wanted the video to start in a forest and build the first minute to a big landscape shot. It became pretty clear that opening on a majestic wide felt much more epic (excuse the pun). That also highlights the flexibility of working with Unreal.

LBB> What are some of the challenges, benefits and things to consider when using a virtual camera in a CG environment? Especially considering remote working. 

James> There is always an element of being in a room that helps the creative process. Having the ability to point at a screen and say, ‘The camera moves from here to there’. Doing that remotely was a challenge. We’d do Zoom calls and analyse every shot in the video to adjust and try several camera passes before getting it right. 

But in terms of the creative side, yes. It allows you to put the camera in places that would not be possible with a conventional camera or would take days to coordinate and rehearse. So there is a huge benefit when doing a project like this one.

LBB> How do you see the future of the Unreal Engine, concerning creating films instead of games? Is it something you look forward to working with again?

James> For sure. I think getting into how the two can complement each other now, both creativity and logistically will only broaden its usage. It was obviously built for gaming, and that’s their core business, but I know they are very keen to start incorporating it in the film and TV space - which makes total sense given the speed at which we are moving technically. 

You’ll see it more and more, but where it is in, say, 10 years? I don’t know. I’m certainly a little torn, as nothing beats an actual location and the ambience of being present within natural elements that translate onto the screen. Still, when the level is getting so realistic, and if it becomes competitive financially, the shift will happen based on that alone.

LBB> Before you go, would you like to add anything?

James> A small group of dedicated artists created this video, and I have never been in a situation where management and label were so hands-off and forgiving. I want to acknowledge them for allowing the art to be art.

Shout-out to:
Katzki and Rüfüs Du Sol, Danny, Irene & Cameron at Team Leirsury. 
Devin Sarno, Natasha Killbarda, Ashley May, Rob Levin at Warner Records. 
Claudia Brand & Joey Hunter at Entropico. David Parker, Rich Rama at JANE. 
Matt Osborne, Blake Rice at Company 3. 
Nicholas Wayman Harris, Michael Raimondi & Julia Armine. 
Phil Crowe & Jess Ambrose at Parliament.

view more - Behind the Work
Sign up to our newsletters and stay up to date with the best work and breaking ad news from around the world.
LBB Editorial, Tue, 21 Sep 2021 16:27:00 GMT