Ryan Staake has somewhat of a knack for knocking out killer music videos without the presence of the actual musician. First there was his promo for Young Thug's 'Wyclef Jean' in which the rapper's surprise absolute lack of presence on set and some genius thinking from Ryan led to one of our favourite music videos of recent years. And then there's 'Cross Me', this trippy, CGI heavy glitch bomb for Ed Sheeran and Chance The Rapper. Both artists do star in the film, but only digitally rendered versions of them. The film is set in a motion capture stage and uses the technology as the focal point of the narrative, a female dancer flicking in and out of the real world and all manner of digital versions before things take a slight dystopian turn at the end. What’s more, Ryan and the crew at Pomp&Clout also created a Snapchat filter to hype up awareness of the film’s launch (we’ve got some cheeky footage of Ed trying it out below but you can download it here).
Check out the film below and then learn how the hell Ryan and the wizards at MPC made it. LBB's Addison Capper spoke with the director and MPC's CG supervisor Dominic Alderson to find out (and picked up some ace behind the scenes footage along the way).
LBB> Where did the idea to reference the motion capture aspect first come from? From a filmmaking perspective, that's usually something you look to hide! Were you inspired by VFX breakdowns at all?
Ryan> I’ve always been interested in the look of a motion capture stage as a location, there’s something so visually pleasing about the austere minimalism, pragmatic colours of tape on black, and reflective balls on the suits. I’m a huge fan of VFX breakdowns and have always geeked out on the look of the various passes, the bones, the pre-renders, the lighting passes, etc. It’s such a clear way of showing the step-by-step process of something being created.
LBB> Any why was it a good fit for this song in general? What were your thoughts when you were first approached by Ed Sheeran?
Ryan> I honestly didn’t think he’d be interested in this idea and was incredibly happy when Riff Raff & Joceline Gabriel told me he was. It’s a bit of a fringe idea for anyone, and the fact that a pop powerhouse like him was into it was incredibly exciting. Lyrically, I was playing off of the “cross me cross her” line, almost re-interpreting its meaning to play with the idea of crossing from one reality to another, or one person controlling another. It’s a bit of a stretch, but I think it plays well. Furthermore there’s the fact that the lyrics are about important women in Ed & Chance’s lives, and it was interesting to give a female dancer the power to quite literally control male performers.
LBB> In your email to me about the film you said there were "some really interesting methods and ideas at play here". I couldn't have worded it better myself! From a technical point of view, can you tell us more about how you pulled this off?
Ryan> MPC used every tool in their toolkit on this, which was fascinating to see. I'll ramble for a bit… we tracked both Courtney and the Steadicam rig within the motion capture volume, which allowed editor Sam Bould and I to cut from live action to CG at any time and have a pretty close match cut. As far as scene modeling, the scenes were created in a mix of Maya, Houdini, Cinema4D, Unreal Engine, and probably software I was never even told about. Full body models of Ed, Chance and PNB Rock were created from photographic reference, a bit of stock and a trip to the Ed wax statue at Madame Tussauds. We did a bit of secondary lip sync using an iPhone X as a facial tracking camera to give a bit of life to the models. We made a trip to visit Ed on the set of another video and took more photo references of his arm tattoos because we couldn’t find good enough ones online. Tons of different wireframe/process passes were rendered out to allow us to glitch between different looks in Flame in both the performance sequence and the fourth-wall breaking ending sequence. Rumour has it that the bloated project files were beginning to crash Flame by the time we were delivering.
LBB> Dominic, you were involved from the off on this project - how did that aid the process? What was the process like with Ryan?
Dominic> Cross Me is one of those examples where timing played a big part. It was the first time I had worked with Ryan and flame op Zdravko Stoitchkov so we set ourselves up in a suite and bashed over Ryan's brief and thumbnails and the creative juices were just allowed to flow - suffice to say we got on well. There was literally no interference at this stage and I think this was partly because Ryan's idea was so whacky and experimental, making it harder for someone to judge the work at an early stage. It was very much a case of trusting the process.
Usually when working on an advert or film, it’s quite clear how you go about creating a visual effect. Most of the time it’s something realistic and clearly plotted in storyboards or moodboards, so it can become second nature. In this case we really did have a chance to play in an open sandbox and throw in ideas and solutions given we were on a very tight deadline. It was like being back at art school and having the chance to experiment, only this time with top talent at your disposal. Ryan clearly knows what can be done in this growing, technically driven art form and that a little trust in the artists will allow an idea to grow and find its feet. This is quite rare and to some risky but the results speak for themselves.
LBB> From a technical perspective, what processes were involved? What particularly innovative aspects were there? I heard that gaming tools and even iPhones were used...
Dominic> This year has seen MPC undertake quite a bit of character work with Elton John's Christmas ad, Cross Me and another very big character job currently in production. To that end we are developing new techniques to achieve detailed lip sync and character animation. In this case, to achieve the level of lip sync we thought we'd need, we used an in-house app developed for the iPhone that takes the technology for the 3D emoji tool and re-purposes it to capture the data of a moving face. That data is then fed into our face rig tool which has over 52 expressions. The face rig tool then uses the captured data to figure out which expressions need to be triggered and how much of each should be dialed in.
The icing on the cake is that we are able to transfer these expressions onto different heads, which in this case was very necessary. From an animation perspective the motion capture was very clean so we spent only a week in clean-up mainly fixing arms, fingers, legs and the odd neck movement. The camera was however not tracked to a level that would work in production so we had to rely on our talented camera tracking team to track a new camera, which meant nine minutes of tracking as the edit is made up of mainly three different takes. The 3D was developed in Houdini, Maya and Cinema4D which meant we had to make sure we had our three hero characters working in both Houdini and Maya and that, when rendered, they looked the same. Having Arnold at our disposal made this process less painful. Caching out animation, adding lip sync and getting the timing right was certainly a massive undertaking.
LBB> What was the production process like?
Ryan> I’d created a second-by-second spreadsheet of the sequence with sketches and little descriptions of beats in the choreography as well as what we were planning to animate. I think everyone on set thought it was hilariously nerdy to block a dance with a spreadsheet, but it worked! In the end, a good amount of the animation evolved and changed as we started to see first passes in post. For example the ‘gold queen’ scene was initially going to be a massive queen bee, but it didn’t seem to match Courtney’s motion well enough so we went with the gold scene.
Dominic> Like most jobs we start with a live action edit and add in VFX requirements from there, and in this case, that was no different. I must say though, the edit of Courtney's dance was so good that we actually didn't want to put anything over it initially! Ryan's idea was that the video was meant to play like a single take, but there are a number of clever cuts in there to make the action more dramatic. We then started going through the edit, breaking down into internal shots, colour coding over the video wherever we thought we'd cut into CG.
The edit then just slowly took on its own life. We'd update shots with Ryan's sketches or a reference image and then repopulate time and time again with new material, firstly with just the motion capture on a female CG model and then slowly with the CG work we were developing for each shot. This way, at the end of the day we could quickly start to see the development and progress we were making and whether it was working with the audio track. It was a very creative process, a lot of fun and it kept the momentum going - which was important given we had three-and-a-half weeks of production.
LBB> There's a lot going on in the digital elements of the film and such a variety of things happening. What can you tell us about them?
Ryan> I liked the idea that the characters and environments would kind of switch roles at times. For example the deflating gold figures at the beginning allude to the gold queen we’ll see later, the car that the muscle person lifts has a ‘little’ cameo later on, the coloured particles are both an environment and a character and then simply a character. Looking back, a lot of these scenes are visual ideas I’ve had rattling around in my head for some time and have finally spilled out into this visual trip. I did find that I was describing a lot of these scenes to the animators with movie references: “the heart pulses like in The Mask… the arrows all hit like in 300… it’s a mirrored dance duet like the end of Annihilation… the phones all fly onto him like the goo that goes down Neo’s throat in The Matrix… lasers like in Entrapment… motion lines like in Akira… etc.”
LBB> From a choreography perspective, how much direction did you give to Courtney? And how did you find working with dance along with the technical elements of the animation and motion capture?
Ryan> Erin Murray, our choreographer, did an incredible job of interpreting my ideas (and spreadsheet) into a tightly choreographed routine for Courtney. There was an incredible freedom at points where we could say, “okay she walks up stairs, and they’ll turn into… maybe a hill, maybe arms, maybe boulders, we’ll figure out the details later, we just need her to walk up something”. This allowed us to choreograph quickly, which was critical because we only had two days of rehearsal before shooting.
LBB> When considering the character design of each artist, is there a particular era / platform that you looked to for inspiration?
Ryan> I gave the reference of “a modern PC game running on a top of the line graphics card”. I think MPC’s team went beyond this to hit a sort of hyperreal look in which we made it clear that this wasn’t intended to be photoreal, yet had a ton of detail and humanity.
LBB> It all gets quite dystopian at the end! What inspired the actual narrative?
Ryan> I felt the performance idea for a full 3:30 would just be too long and drawn out and wanted to switch it up with something for the final third. This idea of what’s real/what’s fake seems to be one of the most important themes of our times (both in politics, media, science, climate, etc.) and I wanted to explore that idea loosely. It was also just a really fun way to take the story out of the mocap studio into a slightly more gritty, realistic world. We alluded to this idea of little CG remnants entering reality throughout the edit as well, a little playful foreshadowing.
LBB> Are there any hidden messages / easter eggs hidden in everything?
Ryan> When Courtney is waiting for the subway, she sees someone on the train… guess who?! There are Ed related posters on the train. I wanted to add in a 3D scan of myself into the performance finale, but we ran out of time.
LBB> You also created a Snapchat filter to promote the promo - what can you tell us about that? How does that kind of work fit in with what you're doing with Pomp&Clout?
Ryan> We founded Pomp&Clout as a bit of a nexus for the mediums of film, technology, design, etc. We’re at our best when we find ways to weave different mediums together, and are finding that viewers/labels/agencies/brands are responding to this approach. To us, the different social and mobile platforms we all engage with offer unique opportunities for creative experimentation and unique viewer experiences. Mobile specifically deserves more than simply editing a portrait version of a landscape film… these apps have intelligent, context-aware AR camera algorithms backed by millions of dollars of R&D. It’s an infinitely exciting creative playground to explore and we’ve jumped in head first—stay tuned.
I previously worked at Apple as a UI designer, and recently linked up with a former colleague to create an AR/location based experiment called Mirage
, so I’m quite at home working with AR & mobile.
LBB> What were the trickiest components and how did you overcome them?
Ryan> The label was very concerned over which moves we would show Ed doing versus Courtney. Luckily this format allowed us to elegantly ebb and flow from one to the other.
Dominic> By far the trickiest job from my perspective was dealing with all the data from the motion capture, the lip sync and the character rigs and managing this so that the artists could quickly and efficiently pick up caches and work without being slowed down or hindered by technical issues. This meant a lot of spreadsheets recording edit information and frame ranges and a lot of painstaking back and forth doing quick playblasts to check animation and lip sync alignment in the edit. Getting our CG characters to line up with the dancer was also a tricky process requiring a lot of manual edits to the CG scenes.
We also needed a fast way to switch between characters so our character rigs had to be efficient in that we could reference in any of the characters whether the dancer, Ed or Chance the Rapper and they would all work seamlessly with the animation / mocap data. The trickiest character was definitely the rainbow particle lady and she was developed in Houdini which allowed us to control the movement and size of the particles but also how they should interact with the environment, be it colliding with Ed or falling with gravity.
LBB> Any parting thoughts?
Ryan> Our team at Pomp&Clout made a Snapchat AR filter to go along with the video and it was #5 on the Snap Lens community, you can download it here
And here’s a video of Ed trying it:
Dominic> Cross Me is one of those rare projects in that it really allows artists to open up their tool box and explore the possibilities. It's these kinds of projects that push us into new directions so we are thankful for the opportunity and hope that more directors take note. VFX artists are not just technical operators, they are a very unique breed of creative artist and technician and they know so much about the entire filmmaking process, from camera lenses to composition and of course making the unimaginable. As the industry continues to develop and grow, it's my belief that this kind of artist will only become more instrumental. Thanks again to Ryan and RiffRaff and, of course, Ed Sheeran for trusting in us.
Directed: Ryan Staake
Starring: Courtney Scarr
Production Company: Riff Raff Films
Executive Producer: Natalie Arnett
Producer: Tom Knight
Directors Rep: Joceline Gabriel @ Hands London
Production NYC: Pomp & Clout
P&C EP: Ryen Bartlett
P&C HOP: Kevin Staake
Line Producer: Andrew Chennisi
DOP: Kristian Zuniga
Choreographer: Erin Murray
1st AD: Jeremy Nachbar
Steadicam: Michael Klein
Commissioner: Dan Curwin
Edit: Sam Bould at Cut & Run London
CG Supervisor: Dominic Alderson
2D Supervisor: Zdravko Stoitchkov
Producer: Sandra Eklund
Colorist: Duncan Russell