2D to 360 and Back Again: Jacob Collier’s “Make Me Cry”
Last December, Light Sail VR shot a music video for the incredibly talented Jacob Collier and the first single of his new album Djesse Vol. 2. It was an incredibly rewarding experience and we’re very proud of the finished product. Jacob’s creativity and collaboration is so vibrant throughout the piece and he hid so many fun references for his fans to explore.
While initially this was supposed to only be released as a 360 video, there were concerns about how sometimes people watching 360 videos tend to feel like there is too much to take in and how the story can get lost. We were asked to make a 2D version. The request made a lot of sense. After all, headset adoption is not incredibly high and while engagement with 360 videos is, the view count is often lower than the averages for a 2D film. Personally, I find myself less apt to watch any 360 video unless it is in a headset. The whole moving the phone around concept never appealed to me.
We knew we could do a simple reprojection of the 360 video and call it a day, but we decided since this was going to be the version that probably garnered the biggest audience, we needed to create something more creative. Below, you can watch both the 360 and 2D versions of the music video. These were both created from the same 8K finished lat long master file.
I think what excited me most about this project was the fact that it was an excellent example of shooting a 360 video with a single high end 2D camera and then finishing both the 360 and a reprojected 2D version. It’s something we want more clients and collaborators to embrace. The audience reaction has been very positive.
So without further ado, read on to find out a bit of the behind the scenes magic.
TECHNICAL CONSIDERATIONS
When Jacob and his team came to us wanting to make a music video, one of the first requests they had was that they wanted it to feel cinematic. They had tested the YiHalo and weren’t happy with the results. We love the YiHalo, but know exactly how they felt. There is no comparison between a collection of small cheap action cameras shooting 8-bit 4:2:0 highly compressed H264 and a RED Helium that shoots in raw with 16.5+ stops of dynamic range. It’s night and day in terms of cinematic quality.
Since the creative concept of the music video already had us duplicating several copies of Jacob playing instruments and we were filming in a controlled environment, we knew we could utilize a technique called “nodal capture” to create our 360 image from a single standard 2D camera.
This allows us to stitch each camera angle together with minimum parallax between each quadrant. It’s a clever technique that when done right can still deliver the feeling of having the entire 360 sphere captured at once. The added benefit is that we have a very high quality image that looks cinematic.
Jacob was going to move around the space quite a bit and we had large flats to contend with so we paired our RED Helium camera with the Entinaya 250 degree fisheye lens. We then still filmed in four directions (north, east, south, west) and that gave us a ton of overlap for compositing.
Shooting nodal comes with the trade off of needing to have a very detailed plan of attack. For that, we turned to Unity where we were able to mock up and pre-viz all the blocking. It was incredibly important for us to figure out the blocking ahead of time so we knew that it would all stitch together seamlessly when we merged the camera angles in the edit.
The shoot consisted of over 35 individual shots that we stitched and merged together to create the seamless 8K final video.
Creatively we wanted to embrace the sense of childlike wonder that I remember from my days in grade school theater. Michel Gondry’s The Science of Sleep was a touchstone reference for us as well as Harold and the Purple Crayon and Where the Wild Things Are. We very much wanted the feeling to be magical and nostalgic.
More importantly, we really wanted mostly everything to be done practically. All the sets were moved by hand to create the transitions between scenes and a “hero” Jacob singing the main vocals would journey around the 360 space, helping guide us through the story.
POST PRODUCTION INTO 2D
With all the compositing, stitching, and rotoscope work of the 360 video completed, we ended up with an 8K file. Now, this is very rough math, but seeing as the FOV of 360 video players is somewhere in the 70–90 degree range, I estimated that an 8K file would give me the ability to generate a 1080p HD 2D version.
We utilized Re:VisionFX Re:Lens software in Adobe Premiere Pro in order to convert the video from lat long projection to either rectilinear or tiny planet. It’s actually really fun to use because it feels like you are directing a 2D camera with a zoom lens that’s shot from the center of the 360 world. I found that I could zoom in, pan, tilt, and animate all the keyframes to feel smooth and follow the story.
The hardest part was figuring out what we wanted to feature in the 2D cut. Jacob and I sat together as we cherry picked our favorite moments that we felt told the best story and worked on framing for timing and flow.
THAT’S A WRAP
For me, 360 video proves again to be an endlessly creative medium of expression to try new things both on a technical and story level. Taking this piece into 2D offered us a chance to experiment in cool ways we haven’t thought of before. What do you think? Success? Failure? Either way, let’s keep pushing this new storytelling medium forward.
