NEOSHIN EPISODE 01: COLD BLOOD

NEOSHIN is our diploma project at Filmakademie Baden-Württemberg.

With a core team of 4 people and many additional helping hands we are creating a 3.5 minute long cyberpunk music video of the band NEOSHIN which aims to bridge the gap between narrative film and music video where the real band will both exist as a virtual character of the story and in the real world.

Of the final film, around 2.5-3 minutes are realized in full CG while 0.5-1 minutes of  real band footage is integrated into the virtual locations.

A general overview over all major aspects of the project can be found in the video below:

To integrate the real band performance into our cg locations we are making significant use of virtual production techniques. Using large LED walls in combination with real-time rendered scenes in Unreal Engine 4 and a tracked camera, we eliminate the need for a greenscreen and also get accurate lighting and reflections on specular surfaces, which is especially visible on the reflective face maskes the band is wearing.

Those masks are customly created and feature an LED matrix which can play any kind of footage. They are controled through touch designer and synced with the unreal engine content of the LED walls and the music through OSL signals. The following behind the scenes video gives an outlook of our virtual production shoot:

Our main character AYUKO is a digital double of a real person. To realize this, we conducted a scanning session where we captured the likeness as well as a set of 65 blendshapes based on the FACS system and a wide range of reference material to create a believable, digital human.

Our characters are then brought to live using an optitrack system for motion capture as well as Dynamixyz to capture the facial performance. In addition to this, the full live performance of the band is captured as well in order to allow us to seamlessly blend between the real band and their digital doubles. The following two making ofs provide insight into our process for all of this:

The slider below shows the cleaned up scan as well as a selection of some raw FACS shape scans, which were then later used to assemble the final rig.

This was one of our first work in progress images we did in order to match the skin shader as close as possible to the real reference. Since we focused on taking accurate reference material such as grey/chrome balls, hdris and the exakt measurements of the light and camera position allowed us to rebuild the entire lighting stage and use this as a base to match the reference in CG.