Shifted.mov - Unreal Engine 5 short movie

Introduction

This is my first short movie made from scratch during my Enviroment Masterclass at Skyup Academy.

I followed through every process of the making, from the designing to the final editing. During this process, I challenged myself and experimented with all kinds of things.

The main focus was trying to use the majority of the tools inside Unreal Engine 5, even if during the process, i decided to add some external tools as well, to have fun with.

Initial development

The main focus was to use and experiment with the metahuman ecosystem, because i wanted it to be the main part of the composition. I got inspired by an old game of my Steam library which is Half Life: 2 where the cutscene from the intro of the game suits perfectly for my needings.

I started focusing on an environment that had to go along with the main monologue that i was writing down for the main scene.
I started everything by setting up the key ideas, which in my case were:

1. Making a cool monologue;
2. Having a surreal/dystopian vibe;
3. A set of 2 enviroments to go along: one completely natural and the other heavily artificial.

For the script i brought up an old video showing a deep fake with a cool monologue about perception. I transcribed it and used as a base idea for my script, with a bit of creativity it turned out pretty well.

The environments were created following the context of my monologue, where my main character would've been an in between from a Matrix character to a contemporary reinterpretation of Osiris from the Egyptian mythology: so her look had to be something like a total black outfit, a pale skin tone, black&white tattoos with an egyptian style and some gold accessories such as earring and necklaces related to her character.

On the other hand, the key enviroment was heavily insipered by the designs of the series "Made in Abyss" by Akihito Tsukushi which have the kind of natural surreal and fairy vibe i was looking for.

1. Main Take metahuman with Script

This one was the main focus of all the work: it's where i've done everything related to the metahuman monologue. Using after effects, i've added in the backgrond some projections from the Fairy Forest Environment to follow along with the script.

Metahuman Creation & Animation

The base model came straight up from Metahuman Creator where i designed the character and the looks, after importing in Unreal Engine where i added some final touches such as tattoo decals, custom jewelry with in-engine physics simulation and custom emitting masks for the face.

The animation process was done with the Metahuman Animator suite built-in Unreal Engine to bring my animations into the metahuman by just using an iPhone recording myself for the motion capture. It was fun and challenging and in the end, after some polishing, I came up with something that i thought would have worked out.

During the recording, i also took up in a separate channel the audio track of the motion capture to have an isolated track to work with for the fx and mixing process.

In the end, even if it still has some shenanigans to work around, it's been a cool tool to use.

Background shader and fx

I wanted to add some movement and fx like a slow tornado to the background instead of simply making it black; so i went on and played with the node graph inside the editor to find a combination of effects that i would've liked, and, after some tweaking i came up with a cool shader which i paired to some mesh cones to create the final shape of the tornado i was looking for.

This feature was extremely over enginereed and complex and an overlay in compositing would have done the same job probably, but it was still fun and challenging to work with.

Camera setups and Sequencer

Lastly i was left with making the sequence of the shots match the storyboard i had in mind, so i started placing cameras in the spots i needed to create my composition while also tweaking the scene to fit my needs; add some animation keyframes and the motion capture to the metahuman and i was good to go. In the final shots i used 4 differents cameras where i choose to render the animation to have the full clips to work with during the composition and editing process.

2. Enviroment Fairy Forest

Landscape & Blocking

Making of the custom tree hero asset

The main prop of the enviroment was of course going to be the trees that are scattered through the scene, actually my reference had this super-oversized trees that i couldn’t find anything that could work online, so, i had to start creating my own from scratch for the sake of learning new things.

Luckily, i had to only make the bark because i didn’t need all the branches and stuff to make my shots work, so i hopped on TreeIt and made an oversized high-poly custom tree bark to match my needings. The base mesh was then fine-tuned in Cinema4D so i could have a closed mesh with a reasonable UV to work with.

After the mesh was ready i opened Quixel Mixer to start texturing the bark from scratch. The cool thing about Mixer  is that i can directly have access to all the Quixel Megascans materials and textures, so that i could have a great starting point to work with. Then i just kept working and layering different channels to get the material is was looking for.

Getting the remaining assets from Quixel Megascans

While i was doing the hero tree bark i also started getting all the other props i was needing for the enviroment through Quixel Megascans. The main props i was needing for my enviroment were primarly rocks, and various foliages, but i also got some other stuff such as the tombstones which i had to tweak a little in Unreal Engine by using the Modelling Toolkit.

Other then that i also got a large amount of textures and shaders for the ground, the rocks and all the other details such as imperfections and various noises.

Setting up the landscape material

Now that I got everything up it was time to set up the scene, first of all the landscape. Starting from a base shader i made a standard landscape material with some minor features such as triplanar projection, Runtime Virtual Texture sampling for the props, landscape layers to paint on and a lot of variation layers and randomization on top of all.

On top of everything, i also added a stack of decals to the ground to give much more detail and randomization other then making it feel more genuine.

Custom rock shaders

While the landscape was being populated with the assets i worked also on a custom rock shader, so i could blend them together to match my reference; some of the features of the material where upcycled from the landscape shader, such as triplanar projection and RVT blending, while i also added some other tweaks such as procedural moss on top of the rocks, and several layers of randomization to give a cool look.

Procedural Content Generation for the hero tree asset

Everything was starting to look as good as i had it in mind, and while i was scattering all the plants and stuff with the foliage tool i was left with creating all the vegetation and vines that were climbing all over the trees. And now it was time to start experimenting with the PCG Framework to get the job done.

Basically i managed to use the vertices of the highpoly mesh as samples where i could spawn some groups of meshes and after some tweaks i managed to do the following:

1. Scattering the hanging vines from all the vertices that were facing down;
2. Placed all the moss/weeds on the top and the sides of the tree;
3. Generated some other vines that were climbing all over the direction of the trunk.

And after a lot of troubleshooting and fixes i managed to get this result, which is kinda cool because now for every tree placed i could have a random generated foliage that would follow the rules above.

River and waterfall setup

The last major achiviement was to get the waterfall look good; as for the water, i struggled between several versions and methods to make it work and after a lot of iterations i went with a setup that consists of a base cylinder mesh slightly adjusted with the modelling tools with the base water shader i made earlier applied to it, then i duplicated the mesh again and applied this time a foam waterfall material which i stole and tweaked from the Standard Water Material by Epic Games and finally, on top of that, i added a niagara particle system to get the waves and foam to feel more natural.

This last part was the most time consuming just because i was new to particles and i still can’t figure out the potential of that, but after some guesswork and some studying i got the basics figured out and managed to get the final look i wanted

Lighting and fog

Finally i was ready to make the last major step to complete the enviroment, which is art directing the lighting and the atmosphere/fog. My goal was to achieve an overcast yet sunny look, with some godrays or some heavy fog to show also the depth of the scene and give it the surreal-fairy look i was looking for.

For the sky and the enviroment effects of all the scene i used the  add-on UltraDynamicSky, which was awesome yet trivial to work with, primarly because there are way too many setting that i could tweak and think off. Then after some guessworking i managed to get all working and running as i wanted with features such as dust, ground fog, volumetric fog and volumetric clouds and cloud shadows.

The scene was almost ready but i wanted to manage and art direct the fog how i really needed for the shots in a much better way, so, with the help of another addon called EasyFog i started working with some animated plates that i placed to match with my cameras to get the work done in an easy way and with some cool parameters and tweaks to get my desired look.

Camera setups and sequencer

From this scene i needed to output two different shots: one needed as a background to combine with the main script shot, and another one in POV which is used for the main story as a transition between the two different “worlds”.

So first of all i started placing the camera according to my storyboard and my idea of the shots, and i used the river as the main thread that would follow along with the speech of the metahuman, so that i could give some kind of visual connections between everything i put together. Then i assembled and figured out all the single shots which i then put together on a master shot to have some feeling of how it would turn out.

The cameras and the shots were pretty straightforward: i just used some simple camera panning movements and some slight tweaks such as using distorsion, chromatic aberration and some anamorphic lenses for a more cinematic look.

The POV shot was easier to work out just because i didn’t need to tweak much with the cameras or any fancy movements, i just had to spend a lot of time working on the keyframes to match the type of movement i was trying to make work. After getting the main keyframes right, it was just about working inside the main keyframes and tweak everything to make it appear natural and smooth, and after some trial and error, i managed to get a result i was happy about.

3. Enviroment Studio

Setting up the SceneCaptureCube

First of all, i wanted to have projected on the screens of the room the exact same enviroment of the Fairy Forest, in the exact same position where the previous shot should stop to get a seamless transition between the twos.

I did some research while trying to figure out which was the best method to make this and while i was thinking about rendering some kind of HDRI or cubemaps of the previous enviroment, i found about SceneCapture components and in particular the SceneCaptureCube, which actually renders a real time cubemap of the enviroment where is placed.

It is performance intensive, but for my offline usage was more then ok, and, after placing the SceneCaptureCube in the same ending position of my previous shot, i managed to get a real time cubemap of my enviroment stored as an asset for my shader.

Tv screens shader

After getting the SceneCaptureCube running and after getting the main props of the studio I started thinking about how to use the cubemap on the scene, and so i came up with this idea: making a giant cube that will be the main screen for all the TVs and place all the TV Bodies on top of that so that i could fake them being singular screens, instead of a single giant cube.

And it was working as expected without any major flaws actually, so now it was time to make the shader for all the TV screens, and this one was pretty intense to get working as i wanted.

Long story short: I had to get the cubemap projected inside my main cube, all done with a node called “interior cubemap” then i had to interpolate and rotate every single side of the cube to match and align with the other sides. After that i added on top of all a pixel grid overlay and a slight offset for each color channel so that it would’ve feel more like a normal led screen.

So far so good, now i was just missing some kind of glitch/crash fx that i needed in my storyboard. From here i had plenty of choices to think of: from post process overlays inside Unreal Engine to actually making it as a post process VFX inside Adobe After Effects. But, as for all the other things, i wanted to learn more new things, so i tried to make all this working inside the same shader, with all the parameters i wanted so that i could control each and every single FX of the shader in the sequencer.
This is the actual list of the feature i managed to get into the final shader:

1. Signal distortion/matrix style transition;
2. Windows DDOS overlay;
3. Signal distortion overlay;
4. Windows error pop up messages;
5. Source Engine inspired error messages on top of everything.

Then i just had to start working on one by one. First of all, i gathered all processed all the images i was needing for the FX, from the DDOS overlay to the error messages, then i needed to figure out how to make the signal distortion thing work, just because i wanted to have an animated fx, but, sadly, i didn’t figure out how to get unreal shaders work with gifs or looping videos, then i found out about flipbooks and how they’re used inside gaming developement for sprites for example, and i wanted to do the same for the glitchy distorsion.

So i managed to find online a GIF of a glitchy overlay which i exported then as a PNG sequence, then with a free tool called GlueIT i managed to create a single flipbook image from it and thanks to the “Flipbook” node inside Unreal i managed to make it work all along with all the other FX. Then after polishing everything i had all my FX ready to be keyframed on the sequencer and properly aligned with all the TV screens.

Animation and sequencer

As for the previous enviroment, i needed to make a cohesive transition and shot for the POV sequence between the two enviroments, so following the same workflow i used for the previous enviroment, i managed to get a POV animation which was feeling natural and smooth, all by keyframes, simple camera shake and a lot of little touches here and there to make it look good.

After that i just keyframed all the parameters from the TV screen shader so that all the FX were running along with the shot.

4. Export and Post Production

5. Audio design

So far so good i was able to get all the visuals done, so I hooked up with an old friend of mine which now is studying Audio Engineering in Milan. He really liked the project so we started working together on everything related to the sounds, musics and voiceover i needed to deliver the full project and bring it to life.

We went into his studio and gathered all the ideas and references toghether, then we started with a pre composition of what the final track will look like. With that ready we arranged a small track where i would record my script and motion capture on, so that the speech won’t be off-beat with the rest of the track.

The major work went around processing and making the voice sound good, that’s why i recorded the voice in a separate track while doing the motion capture. After that the track went through a cleaning step, an interpolation via an AI web-tool and a mixing and fx step where i got the final polished monologue of my metahuman.

With the voice ready we finalized the track underneath everything while using the main edit of the video as a reference, so that everything would be already in place and ready for the final composition.

After all the work was done and everything sounded good, i finally merged everything toghether, having as a result the short movie i’ve uploaded.

I would also like to thank again Giovanni Celidoni for joining me in this part of the process, soon he should be able to upload his work as a standalone track on his soundcloud profile.

Lastly, i would like to thank my teacher Christian Tagliapietra and all the Skyup Academy team for everything that they left me during the lessons and the various revisions we had together. Hope someone will enjoy this movie as much as i enjoyed the process of making it.

Feel free to comment any suggestions and opinions about my work, i'm open about everything and still learning stuff. Hope it can inspire someone or lead to something bigger in the future.