Category: Post Production and VFX

Unreal Sequencing

Along with the VFX, creating the sequence took the most amount of time to do in Unreal Engine 5. Despite this however, using the master sequencer and setting up the cameras for the different shots were fairly easy to do.

The most engaging part of this area was creating dynamic shots since I learnt very quickly that dynamic shots aren’t confined to movement.

They could also mean changing the focal length in order for a particular item to become more visible. It’s a more subtle change but it helps compliment the song’s melancholic pace. This was also a great opportunity to utilise the space using wide camera shots were the best to use in terms of making the characters feel as if they’re in their own space despite being in the same room.

Saying this, however, timing these shots in line with the music was the hardest part of the project as it’d take multiple tries in order for a single shot to move with the song seamlessly. Looking back, whilst it would’ve been easier to focus on timing in adobe premiere, there would’ve also been a risk of not having enough footage for the music video. So whilst it took a large portion of time, it was still safer to line up the shots in time with the music in Unreal rather than primarily using editing software.

Post Processing

Once I finished the basic sequence in Unreal, I was able to tweak and cut down the unedited footage using Adobe After Effects. There are 3 separate types of the same sequence that were exported due to each version having different VFX and character poses. This also allowed me to point out any potential errors within each version that weren’t initially visible when working on the sequence.

Whilst editing the footage, I added in the lyrics to make the video authentic to the genres it’s based off of as a lot of the example MV’s from the Vocaloid community often contain lyrics for the audience to follow along with the song, deciphering their own interpretations on what the song was trying to convey.

This is the point, where I decided to mix 2D and 3D visuals into the video together. Not only did it fix the problem of the lack of animation and facial expressions for the main character but it helped add subtle, visual details for the audience when interpretating the narrative.  

These 2D visuals really helped flesh out the story and make the main character look more human, whilst her partner looks lifeless which creates a great contrast between the two personalities. To make the 2D visuals, I drew them out using Paint Tool SAI and exported them into Premiere during the editing process.

Conclusion

In summary,  the project was an overall enrichening opportunity to explore different techniques and artistic approaches when it came to designing and making the music video. The most interesting learning experience came from the film making side of the project as it allowed me to explore different techniques and compositions when filming all the scenes. Combining the 3D and 2D mediums was also an enlightening experiment that surprisingly worked well as they didn’t clash too much – instead in an artistic context, they were able to complement each other in story telling whilst making the video more lively and engaging. The VFX, whilst I struggled on for a bit, also helped me to understand how to make artistic effects as well as helping me compromise in design decisions that couldn’t be implemented in a stylised environment.

To improve the video in the future, I want to create the full sequence rather than only focusing on that particular part of the song. This way, the video would be able to provide more context whilst also allowing the song to flow a lot better. Whilst it may not suit the tone of this particular project, in future MV, I’d like to explore more complex VFX as I’ve gained confidence by learning and implementing simpler effects.

Small details such as  tweaking the duration of certain focus scenes (e.g flowers in vase shot) so they’re in full focus, adjusting certain asset errors such as the floating dining table or the flying cutlery or editing the chairs so that they fit with the room’s symmetry are also minor changes that I’d like to implement in the near future.

 For now, in conclusion, the music video was an overall enjoyable and interesting experience to make and It has given me the inspiration and confidence to make similar projects henceforward.

Since the MV was going to take an artistic approach, I decided to try and see if I could keep the sequence limited to one set piece, keeping the idea simple whilst being able to try out different props and compositions for the scene. By setting up the scene in unreal engine, it becomes a great opportunity to see which shots need to be tweaked or overall scrapped, much like the first storyboard.

In the second storyboard, the MV would take place between two separate spaces. However, using the room’s size, the camera shots were able to create the illusion that the characters were in separate spaces during certain scenes, giving them their own unique space once the second half began. This was achieved by dividing the room into two seperate spaces and adding their own unique pieces to their areas,

Initially, there were some concerns with the complexity of the set piece, but when seeing this complex MV take place in only 2 stages (with the majority of visuals being 2D). I wanted to challenge myself by creating an MV with only one set and only using camerawork to create these illusions mentioned prior.

In addition, the visuals of this MV inspired the block colour aesthetic. Originally, the characters were going to be textured normally. However, they did stand out from the set piece too much, especially with the lighting. So instead, I gave them block colour textures in order for them to be the direct focus of the scene whilst also blending in with the aesthetic.

Given the inspiration, when building the composition for the set, the main objective was to make the room both artificial whilst still visually resembling a dining room. To do this I used symmetry composition by using one of the cameras to simulate a wide shot and the flooring as references. This helped me centre the dining table, cake set and window directly in the middle, creating an almost artificial but picturesque feeling to the room. This tone is further emphasised by the perfectly placed food on the table which also becomes part of the main focus. This also tells the audience that something isn’t right visually without directly telling them the reasoning behind it.

The orange, directional and spotlight lighting creates a warm ambience whilst also making the colours pop. This also helps sell the dining room image despite the set looking unnatural which, in a different interpretation, could make the room feel more ominous.

Without the lighting, the colours and food seem bland and sterile – the composition isn’t as interesting or visually appealing.

Experimenting with Niagra system – the first effect I created was the lily particle system. For this, I created the particles by using the hanging particles emitter and then changing the sprite texture to hand-drawn lilies petals and adjusting the opacity so the texture became an additive.

Afterwards, I added lighting to the particle system and adjusted the spawn rate so that the lilies didn’t obstruct with any of the obstacles or the character. Whilst it’s still a simple effect, this system helped me to understand the interface of Niagra and the ability to explore the different variables that could be easily adjusted.

Outdoor VFX

Due to location of the window in the set piece, adding a HDRI map wouldn’t have been suitable for the outside area as it’d be difficult to see. But on the creative side, HDRI maps would’ve directly contrasted with the stylised room so I decided to opt out – instead making a scenery by only using VFX.

This way it’s still stylised but not overtly eyecatching enough to steal away from the main focus of the shot: the characters and the props. Before adding the effects, I added a black backdrop in order for the parrticles to stand out.

So for the stars, I used the same hanging particle emitter and changed the sprite for a more rounded look as well as raising the spawn and distribution rate in order for the stars to appear more spaced out.

Whilst the rain drops were made using the fountain emitter and to keep particles set in a closed area, I added a box location and stretched out the dimensions of the box in order to keep the particles spread out.

 Afterwards, I added a velocity variable which was kept at -1000 in order for the rainfall to be noticed in the shot and finally, I stretched out the sprite size to simulate rain drops.

Creating the Ribbon VFX

Whilst this effect was another experimental piece, it still fit the stylised and unnatural tone of the set. It doesn’t have a particular meaning but it helps convey to the audience that the video is based off of the main characters perspective, seeing the food as nothing but scribbles on a plate and later seeing those scribbles on her partner’s face showing her distaste, fitting in with the songs themes of feeling unsatisfied in a relationship.

Originally, I wanted to use the ribbon system to create an effect with lyrics spiraling around the main character. However, after exploring the settings and being unable to format the lyrics into the spiral I decided to create the scribbles instead.

To create this, I started off with a simple sprite burst emitter, changing the sprite renderer into a ribbon one, switching the sprite renderer into a ribbon one. Then after changing the life cycle to “self”, increasing the spawn rate to 100 and changing the shape location to torus and it’s distribution to random, it became a tangled mess of light. I was then able to place it on the food and later on the second character’s face.

Bones and Plates VFX

The final effect I added into the scene was a mesh particle effect featuring bones and plates, these effects mainly serve as a homage to the music videos that inspired this project’s creation as the imagery simply represents death, further implying that she sees her partner as the cause for their relationship failing (and also because in the story he accidentally kills her.).

 In aid for the making of this system, I used the mesh particle tutorial from the Unreal Engine 5 Documentation. Similarly to how I created the rain, I started off by using the fountain template emitter. Then I changed the sprite renderer to the mesh renderer and added the bones asset to the particle mesh section.

Afterwards, to make the movement of the bones more dynamic, I edited the rotation velocity in order for the bones to move in random directions. Finally, since I wanted to add multiple meshes to the system, I copied the effects settings as a template for the plates and adjusted the spawn rates for the two particles.

Inspiration and Story planning

For this project, I wanted to create a 3D music video using Unreal Engine 5 and Adobe Premiere. By creating a music video, there’s opportunities to experiment with different shots, composition methods and vfx. Another interesting aspect I wanted to explore was the editing and post production aspect as editing and creating a seamless story would become a large and vital portion of the video itself.

The song chosen for this piece is called Cutlery, A Japanese Vocaloid song created by ewe which talks about the anxieties and struggles of a one sided relationship. The music video made for the song is presented in 2D and is a generally slow paced which emulates the melancholic tone of the music’s tune. The song is also accompanied with strange but symbolic imagery which fits the character’s relationship falling apart throughout the video.

Whilst 60 seconds could only provide a snippet of the song’s full story, by snipping out the second half and just focusing on the initial buildup and main chorus, the song’s narrative remained uninterrupted. With this project, the main goal is to create a different story whilst still maintaining the song’s themes.  So the plan started off with creating a rough storyboard to have a better understanding of the story and the song’s beat.

The story logline created for this music video is as follows:

The main character is stuck in purgatory. She has recently been accidentally shot by her husband and now must spend the rest of her time with the same man she hates in a confined space that vaguely resembles their dining room.

Storyboard Planning

In order for the MV to follow the rhythm, The storyboard generally has a lot of cuts with static or moving shots. The combination of both creates a diverse sequence that keeps the audience engaged. On the other hand, this also allows to exploration in compositional shots.

When finding inspiration, I also used  examples from existing 3DMV’s of the same genre.

The original idea was to have the MV solely be 3D with stop motion animation to accompany the slow and melancholic pace of the song. But when exploring Unreal’s animation editor – there was a realisation that rigging different poses for each frame would become time consuming very quickly.  Looking back at the examples, The use of camera work in these videos create a fluid and dynamic feel, making the compositions and storytelling feel a lot more engaging with the audience, even with the songs aren’t as upbeat or fast paced.

To counteract the rigging issue, the storyboard was changed to instead focus on the camera shots to simulate that same dynamic movement shown in the examples whilst still maintaining that slow and choppy pace.

To make up for the lack of animation, especially in the characters facial expressions, I decided to mix the 2D and 3D genres together by adding hand drawn frames later on in the post production part of the workflow.

Finally, I made the storyboard frames into an animatic – this process further helps visualise what the music video would look like, especially during the editing process.

VFX planning

For the VFX, I created a list along with the storyboard. I saved the more elaborate ones for the second half of the video.

By having a list, I was also able to discover and learn a variety of different techniques for implementing these effects. This also helps to plan out which VFX to implement throughout the project based on how complex they were. Looking back in retrospective, A lot of time in the project was taken up from learning about Unreal’s Niagra system interface and experimenting different VFX emitters.

Nevertheless, using Niagra has been an enlightening experience as it opened up a lot of room for experimentation. With the storyboard, I was also able to sketch out what some of the effects could’ve possibly looked like. These effects were also inspired by other 2DMVs of the same genre. This is focused more in the VFX blogpost.

Song

Kamiyama, Y. (2017) Cutlery [Media] (Original Version) Available at: https://www.youtube.com/watch?v=HHhFX9zUV2s [Accessed 5/10/2022.]

Tanabe, R. and Kusunoki, T. (2021) Cutlery | Kanade & Mafuyu (English Subbed) [Media] (In-Game Cover version) Available at: https://www.youtube.com/watch?v=0sSMsV71YCI [Accessed 5/10/2022.]

Music Videos (Inspirations)

(All screenshots and GIFS were taken directly from the music video and were edited using Adobe Express)

DECO*27 (2014). DECO*27 – 弱虫モンブラン feat. GUMI. [Video] Available at: https://www.youtube.com/watch?v=B-A2eYlbrwQ [Accessed 21/10/2022].

DECO*27 (2019). DECO*27 – 乙女解剖 feat. 初音ミク. [Video]. Available at: https://www.youtube.com/watch?v=7zwCIz-Ohn4 [Accessed 21/10/2022].

柊キライ (2020). Bocca della Verità / Hiiragi Kirai feat. flower. [Video] Available at: https://www.youtube.com/watch?v=wp9Z2wZ-2bs [Accessed 22/10/2022].

Karin’s Channel (2017). Hatsune Miku: Project DIVA Future Tone – [PV] ‘from Y to Y’ (Romaji/English Subs). [Video] Available at: https://www.youtube.com/watch?v=s3W9x2EZBWA [Accessed 14/12/2022].

Karin’s Channel (2017). Hatsune Miku: Project DIVA Future Tone – [PV] “Interviewer” (Romaji/English Subs). [Video] Available at: https://www.youtube.com/watch?v=o4WMjw7n6Cg [Accessed 12/10/2022].

Karin’s Channel (2017). Hatsune Miku: Project DIVA Future Tone – [PV] “Two-Sided Lovers” (Romaji/English/Español Subs). [Video] Available at: https://www.youtube.com/watch?v=RbVH-nvFGuk [Accessed 12/10/2022].

Karin’s Channel (2017). Hatsune Miku: Project DIVA Future Tone – [PV] ‘Time Machine’ (Romaji/English Subs). [Video] Available at: https://www.youtube.com/watch?v=_-0xXtEH_y4 [Accessed 12/10/22].

WeirdCat (2021). Project sekai || Lost one’s weeping 3DMV [eng subs]. [Video] Available at: https://www.youtube.com/watch?v=yaJErT-GjLw [Accessed 05/11/2022].

Asset List

Nessuvia. (2021). Nishikiyama Akira version. 2 (Y0) by Nessuvia on DeviantArt. [Model]. Available at: https://www.deviantart.com/nessuvia/art/Nishikiyama-Akira-version-2-Y0-866146428 [Accessed on: 10/11/2022].

RGG Studio. (2015). Yakuza 0 [Video game and Model]. SEGA: Tokyo, Japan. Available at: https://store.steampowered.com/app/638970/Yakuza_0/ [Accessed on: 20/11/2022]

Mimi model made from VROID studio

pixiv inc. (2020). Vroid Studio [Software]. Available at: https://vroid.com/en/studio [Accessed on: 10/10/2022]

laanita. (2017). Lily by (laanita) . [Found Asset] Available at: https://sketchfab.com/3d-models/lily-f904c01714f8433f9c37795d44a73346 [Accessed 10/11/2022].

Gajk.Mv. (2020). flowers with the vase by (Gajk.Mv). [Found Asset] Available at: https://sketchfab.com/3d-models/flowers-with-the-vase-0f58352d377840cdbeef0bd71b24aff6 [Accessed 10/11/2022].

tgiorgianni. (2020). Pancakes by (tgiorgianni). [Found Asset] Available at: https://sketchfab.com/3d-models/pancake-animated-370fb914cdcf434a809da5c633e35c48 [Accessed 10/11/2022].

ditovirnantio. (2020). Salmon Nigiri Sushi by (ditovirnantio). [Found Asset] Available at: https://sketchfab.com/3d-models/salmon-nigiri-sushi-low-poly-4a09a5fdbc1240f5a56abf874a4415a5 [Accessed 10/11/2022].

m31odyr. (2022). Small Wooden Kitchen Table by (m31odyr). [Found Asset] Available at: https://sketchfab.com/3d-models/small-wooden-kitchen-table-31dff1c5561e43049d75391ba24c0318 [Accessed 10/11/2022].

shuvalov.di. (2021). Chair by (shuvalov.di). [Found Asset] Available at: https://sketchfab.com/3d-models/chair-c226b9e0cace4a20bf017b7061ada18d [Accessed 20/11/2022].

Zab. (2019). Bone by (@lixyco). [Found Asset] Available at: https://sketchfab.com/3d-models/bone-f84a0805c2dc448fae864ee67f4c5836 [Accessed 20/11/2022].

Teixeira, M. (2022). Sliding Window by (@miguelteixeira.11). [Found Asset] Available at: https://sketchfab.com/3d-models/sliding-window-cdae97e160ae46c49644d3872a9e9211 [Accessed 10/11/2022].

Medeiros, V. (2020). Plate by (@valumlab). [Found Asset] Available at: https://sketchfab.com/3d-models/plate-3966eab192da44d7ad1e1c9d0afb08d5 [Accessed 20/11/2022].

Falxxx (2022). Long Table by (Falxxx). [Found Asset] Available at: https://sketchfab.com/3d-models/long-table-86f23ac640434aef8795e3509510ea4f [Accessed 10/11/2022].

Citrus (2021). Japanese Pudding by (@citrusfriendd). [Found Asset] Available at: https://sketchfab.com/3d-models/japanese-pudding-2fe0451867b5422980b02064ed3efdb0 [Accessed 10/11/2022].

Razavi, S. (2019). strawberry Cake by (@sia7ash). [Found Asset] Available at: https://sketchfab.com/3d-models/strawberry-cake-90b03e4e1ce24f5592bbfbb15b41c442 [Accessed 10/11/2022].

ddd. (2021). Strawberry Cake by (@endonoriko). [Found Asset] Available at: https://sketchfab.com/3d-models/strawberry-cake-c96ec380d87b4eb6b55549cc17a0e395 [Accessed 10/11/2022].

Any 2D textures / assets were made in PAINT TOOL SAI.

Extra Referencing

Template Supply. (2020). Free anime storyboard template [Image] Available at:

https://public-files.gumroad.com/variants/l14fapbhhae5s7sg7md9yplvvign/3298c3eb001bbed90f1d616da66708480096a0a1b6e81bd4f8a2d6e9b831d301 [Accessed 25/09/2022].

Documentation / Tutorials

docs.unrealengine.com. (n.d.). Create a Mesh Particle Effect in Niagara. [online] Available at: https://docs.unrealengine.com/4.27/en-US/RenderingAndGraphics/Niagara/HowTo/MeshEffect/ [Accessed 01/11/2022]

docs.unrealengine.com. (n.d.). Create a Ribbon Effect in Niagara. [online] Available at: https://docs.unrealengine.com/4.27/en-US/RenderingAndGraphics/Niagara/HowTo/RibbonEffect/ [Accessed 04/12/2022].

Aspland, M. (2022). How To Make Rain | Niagara Rain Particle System – Unreal Engine Tutorial. [Video] Available at: https://www.youtube.com/watch?v=s_UQYuM1RWU&t=47s [Accessed 04/12/2022].

TORA_V4. (2019). How Does the ‘Two Sided/Faced Lovers’ PV in Project DIVA Work? [Video] Available at: https://www.youtube.com/watch?v=PLF772o2Saw&t=1s [04/12/2022].