Duration: November 2021 - March 2022

Tools: Unity HDRP (C#, Compute shaders, Shadergraph, VFX Graph) - Ableton Live

Role: everything

———

Description:

Around October of 2021 an idea for a short animated film project began brewing in my head. There was also an NFT art boom still happening at the time, and a good artist friend of mine Sutu Eats Flies recommended that I explore concepts for the film as a limited series of digital art NFTs. I thought it would be a good incentive for me to finish my work and also learn more about all this blockchain stuff people were raving about.

So from November 2021 to March 2022 I worked on a series of 9 short musical animations exploring the concept for my film, releasing each one as an NFT on the Tezos blockchain: https://objkt.com/profile/tz1btcUVLt48gcCTtHebRpAL8tGnq3DE8Ya2/created

The series is called "Phantom". Each piece explores a concept for a short audio-visual film and live performance I have currently in the works, the story of which revolves around the stylistic and musical interpretation of an artificial intelligence learning how to achieve consciousness.

I learned a lot about the process of promoting and selling art through the lens of artificial digital scarcity and digital ownership through blockchains - both very new concepts to me. Tezos was a fun blockchain for me to use - I found the community to be really engaging and welcoming, and I liked how for the most part artists on Tezos went with lower price / higher edition formats instead of expensive single editions. I found it gave a lower barrier to entry and made the experience more enjoyable for myself.

This is not the end of my journey into making and selling art. The connections I made with collectors who became fans of my art were strong and left a lasting impression on me. I’ve learned this is what keeps me motivated to finish my art projects - knowing that something I create could leave a strong impact on even a few people makes it easier to push through the difficult times and struggles that can happen when making art.

Breakdown

Phantom 1 - Shadow Runner

The majority of this piece is made up of instanced meshes and flowing tendrils attached to the body of an animated humanoid. In order to implement this in Unity, I took learning from artist/coder legend Keijiro Takahashi. From his github I was able to learn how to use C# Jobs and compute shaders to generate textures (or now with Unity 2021 compute buffers) that can be utilized in VFX Graph to render many instanced meshes or particle strips.

The tendrils are simulated using basic spring physics + Perlin noise, where each strand calculates the new position of its vertices in parallel to one another using C# Jobs. This would be trickier to implement in a compute shader because the spring physics algorithm used here requires knowledge of prior vertices in the strand, though if it were parallelized on the GPU it would be most likely faster.

The source points for the tendril roots and the instanced spheres comes from calculating random positions on each triangle of the running man mesh, using barycentric coordinates calculated on each triangle. The math involved in this algorithm I learned from another Keijiro github: smrvfx

I wrote the music in Ableton Live, drawing some inspiration by one of my favorite musicians, Machinedrum.

Phantom 2 - Shade

Using a rigged female body mesh and dancer mocap data from the CMU mocap library, I built a system for echoing the history of the model’s dance animation, and playing with altering the shading of the model over the history of that movement.

I wrote the music for this in Ableton Live, using dark and moody piano music as a reference, and animated the parameters of my system to the music to create a small interpretive dance story of this ethereal being transforming shapes until finally settling into a still statue, echoing it self backwards into eternity.

At the root it’s technically very simple, using the step of each clone (1.0 / numberOfClones) as a look-up into an animation curve while adding other elements of variability into the mix that can then affect any material property to alter the shading of the model differently for each clone, and then animating between different parameter sets using these animation curves to create different looks over the course of the music.

Phantom 3 - Crush

The idea for this piece is about the interpretation of loss of data, where through progressively “bit crushing” a signal (reducing the range of values with which it can be expressed), you deteriorate it to the point where it is no longer recognizable to the original.

Using VFX Graph, I created a system to pulse various sinusoidal waves over a grid of cubes, animating their displacement and emissive brightness with the pulse.

The falling animation of the character was hand animated by myself (and thus reveals my lack of character animation skills!)

Phantom 4 - Ritual

This is one of the more elaborate and ambitious of the pieces in the Phantom series. It is also the first piece that explores color and texture, and the last piece in the series that is done in this dark / dreamy / hazy style.

For this piece I tell the story of the creation of a synthetic avatar through a ritualistic dance between four automatons. The dance was choreographed and performed by myself using an XSens suit for motion capture. Thanks to JC Leon for assisting me in the mocap shoot! Once the mocap data was processed, I applied it on to the four different automatons in the scene. To capture the data properly, I had to mark off the floor with tape so that I could match the virtual environment with my movements.

Some of my inspirations for this piece were Cirque du Soleil and taiko drumming.

This may be the last piece in the series that has this particular aesthetic, but I definitely plan on reviving it once my film is done.

Phantom 5 - Glitch Runner

This was the first piece that deviated from the dark / moody / dreamy aesthetic, and entered into more synthwave / retro graphics territory, where the idea of the film is that the protagonist is running through a stylized interpretation of its own A.I.

In this piece I borrowed some of the RGB delay aesthetics and techniques I developed in one of my older works, Glitch Drum. I also wanted to explore the idea that the protagonist of the film was made up of small cubic voxels, and that as it got further along in its quest to understand consciousness, its voxel form would become more detailed and high res. I ultimately scrapped that idea, but I’m still keen on exploring this aesthetic in the future.

Real-time voxelization of a skinned mesh proved to be a difficult task, and to be honest I don’t fully understand how the math works still, but I was able to re-purpose the code of graphics programmer mattatz so that it used VFX Graph to render the voxels, which made it a much more flexible artistic tool.

Phantom 6 - Grid Runner

Continuing to riff off the idea of the previous piece, in Grid Runner I explored more synthwave musical elements and RGB delay like visuals using procedural grids that ran through the landscape, of course inspired by Tron light cycles.

Phantom 6 - Data Runner

This was a pivotal piece for me because it was the inception of what is now becoming a central aesthetic in the film, soft emissive line and dot patterns. In order to make this system, I tried a number of different approaches using VFX Graph, including instantiating multiple VFX Graph components at runtime. However I ran into a problem with Unity, too many instantiations of VFX Graph at once crashed the game. While I could have probably gotten around this, I also knew having too many instances of VFX Graph is costly and inefficient, so I found a much better way that used fewer VFX Graphs but passed in the data comprising the lines and dots using textures generated before hand.

The textures simply contained the position data for all the points comprising the particle strips and particle quads. Creating the textures was done either through procedural shape generating techniques or using a pipeline I made in Houdini to convert a mesh into a text file that writes positions for every vertex and every edge between each vertex, which than gets converted into a texture.

Note, after Unity 2021, I was able to swap the usage of textures for compute buffers, which is much more straightforward of a container for this data.

Phantom 8 - Hyper Cube

One of my favorite pieces in this style, Hyper Cube uses a special vertex shader to animate all of the moving elements in the environment. The environment is one large mesh, and using data that was created in Houdini and stuffed in the vertices, I was able to move the objects independently of each other. I use a custom gradient in a reflection probe to provide the rainbow like reflections across the metallic surfaces.

Phantom 9 - Descent

The idea for this piece is that the avatar is entering into the final stage before discovering consciousness. I wanted to evoke a feeling of mystery and sacredness. I used the Houdini shape generating pipeline I made in Phantom 7 to generate the elaborate dot/line patterns that lead the avatar into the descent of the environment. The patterns are based on the menger sponge fractal pattern.