Morphscape project recap

This past week, I assembled a slideshow for my final senior thesis defense presentation at Rochester Institute of Technology. If you haven't been following my blog, it is a pretty thorough overview and documentation of the work I have done and the workflows I've developed over the course of the past two semesters working on this independent project.

Although my thesis project is "officially" complete and I've demonstrated what I set out to prove initially at the beginning of the year, I will still be refining the project as well as continuing my research and development on the theme, processes, and visual style I have been pursuing-- so stay tuned! 

MORPHSCAPE: Interactivity

After putting together a visual showcase for last week, I've decided to switch gears for a bit to focus on the interactive component of this project, which is my intended end product. All of the dream events featured in the cinematic are distance based, and so they happen automatically as the player moves forward. This is works well for a movie, but for a game, it is a bit linear and passive. I've been revisiting the work I did several months back to create a volumetric painting system using Unreal Engine 4's Canvas Render Target 2D. Additionally, I've started to design a very simple character and its movement controls.


My initial idea was that the character would be a flying character set up in third-person. The camera would aim to look at a 3D cursor that is at the projected 3D location of your current mouse position. To control the character, you would use W A S and D. When the character would move forward, it would move forward in the direction of the camera's aim, towards the 3D cursor.

After testing this out, it felt pretty hard to control. I've always found flying controls in games to be the trickiest to get the hang of -- and they are definitely difficult to make intuitive as well. Additionally, I couldn't work out a camera arrangement that prevented the character from blocking a good chunk of the environment.

I decided instead to make the character act as a 3D cursor, which feels much more effective already. This also is much clearer and easier to control the painting mechanism, where the 3D cursor/character acts as the paintbrush. Now, the orb character flies to the projected location of your mouse in the world, while the player camera remains stable. The player camera is still controllable with WASD and will rotate to look at the cursor, but it is grounded.


The 3D cursor is set up mostly in the Player Controller class. To enable the mouse events, I've checked the Mouse Interface features in the Player Controller settings.


The base feature of the 3D cursor is that the mouse's 2D location needs to be projected into 3D space. I've used a node called Convert Mouse Location to World Space. This node alone doesn't put the location in the correct spot though. You can multiply the Direction vector from this node by a user-specified distance, and then add that to the Mouse Location. This essentially puts your cursor at the specified distance from the initial mouse location by moving it out along the direction vector. (See below.)

For testing purposes, I had attached a sphere to the cursor location to be able to easily visualize where it was in 3D space. As an additional control, I mapped the Mouse Wheel input to be able to increment and decrement the Cursor Distance variable, within a certain range so it is capped at a min and max distance.

Upon testing this, especially with the painting mechanism I am implementing (since the 3D cursor serves as the brush for painting), I realized it would help a lot to be able to have the cursor "stick" to nearby surfaces, essentially detecting anything in its path within a certain radius. Without this function, you would really have to guess where invisible, paintable objects are by just clicking around aimlessly in world space.

On every tick, I've made this the first thing that the Player Controller class checks. This makes it so, if there is an object hit under the mouse, it takes precedence over the default free-floating cursor. The node Get Hit Result Under Cursor by Channel checks for and returns any object hits under the mouse cursor. 


The most basic way to implement this is to then simply set the 3D cursor location to that exact resulting hit location. However, getting a hit under the mouse doesn't account for how far that hit away is from the 3D cursor location in world space. This makes it difficult to break your 3D cursor away from objects that your cursor may unintentionally be on top that are far away, due to perspective. My solution is to get the world location of the character and find its distance to the hit location. If the distance is greater than the user-specified threshold, then the cursor location defaults to the free-floating 3D cursor.

Additional things I'd like to implement here are some more visual indicators - perhaps when your cursor sticks to an object that is invisible, I could add a decal to the surface of that object.

There isn't much more that goes on in the Character class, but it does need to be set to the cursor location. You can grab the Cursor Location variable from the Player Controller class by getting the Player Controller and casting it to your specific Player Controller class.

I've used a VInterp To node to interpolate from the current location of the character to the current cursor location. This makes for much smoother movement, and adds the effect of the cursor following/lagging behind where you are mousing to. My character also has ribbon particles trailing behind it, so this allows the particles to trail for a bit longer.


Here is a very rough first pass of my idea for a look for the character that I have been testing with. I imagined it as a light orb because I feel like this is reminiscent of neurons, synapses, and electrical brain activity. I also wanted its look to allude to the dominant painting theme that defines both the visual style and interactive function in this dreamscape world.

The Character Blueprint contains a Particle System that moves along with the location of the character meshes. This Particle System uses the Ribbon Type Data to emit as a ribbon trail. It is emitted from the sphere character mesh by using the Bone/Socket Location module in Cascade to hook it up to the joint of the character skeletal mesh.

This example of the character moving on the left shows a (slightly choppy due to the screenshot capturing process) example of how the particles follow behind. 

Ribbon particles apply materials such that the leftmost part of the texture is towards the start of the ribbon trail and the rightmost side of the texture is towards the end of the trail. So, I've used a painted brush stroke texture as the opacity map, but I've blended it with a linear gradient that becomes more black towards the right side of the opacity mask. This makes it so the particle trail dwindles towards the end of the trail. (See right.)



My project idea started with the idea of painting in world space to reveal different worlds/objects. This is a concept I want to carry through to my final product. Early on in my progress, I had done some testing with Unreal Engine 4's Canvas Render Target 2D Blueprints. Due to a couple of issues I encountered with them, this testing fell by the wayside for a while -- but after resolving these

This is my progress so far in implementing this feature. As of now, it is working pretty well to my liking, though there is a lot of potential to explore here, as this is just a test with a simple sphere. 




Here are some features I've added this week:

  • Left mouse button applies paint.
  • When applied, paint stays permanent.
  • Objects can have paint only partially applied/be revealed in parts.
  • Paint can be cleared completely (mostly for testing purposes at the moment.)
  • Brush size is adjustable.

In the near future, I'd also like to add the ability to erase paint in parts. Decals on the surface of the object where you are about to paint would also serve as a nice visual indicator.

In a few releases of Unreal, the Canvas Render Target 2D was causing crashes, but this has thankfully now been resolved, which allowed me to continue working with them.

I also realized that the Create Canvas Render Target 2D node cannot be created within a Construction Script. Doing so resulted in an error that prevented me from saving the map, due to graphs being linked to objects in an external map. Instead, create it on Event Begin Play and the problem is solved.

MORPHSCAPE: Preview cinematic and thesis summary

This week I've made a cinematic to summarize and showcase my work on my senior thesis project so far.

At the beginning of this project, I set out to create a real-time dreamscape environment as a means of achieving two major goals in my work. The first, to build an environment and assets equipped with the parameters to enable visual transformations before your eyes; and second, to develop a custom painted aesthetic.

This aesthetic I have developed relates the spirit of impressionist painting conceptually to the transience of form in dreams -- how objects are ever shifting, becoming other things entirely as your brain assigns them new identities. With loose, expressive paint strokes breaking up the silhouettes of models, my goal has been to suggest the vague impressions of our experiences that appear in our dreams. What exists is just a malleable impression at the whim of your subconscious mind. Impressionist painting can be considered a study in the way we perceive. I find that this relates to the way our brain "sees" and records visual information to memory. We recall that condensed perception in our dreams and memories.

From a technical standpoint, working in 3D graphics as opposed to painting requires much more overhead to overcome the default look of the tools and be able to demonstrate a personal style. This is an area that has been very important to me in my work, to avoid letting the medium I use dictate my vision for a project.

One of the main issues I've addressed has been the hard-edged borders of models and the seams between intersecting models. This is an area where 3D can really reveal itself as 3D, as opposed to a painting, where strokes are able to make form fluid and seamless. The hard-edged 3D model contradicts the idea of malleable form, because it exists as a very concrete, defined form. To me, this breaks the dream-like illusion, as our dreams aren't always so clear.

While rendering progresses in terms of achieving greater and greater photo realism, making a material interact with light as a painter might depict light still stands as a challenge. In my materials, I wanted to regain some of the expressive, happy accidents that can give a painting so much life and character. Using a technique based in meshes and materials as opposed to a post process gives this look a more dimensional quality that I wanted for this project, rather than looking like an additional layer on top of everything.

Another goal of mine over the course of this year has been working with the idea of "smart assets." The development of systems like Unreal Engine 4's Blueprints have made the gap between art and programming smaller than ever. Every asset I have created for this project is thoroughly parameterized, with parameters communicating between materials and Blueprints. Building assets to be "smart" allows for more functional assets but also more flexible assets in a production pipeline to accommodate rapid visual iterations on the same asset. My semi-procedural spline tree system built in Blueprints is a testament to the flexibility that the overhead of setting up a smart asset yields.

Looking ahead, I'd like to realize my vision of making this environment explorable and interactive. The trees will sprout up from nothing, and the rock formations that line the shore will melt away into the ocean. My vision is that this will be triggered by user input rather than simply distance from an object. A character and music will help to tie everything together into a cohesive, experiential dream playground.


Over my spring break, I went on a week long trip to the Southwest with the College of Imaging Arts and Sciences honors group at Rochester Institute of Technology. Seeing so many unique rock formations was enlightening and a great source of appropriately timed inspiration and reference. This past week I've been dipping my toes back into my project with some newfound perspective.

Here are several things I've worked on this week, or will be working on in the near future:

  • Populating the environment. Although minimalism can sometimes feel dream-like, the map I've been working with has primarily been a test scene. I'd like to give this shoreline area a sense of space and add more rocks to accompany the main rock arch I've been testing with.
  • Pushing the painted visual style to be even more loose and expressive. I've started to do that this week just a bit with the rock, but I'll be continuing to work on this for the ground, the ocean waves, and the boat.

Recently I rediscovered my love for JMW Turner's paintings. His ability to effectively capture and communicate a scene or a moment with such expressive brush strokes and colors is something that's really inspiring to me, and my vision for this piece. I admire the way solid objects seem to dematerialize into atmosphere -- that fluidity of matter is something I'd like to achieve in my work as well. With all of his seaside themed paintings, it's a wonder I haven't been looking to these paintings all along! But it's never too late for some fresh inspiration.

  • Melting rocks. My vision for the rocks on the beach was that they represent hardened, fossilized memories. Your memories from your conscious brain are what dictate the things your subconscious dreams of. As the player, you can melt them to access these memories and fuel your dreams, creating more of the environment around you. I hope to have the rocks melt and form the ocean that the boat materializes onto. I've been thinking of this as a mechanism that will tie the interactive environment together into a cohesive, self-contained dream playground.

Here's a test I did with flow maps for the wax melting.  This setup involves a normal and height map's UVs panning by a 2D flow map I've painted. It is revealed by a linear gradient mask being pushed down. The mask is blended with the flow mapped height map and a noise to break up the linear quality.

What works pretty simply on a cylinder is a little bit more difficult on a complex model. So I've been spending some time trying to work out a method to get the wax flowing along the surface angle of the rock arch model seamlessly. I've painted a world space flow map using Mari's Vector Brush tool, and I'm using that as a flow map to modify the projected UVs of my flow texture.


  • Materialization effects. I want to create the feeling that the environment is forming organically as you continue to walk, or dream. Here is an example of the material effect I've been working on where the rock arch is materializing as you walk forward.


  • A sky. Here's something I've neglected so far! A sky can be quite expressive and contribute a lot to the mood of an environment. I've started to set one up, though this still has a ways to go. My idea is to use flow maps to create clouds that move in a streaky, swirling pattern. As you near the boat, I'd like this sky to darken and become even more tumultuous. 
  • Refinements. This week I tweaked the boat's buoyancy and added a rock to it so it isn't a straight up and down bob. This is done by simply modifying the rotation of the meshes within the boat Blueprint on a sine wave over time.



  • Procedural spline tree system, animated to enable growing trees. I hadn't worked with it in a while, but last semester I spent quite a bit of my time developing a procedural spline tree system, which is based around Unreal Engine 4's spline and spline mesh actors and uses a recursive method in a Blueprint for generating branches. The idea that motivated me to develop this was to enable trees to grow around you dynamically.

This week I've started to figure out my method for animating the components of the spline tree. My idea is to have an auxiliary spline that serves as a motion path curve that the spline points of each component can follow. Their movement, or distance along this motion path spline would be determined by a Timeline in the level blueprint, so that its movement can ease and not be completely linear.

This works well for the trunk so far. The issue with the primary branches and all of the following tiers will be maintaining the relative position of the child branch on its parent branch as they both animate. Additionally, a custom set motion path for every single branch would be a ridiculous amount of setup and user control, so I hope to automate the motion paths for the recursive, procedurally generated branches.

Here are some lower priority, but still important tasks I've been considering:

A character. I've been imagining this environment being explored in third-person, perhaps by a flying light orb character with a ribbon-like light trail.

Music. An ambient soundscape would set a nice, quiet mood for this environment.





MORPHSCAPE: Sand material blending & hooking up the dream sequence

I've been working with the sand material, which includes a blending of sand, rocks, small pebbles that blend in around the larger rocks, and water. With all of these components coming together in one material, I've had to do quite a bit of tidying up and figuring out the best organization for blending these all together. The rocks and water can be blended in by painting different Vertex Colors on the mesh. I break up the Vertex Color mask by blending in the height maps of each material. This ground material is also set up with displacement. I'm enjoying the way that these different components can be combined (for example, I can have the sand with rocks blending in and water blending over top.)

I've also been hooking up all the parameters to change throughout the boat deterioration sequence. This includes the boat's animated mask revealing rust on all of its parts, the buoyancy height displacing the boat changing, the wave height and speed of the water changing, and the shoreline mask clamping in, overtaking the water. All of these parameters are based on your distance to the boat, which is mapped 0-1. Each parameter has a float curve on a Timeline in the Level Blueprint, which maps its overall progression from 0-1 throughout the sequence. (For example, the wave height may grow as you come closer to the boat, but then decrease again as the shoreline swallows the ocean.) Each parameter has the true, user-defined min and max value, and the 0-1 from the timeline is remapped to that range. Then, the corresponding Material Parameter is set.

I'll be updating this soon with a newer video that shows the water waves growing in height and the sand surging towards the shoreline, but here is the boat sequence as of now!


A little bit on some developments I've made in terms of the concept that has been motivating my design. My idea is that, metaphorically speaking, your everyday experiences and memories are hardened and fossilized into rocks by your conscious mind. While you're dreaming, your subconscious brain accesses all of your memories. You can melt the rocks, which reveals these memories to your subconscious - you bring light to them and they become fluid again. I'd like the environment to be fairly empty at first, and as you melt rocks, accessing your memories, the environment will materialize - your subconscious brain is forming new dream memories from your collective experiences. Although your subconscious mind often accesses your permanent, fossilized memories while you dream, dreams themselves are very impermanent and hard to remember after waking. This is why many of my visuals deal with the ephemeral, like the boat deteriorating away and the constant flux of the environment. Sand also relates to this idea, as it is composed of a seemingly infinite amount of eroded rocks. It is the dust of time, constantly blowing around and never settling permanently.

MORPHSCAPE: Particle paint

Today I was inspired by this article that outlines an approach to achieve a non-photorealistic, painterly style. The idea is that particles are placed on the surface of a mesh with brush stroke shaders applied. There's not really any reason this can't be translated into Unreal, so I figured I'd give it a shot.

My original idea months ago was similar to this method, except I was scripting the placement of plane meshes onto the surface of a mesh using Blueprints. The problem was, this was only feasible to calculate with simple shapes like cylinders.

Particles in Unreal's Cascade have a module called Skeletal Mesh/Surface Location (read Unreal's documentation here) that allow particles to be emitted from the surface of a mesh -- though as far as I know, this is only works with Skeletal Meshes.

I've made a quick illustration demonstrating the two methods I've been working with to achieve this painted style, and each of their features/disadvantages that I'm aware of.

The first method is the one I've set up with all of my assets so far. It has been pretty faithful but there are some drawbacks. This idea was loosely based on this video I saw a while back, which essentially displaces the mesh several times to produce an imprecise, painted silhouette.

Here are some of the results I've gotten working with this today. Right now the material is just using one brush stroke opacity mask and the color is assigned based on the vertex normal, just to see it working with a bit of color variety. It's interesting to see it working and I'm looking forward to investigating some settings more thoroughly to see what kind of results I might be able to get on different sorts of meshes.

This method is also interesting because there are a lot of parameters I can play with on an individual stroke basis. Here's a sheet of brush stroke types that I can use as masks for each particle plane. In this way, I can easily vary the brush stroke type by sliding the UVs around this sheet. This sheet would be shared by all assets and different assets might use different stroke types. (Leaves might use something more loose while rocks might use something blockier and more chiseled.)

I've begun to test this method on more complicated meshes, and there are some issues - for example, some areas of the mesh I might not want brush strokes to generate on. More strokes are going to generate where the mesh is most dense. I might be able to control this density by making another version of the base mesh that will be bound to the same position in a Blueprint, but hidden. Additionally, this mesh would be a Skeletal Mesh to allow the particles to emit from the surface location, while the base mesh could remain a Static Mesh.

I would say that the main disadvantage of the "Mesh Shell Layer" method I've been using is that there's quite a bit of set up involved for the Blueprints for each asset. It relies heavily on communication between the asset's Blueprint and the material parameters per dynamic material instance assigned to each mesh shell instance. Creating instances of these mesh shell layers, assigning a new dynamic material instance per layer, and modifying the material parameters per layer becomes a hefty operation. I've got this consolidated into a function, but the issue is that this function can't easily be shared between each asset's Blueprint. A solution might be to put this function into a Blueprint Function Library that can be used across multiple Blueprints without having to copy and paste the whole graph, but since the function deals with adding Static Mesh components to the world dynamically, it can't be done in a Function Library function.

The mesh shell layer method might also prove a bit more intensive than the particle method. This could be improved by making the mesh shells even lower res versions of the base mesh. I recently had difficulty with this on the rock asset. The rock's base material was tessellated to allow for displacement with the wax. Each mesh shell layer would normally be assigned an instance of this base material. But there is no material override to turn off tessellation, so an entirely new material without tessellation would have to be made. Even though a lot of duplicate information can be consolidated into a Material Function, syncing up these two materials proves to be a pretty tedious process.

Overall though, both methods have their advantages. The mesh shell layer approach works well on the boat because each layer adds a painterly thickness to the thin-walled boat mesh. Each layer can have varying edge texture, be multiplied by a different color, and the holes eating through the structure of the boat can be clamped differently.

On the other hand, the particle method would work better for something with a more organic silhouette, like tree leaves.
The ability of particles to move may also come in handy for some effects as well. Thinking back on the boat rusting sequence, it would be a great in addition to the mesh shells to create the illusion of parts of the boat crumbling away as it deteriorates. I like that by working on the level of individual brush strokes, it gives a bit more visual variety and is more akin to the process of actual painting.

MORPHSCAPE: Boat sequence updates and looking forward

Revisiting the rust material and the formation of rust on the boat hull, I realized it wasn't very believable. I observed a lot of rust on metallic surfaces and realized that my rust was lacking a differentiation between the areas of rust flaking and the areas of rust dripping. The rust flaking areas often appear to be a deeper red orange, and the areas of dripping rust are lighter orange and fade off softly.

An updated screenshot of the rusting boat hull.

The boat's skeleton is slightly improved from before, a little bit less fantasy. I tried to make them appear more fragile, deformed, and decaying. One of the main problems with the previous skeleton iteration was that it didn't follow the form of the hull very well, and it looked more like a skeleton inside of a boat than a skeleton underlying the structure of the boat.   

I added the wheelhouse structure and applied the same rusting effect to it by painting keyframes for its rusting and deterioration pattern.

This is an early preview. The materials aren't well defined on the wheelhouse yet and I plan to add some further model-specific detail maps like AO, curvature, and normal.


I've been starting to work with the environmental effects for this sequence. My idea is that, as you approach the boat, the sand will start to ripple in front of you in waves that surge forward and "swallow" the water that the boat is floating on. The first component of this is to displace the sand, using variations of a sine wave. I have the sand rippling at the moment, but I'll be working on making it more directed and less random.

Details on the process so far:

I've made a lot of progress in terms of organizing my graphs. The boat is one Blueprint, BP_Boat, that initializes all of the mesh layers that make up the boat (for the painted effect) and assigns dynamic material instances to each mesh layer. This can be simply dragged into the editor and should automatically rust away based on distance, as well as connect with other relevant materials in the sequence.

BP_Boat sets material parameters in all of these dynamic material instances. Different components of the transition are determined by different curves. The wheelhouse deteriorates at a faster rate than the hull, and so I've set up different curves to control their rate (i.e. 60% through the transition, the wheelhouse has completely deteriorated.) The water drying up may be another curve, or the increasing size of the waves/sand ripples throughout the transition.

One of the main material parameters that BP_Boat sets is the Rust Transition, a 0-1 float value determined by the player's distance from the boat. This essentially flips through the hand-painted mask keyframes to animate the boat rusting and deteriorating. Mask keyframes are scalar and so they are stored in each of the RGBA channels to conserve texture space within the shader. I've painted the mask keyframes so that black on the mask is solid, 0 - .5 is where rust begins to form, and .5 - 1 is deterioration, contributing to the opacity mask for the material.

The rust drips are formed by shifting the UVs for the mask downwards and lerping it over itself incrementally several times. This gives a somewhat soft blend directionally downwards. Using this mask, I blend in a lighter rust material. In the standard mask, I blend in a deeper, flakier rust material. The masks, which originally are soft gradients, are blended with different noises - the flakes blend in a cell pattern at the edges of the rusted areas, where the drips use a more directional vertical noise. These are all blended in the UE4 shader so tiling can be adjusted, etc.

Another example of material parameter that BP_Boat sets is a mask based on which mesh layer the material instance corresponds to. This allows for effects like a color multiplier per mesh layer, or a UV offset per mesh layer. Connecting the various materials back to a centralized Blueprint has proven very helpful in directing all of the parameters for this sequence.

Some things left to do:

-Refine the wheelhouse materials, add glass to windows
-Replace placeholder sand and water materials
-Refine sand and water wave displacement
-Natural displacement of boat along with sand/water waves
-Sand particles blowing

I'm looking to finish the majority of this in the next couple of weeks, but some of these tasks might take a backburner for a bit as I start venturing into my idea for my next sequence.

Wax rock beach:

Over the past few weeks I've devised a much clearer plan and scope for this project that I feel like encompasses a lot of the visuals and moods I wanted to create. Essentially, the project is one transforming dreamscape environment. The primary environment is a beach with large, arching rock formations. The color palette will be desaturated and cool initially. If you approach the rocks, they will start to melt into wax wherever you aim, and the melting will spread throughout the rock from there. The color palette will shift to warmer as the wax melts into an amber-like color. The hero element in this environment will be a large rock formation that looks like a candle. My rough idea is that the candle wax rocks will all melt down into a pool of water, which is where you'll find the boat.

I'm interested in incorporating morph targets to actually melt these rock meshes down. The great news is that Canvas Render Target 2D has been fixed and is no longer crashing in UE 4.7, so I'll be continuing to work on my world canvas mask painting to material blend to the wax material wherever the player aims. The wax material will be much more translucent and use flow maps and displace areas achieve the appearance of wax dripping.

A broad summary of the tasks for this sequence:

-Create rock mesh pieces
-Create hero candle rock formation
-Create materials for rock and wax
-Implement morph targets for candle rock meshes
-Implement canvas render target 2D masking for material blending
-Transition into boat sequence


After approaching the boat, leaving only bones, vines will grow up the bones and grass will start to grow and overtake the sand. Trees will sprout up alongside the path you walk, and increasingly grow larger, with branches looming over you.

I've already started to develop a system within my tree Blueprint to animate the primary branches and the trunks. By developing this, I'll hopefully be able to "grow" the trees around the player.

Some known stuff I'll have to do:

-Hopefully add curved tangents on branches
-Add leaves to trees
-Implement tree animating system
-Add vine growth to bone material OR add a spline-based vine


Some progress on the boat hull rusting away sequence. I hand-painted mask keyframes that denote where rust forms and where deterioration starts to happen. Black on the mask is solid, black to 50% gray is rust forming, and 50% gray to white is where deterioration occurs. There are 8 keyframes in total. The result is getting there, but I still would like to do a bit of tweaking. Some of the keyframes towards the end jump a bit too much from one step to the next. 

This week I focused on developing this shader-based rusting/deterioration transition (23+ hours.) I had attempted a few different methods for making this happen. I originally was working with two separate masks to make this transition happen - one large detail control map (the overall, top to bottom rusting of the boat, holes, etc.) and one small detail control map (the ribbing of the boat.) This method I was first going with allowed for a lot of versatility in the timing of the transition and the blending, since it was all blended together in the shader, but finding a way to blend them together well in the material turned out to be probably too much overhead without necessity to get the visual result I needed. Painting all of these details in one mask and separating them by value seemed to allow for a more natural, organic blending across the whole boat. 


I've been playing around with materials to keep working at achieving the painted style I want for this project. Although I enjoy the look of hand-painted textures, my goal is to take this style beyond the typical flatly painted look by making it a convincing part of a 3D world with dynamic lighting. Particularly, I've been trying to create custom painted looking normal maps to accomplish this.

In my attempt to sculpt normal maps, I still hadn't been getting the look I wanted. Normal maps seemed to kill any painted illusion and overwhelmed my hand painted textures. Also, the chunkier style of my sculpts conflicted a bit with the more textural quality of my paintings.  While revising the material for my tree bark, I made a bit of a breakthrough in my method. 

These images are of the full tree with a modified, painted looking normal map. These also have the additional opacity mapped shell of the mesh over top, adding the textural edges on the silhouette.

You can see the effect a bit clearer in the image on the right. Two instances of my sculpted, tiling bark normal map texture are each warped slightly differently by noise patterns, and blended over top of each other by a dithered mask. This resulting warped normal map is blended back over top the original normal very slightly to only get a slight warped and dithered effect. It adds a slight pixel-level textural effect to the normal map. 

Then, I step the values of the normal map so that the transitions between different depths and angles are sharper and less gradiented, sort of like a posterization or a cel shading effect, but on the normal map. This helps in getting a more faceted look that I just couldn't achieve in my original sculpt.

It was upsetting to see at first that the effect of this normal map wasn't really coming through once my original painted diffuse color was applied, even with a really direct point light shining on the surface.

Something simpler works a bit better to show the normal map. Although I could work to get a bit more slight hue variations in this map, it's a good base. With the painted quality integrated into the normal map, it looks so much more well integrated into a 3D world. 









I spent the majority of this week working with materials to find this painted style (18 hours.) I also completely revised and reorganized my tree Blueprint (13 hours) and some things still need to be done to unbreak some things I broke in the process. I spent a bit of time with post process materials to soften the edges of objects (5 hours.) Lastly, I spent a lot of time trying to refine the look of my boat by painting model-specific detail maps, only to find that so far, I still like my iteration from last week better (12 hours.) Also, the development in my process with the normal maps makes the work I was doing on the look of the boat a bit unusable now, so I'll be working next week on applying this method to the normal maps I'll be creating for the boat.


My goal for this week has been to push the look of my boat further through defining its material and to work on the material blending between the painted boat material and the rusted boat material.

The opacity masked look  has been working decently to get a sense of a painted edge quality, but something I have been lacking that I really admire about paintings is the mix between hard edges and lost edges. The brush strokes on an edge aren't always all completely hard and opaque. The problem is UE4 masked materials don't allow for grayscale alpha values, they clip black and white- it's either 100% or 70% opacity. Translucent materials can provide this, but they're more intensive. 

I discovered that in the UE4 content examples, they demonstrate a technique used to fake translucence with masked materials by dithering. Dithering is a graphics technique used to simulate value using the density of a bunch of tiny dots when a color palette is limited. The DitherTemporalAA node in the material editor provides that kind of effect.

This same method seems to work to my advantage to give my opacity masks a softer look. Even though the little dots might technically be considered an artifact, it unexpectedly works to my advantage to give my opacity masks a softer, posterized/quantized look that aids in the painted style.

These images show a sphere with the dithered opacity masked by fresnel so it only shows up on the view angle dependent edges of the sphere. Here I've also tried multiplying that dithering by different brush stroke opacity masks to yield slightly different edge qualities. I had tried this same technique before with translucent materials to soften the edges by decreasing their opacity with fresnel as a mask, but the transition was very soft and it tended to look like the object was fading away.

Thinking forward, I would love to create a universal Material Parameter that can be modified by Blueprints across all materials to modify their edge quality, globally or locally. As things become more intense, the brush strokes at the edges might become less calm and more hard-edged and haphazard. I'd also like to do some modification to this edge effect based on how much light is hitting the object. I find in paintings, areas that fall into shadow tend to have lost edges more so than ones in the light. This kind of subtlety might really add depth to the visuals. Another parameter this effect could be dependent on would be distance or even focal point. Paintings use strongly defined edges to emphasize a focal point. Usually foreground elements might also be sharper, and background elements might have more lost edges if they are not the focus. 

I've been trying to find a good way to get the model specific detail I need to define the structure of the boat with more than just tiling textures. A method of sculpting or modeling a high poly and baking it down for additional normal detail doesn't really work well here. Normal maps I have tried to create so far all seem to break the loose painted look I'm going for. I hope to continue investigating non-traditional ways I can apply normal maps to bend light and enhance the painted look rather than diminish it.

This map functions as sort of a hand painted AO mask for the boat. It's not entirely complete as is-- I've found recently that instead of spending too long getting carried away painting for hours, I'll just test a little bit in one area to bring in and preview if I'm getting the look I need from it before I continue. You can see here I've only added some of the studs at the front of the hull. On top of this, I can overlay a hand painted curvature map mask to get some edge detail in specific areas. 

Both of these images above show the paint blending to the rust, which follows a similar but more clamped version of the opacity mask for the deteriorating parts of the boat (left.) I think the transition between the rust and paint could be taken much further by having even an additional mask that follows this one but helps to define the edge/transitional border between these two materials. Right now, to break up the blending with more than just an opacity blend, the mask on the left is multiplied in the material by a black and white hand painted brush stroke map for an additional textural quality to these edges.

You can see in the image on the left, the boat looks more thin-walled than the one on the right. This is where the duplicated mesh shells with different material instances applied work as cross sections to really help along the illusion of multiple layers of rust and textural depth.

Some more hand painted maps from this week - the first a rust-like mask I'm using to blend between the paint and rust, and the second a varied paint color for the painted boat material (which still needs more work.) 

I've found tremendous use in my work so far compiling assets within a Blueprint that can be dragged into the editor with all of the components it needs to be a complete asset. For example, for my boat, I have a few duplicate shells of the original mesh. This allows me to keep them all hooked up to one master asset I can drag around. 

Another benefit is that any global Material Parameter Collection values can be exposed to the Blueprint, so that I can modify material effects by clicking on the boat blueprint and changing values in the details panel. This is how I plan to manage and test the different sequential stages of deterioration that I've allowed for in the material. The transition will happen by blending/lerping the hand-painted mask that designates areas of deterioration to the next hand-painted mask with increased area of deterioration (the next stage or "keyframe" of the transition.) So to control the transition, I'll divide up 0 to 1 into the stages of the sequence, and at the end have one controllable value to slide back and forth between 0 and 1 to make the transition happen. This makes it easy to hook up to anything - distance, triggers, timelines or matinee. 



• Splines for primary branches can be dragged out.
• Tangents of splines can be rotated and scaled. 
• User can add up to 10 primary branches that split form trunk.
• Secondary branches are generated randomly along primary branches with awareness for other branches.
• Implemented random streams so you can modify a random seed to change the tree rather than have it change every time the Blueprint recompiles.

Next steps:
• As the tree branches out to further tiers with smaller branches, more small branches will be included in one piece. Experiment with what these pieces are.
• Continue tweaking parameters on randomly placed branches to get a more natural result.
• Figure out what the loose style should be to generalize detail as it becomes more busy (tiny branches and leaves.) 


• Modeled the boat to prepare for the boat deterioration dream sequence - this is an understatement. The modeling was the easiest part once I figured out how a boat actually works, which has been the majority of the problem solving. They have a very specific and subtle form and curvature that was a bit hard to capture initially until studying a variety of different boats. The shape of the hull is complete, everything else is in progress (railing is a stand in for scale.)

This came a long way from what I started on last week, which looked like a toy boat. I can't stress enough how important it was that I got the form of this right before proceeding. 


Very early tests just to demonstrate the concept of layered meshes to get a painted look with expressive detail (i.e. a rusted boat showing through to an underlying skeleton.)

• Mesh is added to a Blueprint as a component, duplicated multiple times (in this picture, 3x) 

 Each duplicate has a different material instance applied of the main material. 


• The main material is a hand painted B+W mask with a smooth falloff from black to white. This is to get enough of a gradient in the gray values to multiply in a tiling hand painted B+W brush stroke mask, that adds the textural quality to these masked edges.

• Rather than hand painting the mask for the boat in the exact way I want it, I compile the mask in the shader to allow for different levels of clamping on each material instance. This is what allows for the dynamic transitioning of the deterioration of this boat.

• Parameters for UV offset and tiling multiplier to ensure the brush stroke pattern is not repeated on each "layer" of the material instance.

 Each mesh is "pushed" in on within the shader on the vertex normals of the mesh. Each layer/material instance is pushed in an increasing amount. This makes it so the mesh does not have to have these layers baked in, so every time I want to modify the distance between each layer, I don't have to tweak the model in Maya and re-export. 


Concept wax sculpts to use in the creation of tiling textures. Not sure I like the result of these - they might be too sculpted and detailed for the painted style I am going for. 

I had a thought that I could use Substance Designer FX maps to generate procedural tiling brush stroke maps that follow a hand painted flow map for their directionality - a similar approach to this .

This might be somewhere I start to explore, especially to gain better control with the B+W brush stroke maps. Hand painting is great for some things, but hand painting tiling textures can often be a painstaking process, only to find you don't like the result after hours of painting. It is also hard to get a very evenly distributed level of detail when hand painting tiling textures. Naturally you hand craft some areas more than others, and these tend to be the areas that stick out when the texture tiles a couple of times.