Working more on the spline mesh trees this week. My original idea was to be able to drag out spline branches to an end transform point, but I've found that it's not possible to show widgets in editor for the Blueprints that are generated by other Blueprints. So, the end points have to be randomly determined (guided by parameters) instead of being able to freely drag out the end transform widget. This was a bit of a disappointment.

Overall this is a just bit less wonky than last week. I overhauled the method for how the branches should be placed so that I could enable them to have an awareness for how far the branches are being placed from each other. It follows a system of dividing up the trunk spline into sections, and each of those sections store a cumulative array of the locations of the branches generated, so that each time a new one is placed in that section it checks the array to make sure it can't place the current branch in a place within a certain distance of another one. 

The positioning of the branches and their angles haven't yet been properly parameterized. The branches are just sticking out pretty straight, similar to a pine tree, and the trunk mesh is a cylindrical tube that goes all the way up. 

I took a break from trying to procedurally generate the branches and decided to make an example tree that demonstrates what look I am going for in the end. I figured it would help a lot to have an end result to strive for when trying to get these branches to generate correctly.

I started creating this tree using Zspheres to form the armature of a trunk and the main three branches that the trunk split up into. I sculpted the base trunk at a low subdivision to get an organic feeling to the mesh and get the silhouette I wanted. This is actually a relatively fast process. 

In Unreal, this is loaded into a Blueprint as a static mesh component with my bark material applied, in addition to a duplicated version of the same mesh that has a different material instance of the bark material applied. The material instance pushes out the vertices of that second mesh along the vertex normal using World Position Offset. This creates an exterior shell identical to the original mesh.

The material instance on the additional shell mesh has a hand painted grayscale brush stroke opacity map applied. It uses the same diffuse texture as the original material, but offset and lerped with a noise.


Through this I've been learning a lot about painting tiling textures. The problem with the diffuse texture I had painted last week was that it wasn't getting the right amount of texture and color that I wanted. It also wasn't even enough all around, so some areas stuck out when tiling.

A lot of the time I'll paint all on one layer and not use layers and masking to their fullest. For this texture, I painted rough brush strokes all around the canvas on a new layer, painting and erasing to get the right shape, and then I used those strokes as a mask to paint within. This allowed me to loosen up my brush strokes by "coloring outside of the lines" but at the same time, keep them confined to the borders of the mask. I also achieved some unexpected results with blending options.

Here is another concept painting I did this week to outline some of my ideas for the familiar dream environment I'll be pursuing.

Next week I'll be working on something else-- a more static, visually based asset -- alongside my trees. Working with procedurally generated assets can be a bit frustrating after spending hours troubleshooting and cleaning up for little visual result. 


Procedural spline trees: 12 hours
Texture painting and material creation: 10 hours
Painting: 5 hours
Presentation, design doc, blog: 11 hours



This week I focused on material creation, using the tiling tree bark material as my guinea pig for testing. Developing the look of the materials is something I'm really invested in with this project. I've been trying to find the balance between realistic material definition and painterly stylization. One of the main downfalls with the typically seen hand-painted style is often that the light is permanently painted into the diffuse texture. Sometimes this looks nice, but it doesn't allow for the light to change and can come off as a bit flat. It's been especially important for me to find a method for creating normal maps that are a bit chunkier and planar looking, because that's sort of the way light hits are depicted in painting.

This image to the right is my first test to create a stylized bark normal map. This would be a tiling map used on all the trunks and branches of my trees. 

I created this normal map in Zbrush because I feel like sculpting is what will get me the closest to a more unique, hand-crafted style. However, in this first trial, I struggled a lot with sculpting this detail on a flat plane. My method was using the Clay Buildup brush to build up the surface, then polish it back down with Trim Dynamic and hPolish. It came out much too busy for my taste.



This second image is starting to get the sort of simplified, planar stylization that I want. 
The way I created this one was using the same sort of tools, but I discovered it was much easier to achieve this multi-directional faceted look by sculpting onto a cylinder rather than a flat plane.




This most recent iteration shows the diffuse and normal working together a bit better, resulting in something a bit closer to what I envision as the material style for this project. Again, he normal was done by sculpting in Zbrush and I painted a texture based on the normal. I'd like to add some more color variation in the strokes of the diffuse color to get that feeling of mixed paint and unexpected hues to make up an overall impression of a color. Bark is a naturally rough material and I'm still trying to figure out the role of the roughness map in my material, how much it might be needed for something like this, or if it could be painted in a certain way to enhance the look I'm trying to get. For example, maybe a patchy brush stroke roughness map with hard-edged transitions between strokes would provide an interesting painterly variation when light hits it. This is my next step in experimenting.

Aaand finally, here is some questionable progress on the procedural spline tree. It's umm. Getting there.

I've found that the best system for this is to add Blueprints within Blueprints. So, basically, the main Blueprint is the trunk Blueprint, which deforms cylindrical pieces along a spline that is editable per instance, and along that spline, branch Blueprints are placed at a random location, both in distance along the spline and radially around the circumference of the cylindrical trunk.  



Problems with this right now:

1. The branch mesh I'm using as a test a bit too sculpted on its own and most of the deformation of it should be left to the spline. I will also need maybe a few different branch meshes that can be interchanged for the sake of variety.
2. There are currently no parameters controlling how these branches are placed, other than how many of them there should be. So, in the future, there will need to be parameters that control the tendencies of the branches (angle, twisty-ness, length) as well as parameters for intelligently placing them so that some aren't clumped together, or colliding with each other.
3. There's an odd seam between the cylindrical pieces on the trunk that I need to figure out.
4. I want the trunk to be able to split into two large primary branches rather than go all the way up.
5. Still working on my method for placing the additional mesh planes with brush strokes along the spline meshes.


Thinking a bit on how I can make some interesting and unexpected trees. Here are some surreal variations on trees- one ties in industrial elements like pipes, the other forms some sort of pseudo house structure. More paintings defining my spaces are in the works.


Sculpting for normal map stylization: 13 hours
Painting diffuse/roughness maps to develop painterly style: 5 hours
Testing custom lighting in materials and post processes: 8 hours
Sketching and painting concepts: 2 hours
Blog: 30 minutes


This week I've made a lot of progress on the systems that are going to be key in achieving my goals and making my dreamscapes.


This week I stared developing a construction script for my customized, procedural trees that are going to be based around spline components. At it's base level, I'm generating this tree in segments of meshes that follow along a spline. I'm also procedurally generating additional meshes/planes on top of the base structure of the tree that have a hand-painted brush stroke texture for an opacity mask. These trees are a really great first test subject for working with principles of procedural construction and achieving a customized look. 

To start, I've just been working with the trunk and haven't created the algorithm for the placement of additional branches along the trunk. Here's a look into the functionality so far.


Nothing too fancy here. Basically, it calculates the bounds of an input mesh and places the next mesh at the distance along the spline that the previous one ends. It uses a while loop to continue doing this as long as the current distance along the spline is less than the total length of the spline. With this, I've used a tube as my input mesh.






This is also pretty simple. Just rotating the mesh to the current rotation at the distance along the spline. The effect opens a lot of possibilities though.






Something I'll need for a tree is to be able to scale down the meshes as the spline continues. So, the user defines a scale range, and a max scale. The base starts at the max scale and calculates each time a segments is placed how much of the scale range to subtract from the current scale based on the percentage of the current distance along the spline out of the total spline length.

The problem with this, initially, is that it leaves increasingly large gaps between the placement of the meshes as they continue scaling down further along the spline. This is because the placement is determined by the initial bounds of the mesh. This can be fixed by multiplying the Z distance of the bounds by the current scale multiplier. 


Something I want specifically for my trees is a really custom painted look. I'm looking to break up the silhouette of the mesh. I can do this by adding extra meshes that have brush stroke opacity masks on them. So, something I need to do is be able to randomly generate planes along the circumference of a cylinder that follows the spline. I've added a user defined parameter for how many of these planes are to be generate per segment (the density of the planes.) 

I used the mathematical formula to find a random point around the circumference, with the radius being determined by the current scale of the cylinder segment.


You'd think this would be easy, but this has been the most difficult step yet. I've run into quite a few hurdles that stumped me for a while, and it's getting pretty close to how it should be. But properly offsetting the location of these planes from the center hasn't been an easy task. I'm still working out the kinks of this (you can see where things get a bit tangly after the spline curves.) I have another method I'm trying underway, so updates on this soon.




Here are some examples of some effects I might be able to achieve when materials are applied to these planes. As of now, I've painted two different black and white masks with brush stroke patterns. For variation, it determines which mask to use by using the sine of the Z world position as the alpha for a lerp between the two textures. The color right now is just the vertex normal of the mesh, for the sake of demonstration and differentiating the color of the planes.

These are pretty early tests and I haven't done too much with them yet. The next steps here are to work on finding a method that balances a brush stroke abstraction with the proper amount of lighting, color, and depth information. A simple color with the brush stroke opacity mask is too abstract for the style I'm going for, so to achieve more material definition there will of course need to be diffuse, normal, and roughness information. I'm also thinking of developing a custom lighting method within the materials -- using a custom light vector from a Blueprint to control the effect of lighting in the material. I spent a bit of time breaking down some of the methods used in Epic's Stylized Demo. Even though the style of this demo is more like a toony illustration than an impressionist look, it works with some of the same concepts I'm trying to execute here and is a good reference for what's been done in this area.


The world canvas fake volume painting system is working! Here are a few pictures demonstrating what it does so far. Hopefully soon I will be documenting a more broken down explanation showing the Blueprints and materials for this.


You can "paint" white on black textures. There are 4 different planes with these dynamic canvas textures at different heights. The height of a traced hit from the player's weapon socket is matched to the closest plane on the Z axis. Then, the hit in X and Y are translated from world space relative to the world canvas Blueprint, to texture space.



In order to be able to draw to this and keep the last area you painted, rather than just having a white circle stamp follow your reticle around the texture and dump the last stroke, it keeps where you have drawn the last stamp. These planes are arranged in two 2x2 grids of planes and Scene Capture 2D cameras capture these compiled planes orthographically to two render target textures. The materials applied to each of these grids reference each other's render target textures, passing them back and forth (essentially recursively.) The only difference between the two materials is that one of the materials adds both the current frame to the last frame, whereas the other only references the last frame. With the two cameras alternating updating every other frame, this is what makes it possible to keep the previous frame and add the next one.


The textures for each cross section plane are laid out in a 2 x 2 square and captured to a render target texture using a Scene Capture 2D camera. Finally, this compiled texture is fed into a shader that interpolates the value in between these planes (using a flipbook texture that shifts the UVs to the corresponding texture for its depth.) This can be used as a mask within any material. In my case, every material should incorporate this to be able to be painted. I visualized the fake paintable volume with a series of stacked cross sectional planes. 


Now imagine this smooth opacity mask could be broken up and made into a more interesting visual transition if it were multiplied with a brush stroke pattern - maybe even an animating one to make the border even more undulating and ambiguous. Additionally, instead of having these planes represent the entire world space, I'd like to set it up to move based on the current player location -- if you reach the bounds of that area's canvas, the planes will move and your location will be the new center of these bounds. In this way, I can get a better texture resolution per area while keeping it less resource intensive.


Another quick concept I did this week. Trying to develop the idea of this ambiguous cave/rock formation structure in a dark, sandy, beach like setting. These rocks could work on a similar system to my trees.

Here's another concept of how I might incorporate some man-made structures into a dreamy landscape. I found this picture I had taken a while back of an interesting house with a door on the second floor and a bridge running from it into a hillside. I thought this was such an interesting and peculiar visual, and it reminded me of a scene in a dream I once had. I'd love to break up this structure into parts and have those scattered across the landscape as well.


It was recommended to me to check out this surreal game LSD: Dream Emulator, where the game is based on the creator's dream journal. Although it was made in 1998, there are quite a few things to be said about this game and what it was able to achieve with such ancient 3D graphics. It's surprisingly dynamic for its time - the way you navigate through these dream environments is not set in stone and changes based on what path you take. I'm not sure how in depth and variable the gameplay is, but it certainly feels pretty organic. The soundtrack is also dictated by numerous sets of patterns played in variable tones. For a game of its time, it ventures into some pretty new territory.

My project will probably be quite a bit less psychedelic and disturbing, since I'm going more for meditative and awe-inspiring. However, both this game and my project are in a format centered around environmental exploration. The main mechanism in this game is that the player is linked or transported to different environments by bumping into objects. My idea is similar, but takes this to the next level by showing this environment transform into another one around you. I also find it interesting how some environments seem recurring but not the same - they share the essence of that environment, but are slightly different.  This is something I'm also looking to achieve.

Anyway, it's maybe one of the weirdest things I've seen, but also pretty awesome. I think you'll just have to scrub through this yourself:


Tree system: 16 
World canvas system: 5
Material tests: 
Research: 1
Painting: 2 
Blogging/work breakdown and blog response: 
Some additional hours picking apart LSD Dream Emulator.


Almost out of the brainstorming stage. I had been overthinking my idea over the past week and I remembered I'm not making a game, and that I really want to focus on just a few things: developing a customized painterly style, developing a system to dynamically and seamlessly transition between two distinctly different dream environments, and using Blueprints to procedurally generate these environments as you explore. I'm not really concerned what the structure is, as long as I can successfully achieve these goals and showcase how I came to that visual output in a tech demo style playthrough.

I originally had been thinking of creating a house, but that started to feel constricting, too based in the development of the characterization of who lived there, and not true to the kinds of freeform spaces I wanted to design. 

Instead, I'm going to be creating two surreal landscapes. All sorts of unexpected surprises can be scattered across landscapes as you explore-- a house, a partial house, manmade objects -- this better reflects the disorganized nature of our dreams.

This week I found a cinematic done for the game Ryse: Son of Rome, and it really inspired me to focus on achieving a painted style in real-time 3D. This cinematic is pre-rendered, but some of the techniques to add a brush stroke quality could apply.  Here are some stills, concept art, and a break down of how it was made. 

Since I often spend too much time painting one thing and realize a few hours later I've turned up with something I didn't even feel passionate about, I forced myself into doing some very quick speed painting exercises to more quickly explore different solutions for the type of mood, color palette, and environment I might want to create.

I'm starting to like the idea of one environment being more warm, inviting, sunny, and foliage based, while the other would have more rock formations, sand, a cooler color palette, and feel more foreboding.

I also did a quick painting to illustrate the transition between two dream spaces at their border.

This week, I successfully got the multiple cross section canvas system working, so now it matches your hit to the nearest Z height and writes to the corresponding texture. The next step is to get it set up so that I can interpolate between these textures to be able to unmask fake "volumes" in shaders. I might edit this and add pictures of this working in the near future.

I've also been doing a lot of testing to develop this painted style, with broken up painted edges and the correct amount of material definition -- no results yet, but this next week I should be moving into more production, less pre-production. Now that I have developed more of a direction in terms of what I'll be creating and my pitch video is complete, I'm excited to start developing my first proof of concepts for my systems and assets.

Pitch video - planning, script, rehearsing, recording, and editing: 6 hours
Painting, sketching, brainstorming: 8 hours
Blogging/blog response: 1 hour
Testing style and shader development, world canvas system: 6 hours


Over the past week I've been thinking a lot about the format of the experience I want to create, which spaces I'd like to build, and what possibilities for dream-like transitions those might provide.

I have decided that the dreamscape's physical manifestation will take the form of clouds. These clouds can function as a pervasive visual and transitional element. Clouds are a common symbol of dreams, and for good reason. Just like clouds, dreams are sort of nebulous, intangible, and evoke powerful visceral reactions within us. A metaphor I'm working with is that if our mind were a water cycle, our brain would produce thought condensation, which would evaporate into dream clouds. Then, our heads would be literally and figuratively in the clouds. Working with this idea, I could allude to this idea by making the clouds become more dense in areas where dreams begin to invade reality.

So far, I have done a painting conceptualizing what those clouds might look like, in color and atmosphere. I'm going with a color palette that reminds me of early morning and sunrise. Early morning reminds me of dreaming because I associate it with awakening from a slumber. Both early morning and dreaming feel very solitary and introspective.

I'd like for the main environment to be a two-floor apartment -- I find we often dream about places that are personal to us or very familiar, and so a home setting feels fitting. At the same time, places we know in our dreams are often amorphous, inexact versions of the place we know in real life that maintain a generalized essence of that space, but with something slightly off. I might be able to create this sense of a disorienting space by randomly switching the placement and types of objects that are in the room each time you come back to it, or even changing the layout of the house. The player might also encounter new and unexpected anomalies in the room -- objects that are completely out of place.

As you explore the house, I want to emulate the nature of the illogical sequences in our dreams, since nonsensical and peculiar happenings are a frequent occurrence. Dreams are disorganized and subject to fleeting thoughts of the mind. For this reason, I'd like the player to explore the house and experience lapses into dream sequences as they look around, seeing morphing objects or sometimes drifting and transitioning entirely to another area for a moment.

While searching for some inspiration, I came across this painting which conveys really well the type of cloud-like transition I want to try to create. 








I've been doing some tests in Unreal Engine 4 with spline meshes. Not much to show yet, but I feel like these will really enable me to create some neat morphing assets, either by deformation along a curve or extension of modular pieces along a curve. Here I've made the beginnings of a Blueprint for a fence type asset, where vertical post meshes are placed at a user-designated interval along the spline. You can drag out the spline points differently per instance of this Blueprint, and the placement of the meshes follows the curve. In picture 3, you can also see I have made it so the user can choose the construction priority of the fence - they can choose whether they want to set the length of the span between the posts, or by the number of posts that they want. In picture 4, I'm demonstrating an "adapt to surface" option I have added, where the placement of the meshes will snap to a surface below it. This is ideal for a fence on a hillside. It opens possibilities for animating the points on this curve as well, which is great for surreal ambient animations.


I am using UE4's new Canvas Render Target Blueprint to dynamically write to a texture that matches up to the size of the world. The player can draw in texture space wherever they are aiming their reticle on the world space plane. By drawing with white on a black texture, the player can be designating in world space a mask. I can create a fake volume by stacking these cross-sectional planes in Z space and interpolating between them. This mask could be fed into the shaders of all objects in an environment to "unmask" them. 


I've been investigating different methods of creating clouds that would be on the ground plane. The best solution I've found for imitating the shifting form of clouds is to use Cascade, the particle system in Unreal. The parameters of the particles can also be changed within Blueprints, which opens a lot of possibility for some interesting transitions using the clouds.

This method of fake volumetric color blending seems to give great control over determining the colors of the clouds in a cube map style (what color is it at the bottom? top? back? front? right? left?) Not much has been done in the way of getting clouds to be properly lit and self shadowed. I might be able to do this by using this method and hooking up these volumetric colors to the angle of a custom light vector. 


Testing and experimentation: 8 hours
Blogging and blog feedback: 2 hours
Pitch video planning: 2 hours
Painting and sketching: 7 hours
Research: 3 hours

This week I feel like I am beginning to visualize and solidify what this project is going to be. More finished sketches for the style of the home to come soon, as well as some early testing with clouds.


This week I've continued working on procedural spline trees, the world canvas masking system, troubleshooting those, and spent a lot of time trying to flesh out my idea conceptually a bit more. 


The strategy I'm using to develop my trees has changed a bit - last week I had started building it by generating cylindrical segments along a spline. My new idea is similar, but instead I am using spline meshes to deform a single mesh between each point on a spline  - so the deformation of a single mesh defines the curve of the tree rather than being comprised of a bunch of smaller segments.

Originally I had been worried about too much stretching between these points, but I don't think I am going to need to deform so much per segment that this would pose a problem. This first image shows this using a cylinder as the spline mesh and offsetting the vertices along their vertex normal by a noise projected in world space in the shader using World Position Offset. It's got a tiling paint strokey normal map on it that I was testing with. Right now, as is, this is pretty bland looking and there are some seam issues.

 Alternatively, it might be more suitable and still provide enough variation to create a customized trunk mesh, or maybe a couple customized trunk meshes, and simply deform them between a start and an end point. This is what I'm doing here with this branch mesh. It seems like this can still provide quite a variety of outputs in silhouette. The deformation of this branch is also affected by the scaling and rotation of the tangents of these start and end points.

My method for breaking down these trees into parts is nothing too complex - a spline mesh for the trunk, the possibility to generate branches along the trunk, the ability for the trunk to split off into several different main branches, secondary branches that can come off of those branches, and finally, tertiary branches that will most likely be planes with texture maps of branches and leaves. The trunk, primary, and secondary branches would be controllable per-instance by user-controlled deformation along spline meshes.

I spent a bit of time trying to figuring out the style for my creation of normal maps. I know I want something a bit chunkier looking, that imitates the blotchy nature of paint strokes. This normal map was sculpted in Zbrush. I'm excited to start working with diffuse and roughness to begin developing the look I want in my materials.

Here's a quick video of a test showing my still in progress world mask based on drawing to a dynamic texture on cross-sections. I worked out a few quirks this week, but this still requires a lot of additional layers to get it looking like a nice, stylized, smooth unveiling. You can see where I paint on each cross-section plane is hooked up to the opacity mask of a shader that has been applied to a sphere. Normally, in game, these planes wouldn't be visible. Instead of using this as a constant mechanism, I'd like it to be triggered only occasionally, at areas where the transitioning to the other dreamscape would occur.


This week I zoomed out to the bigger picture of my project again. Figuring out the level of how far I want to take the idea of an emergent, procedural environment has been a major hurdle for me. I want this to feel fluid, but having so little set in stone has been tough and abstract to think about and deal with, and achieve the sense of an intentional environment that I want. I had initially thought about even making the landscape in tiles and randomly generate those as you move through space and reach the end of one tile. Working too much with procedural generation of the world might inhibit me from designing and directing the very purposeful space and mood that I want. I've decided that I'd still like to work with procedurally constructing assets, because I feel like that enables a lot of potential for a living and dynamic environment, but I feel like the actual landscape and for the most part, the space different objects occupy, will be predetermined. 


I've developed my ideas for the two dreamscapes thematically, based on the mood I get from the kinds of spaces my mind makes up.

The first space is based around the idea of things that are familiar to us, a part of our every day lives, and the way that we combine those experiences in different ways. This dreamscape will be a woodsy area, warmer in color palette with the feeling of patchy, early morning light. Incorporated in this natural environment will be unexpected manmade features; city elements like traffic lights strewn between hills across a valley, architectural elements on trees, cross sections of houses scattered across the landscape. Pieces of our every day lives in surreal, juxtaposed vignettes. This space would be dream-like and disorienting in the way that something isn't quite right, but not unsettling. A dynamic element might be making trees grow with where you are looking.

The second space is a bit darker, a beach with an overcast sky, and large, encompassing, cage-like rock formations. I intend this space to feel a bit more claustrophobic and isolated. The tone would be more in line with our fears, the forgotten, and the foreign. This wouldn't be necessarily a nightmare, just more reflective of our thoughts that are a bit less comforting, unfamiliar to us, and colder. I'd like to incorporate the visual of a large candle melting, but melting a golden amber substance that encapsulates things that have been lost or forgotten, like old electronics, junk, or maybe even graves. Cave entrances could lead into attic-like spaces. I could envision an interesting transition with a boat sitting on the water and the water seeping away, leaving the boat immobile on the sand.

I want the ability to transition back and forth between these spaces while navigating - there won't be an "ending" or a linear chain to follow. These transitions would be at predefined locations/interactions but the transition would still be open and responsive to the player. 

Interpretations of dreams seem to try to look for deep-seated psychological reasons why you might be dreaming about something, like that your teeth falling out is symbolic for your fear of losing money. In my opinion, that's a pretty far reach and a very overgeneralized assumption.

I find the reason why we dream things is pretty unfounded. Sometimes it's a direct and obvious correlation to something happening in our lives. Other times our brain just pairs up various experiences and directs them in a way that takes on the way we might be feeling about something, but isn't directly related to events in our lives. And the remaining times, we might dream things entirely haphazardly. Our brains just pick something and our subconscious reacts and gets carried away with it. 


Testing stylized normal map creation: 5 hours
Dynamic masking system: 4 hours
Tree and spline testing: 6 hours
Blog and blog response: 2 hours
Painting: 2 hours
Brainstorming and conceptualizing: All of the rest of my free time
"Researching" dreams: Every night, and sometimes naps

Next week, I'll be working towards a complete method for generating the tree asset in its entirety, deformable by splines per instance with procedurally generated sockets for branches. I hope to also show some tests with materials and modifying the look of the world mask.