Today I was inspired by this article that outlines an approach to achieve a non-photorealistic, painterly style. The idea is that particles are placed on the surface of a mesh with brush stroke shaders applied. There's not really any reason this can't be translated into Unreal, so I figured I'd give it a shot.
My original idea months ago was similar to this method, except I was scripting the placement of plane meshes onto the surface of a mesh using Blueprints. The problem was, this was only feasible to calculate with simple shapes like cylinders.
Particles in Unreal's Cascade have a module called Skeletal Mesh/Surface Location (read Unreal's documentation here) that allow particles to be emitted from the surface of a mesh -- though as far as I know, this is only works with Skeletal Meshes.
I've made a quick illustration demonstrating the two methods I've been working with to achieve this painted style, and each of their features/disadvantages that I'm aware of.
The first method is the one I've set up with all of my assets so far. It has been pretty faithful but there are some drawbacks. This idea was loosely based on this video I saw a while back, which essentially displaces the mesh several times to produce an imprecise, painted silhouette.
Here are some of the results I've gotten working with this today. Right now the material is just using one brush stroke opacity mask and the color is assigned based on the vertex normal, just to see it working with a bit of color variety. It's interesting to see it working and I'm looking forward to investigating some settings more thoroughly to see what kind of results I might be able to get on different sorts of meshes.
This method is also interesting because there are a lot of parameters I can play with on an individual stroke basis. Here's a sheet of brush stroke types that I can use as masks for each particle plane. In this way, I can easily vary the brush stroke type by sliding the UVs around this sheet. This sheet would be shared by all assets and different assets might use different stroke types. (Leaves might use something more loose while rocks might use something blockier and more chiseled.)
I've begun to test this method on more complicated meshes, and there are some issues - for example, some areas of the mesh I might not want brush strokes to generate on. More strokes are going to generate where the mesh is most dense. I might be able to control this density by making another version of the base mesh that will be bound to the same position in a Blueprint, but hidden. Additionally, this mesh would be a Skeletal Mesh to allow the particles to emit from the surface location, while the base mesh could remain a Static Mesh.
I would say that the main disadvantage of the "Mesh Shell Layer" method I've been using is that there's quite a bit of set up involved for the Blueprints for each asset. It relies heavily on communication between the asset's Blueprint and the material parameters per dynamic material instance assigned to each mesh shell instance. Creating instances of these mesh shell layers, assigning a new dynamic material instance per layer, and modifying the material parameters per layer becomes a hefty operation. I've got this consolidated into a function, but the issue is that this function can't easily be shared between each asset's Blueprint. A solution might be to put this function into a Blueprint Function Library that can be used across multiple Blueprints without having to copy and paste the whole graph, but since the function deals with adding Static Mesh components to the world dynamically, it can't be done in a Function Library function.
The mesh shell layer method might also prove a bit more intensive than the particle method. This could be improved by making the mesh shells even lower res versions of the base mesh. I recently had difficulty with this on the rock asset. The rock's base material was tessellated to allow for displacement with the wax. Each mesh shell layer would normally be assigned an instance of this base material. But there is no material override to turn off tessellation, so an entirely new material without tessellation would have to be made. Even though a lot of duplicate information can be consolidated into a Material Function, syncing up these two materials proves to be a pretty tedious process.
Overall though, both methods have their advantages. The mesh shell layer approach works well on the boat because each layer adds a painterly thickness to the thin-walled boat mesh. Each layer can have varying edge texture, be multiplied by a different color, and the holes eating through the structure of the boat can be clamped differently.
On the other hand, the particle method would work better for something with a more organic silhouette, like tree leaves. The ability of particles to move may also come in handy for some effects as well. Thinking back on the boat rusting sequence, it would be a great in addition to the mesh shells to create the illusion of parts of the boat crumbling away as it deteriorates. I like that by working on the level of individual brush strokes, it gives a bit more visual variety and is more akin to the process of actual painting.