Being wrong, again
So, I was wrong again. You can’t blend normals. This should have been obvious to me, seeing as blending normals might result in two normals nulling eachother out if you blend them. Take a normal, lets say it’s 0.5,0.5,0.5 pre-normalized, and you blend it with a vector -0.5, -0.5, -0.5 with the blend factor 0.5. What do you get? That’s right, zero, you get zero, resulting in complete and utter darkness. This is not good, for tons of reasons, many of which I shan’t explain. Just try constructing a TBN matrix where one vector is 0,0,0. Not so easy! So I removed it, and sure, objects which use alpha (mainly transparent surfaces) won’t get lit correctly seeing as the lighting should take place per alpha-object instead of screen-space. On the other hand, it’s hardly visible, seeing as alpha-objects often consist of glass, where as many layers of glass results in an additive effect because of reflections in the light etc, making the rendering error hard to see.
Good news are that I finally got CSM to work! It wasn’t easy, and I’ll explain why. As you may or may not know, Nebula saves the depth of each pixel as the length of the view-space position of that pixel. What this allows us to do, is to take any arbitrary geometry in the scene, and calculate a surface position which lies in the same direction that we are currently rendering. Why is this important then? Well, as you see, saving depth in this manner allows us to calculate the distance between lets say a point light, and the shaded surface positions it should light, determine the distance between these two points, and thus light the surface. Great! We can literally re-construct our world-space positions by simply taking the depth-buffer and multiplying it with a normalized vector pointing from our camera to our pixel. Now comes the hard part. What do you do, when you have no 3D-geometry to render, but instead have a full-screen quad, as is the case with our global light. After finding this guide: http://mynameismjp.wordpress.com/2010/09/05/position-from-depth-3/, I tried it out, and got some good and some not so good results.
What you see is the same scene, using the above mentioned algorithm to reconstruct the world-space position. If you’re lazy, this is the general concept. In the vertex shader, get a vector pointing from the camera (that’s 0,0,0 for a full-screen quad) to each frustum corner. Then, in the pixel shader, normalize this ray, take the camera position + normalized ray * sampled depth. Sampled depth is the length of the view-space position vector. I thought this could be the result of a precision error or whatnot, but the solution was a proof that it wasn’t. Instead of using a ray and such to recreate this, I simply let my geometry shader (not the GS, but the shader which is used to render deferred geometry) output the world-space position to a buffer which would then hold these values. Using that texture to sample the data, the big black blob disappeared. Somehow, the method used for calculating the world-space position has to lack in accuracy, so I pondered. Normalizing the vector could not be correct. Why? Well, consider the fact that I want a vector going from the frustum begin to each pixel on the far plane. Each vector in that respect is not normalized, seeing as the distance straight forward is shorter than it is to a corner, which can be derived using Pythagoras theorem. Currently, the shadows work, and I’ve baked them into an extended buffer, using both depth as the alpha-component, and the RGB as the world space position components, to save using another render target. I’m not satisfied yet though, seeing as the explanation on mynameisjp.com must have some validity to it, but for now, I’ll keep the data in the render target.
Deferred shading revisited
I haven’t really had much time to write here, been busy with lots of stuff.
When testing the rendering pipeline we’ve hit a couple of annoying glitches with the rendering. The first was that UVs and normals for FBX-meshes was corrupt. The fix for the UVs was pretty straight-forward, just flip them in Y (Why?! I have no idea…), but the fix for the normals wasn’t quite as intuitive. First of all, I would like to start off with saying our target platform is PC for our fork of the engine, just so you know. Many of the rendering methods previously in the engine had lots of focus on compressing and decompressing data, in order to save graphics memory and such, but seeing as problems might occur oh so easy, and debugging compressed data is oh so hard, I’ve decided to remove most of the compression methods in order to be able to get a good visualization of the rendering.
The first thing, which is ingenious, is to compress normals (bumped normals of course) into an A8R8G8B8 texture, compressing the X-value in the first two components (A and R) and the Y-value in the other two (G and B). Z can always be recreated using the algorithm z = 1 – x * x – y * y, seeing as a normal has to be normalized. Anyways, the debug texture for such normals would be a brilliant Red and Blue-Green texture, which is impossible to decode by sight, so what I’ve done is to break the compression and use the normals raw. Well, then another problem rose, raw normals would need a texture with the format R32G32B32, one float per each normal, right? Well yes sir, you are correct, but too bad you can’t render to such a texture! Using a simple A8R8G8B8 and just skipping the A-value would give such poor precision, artifacts would be everywhere. Instead, I had to pick a A32R32G32B32 texture as my render target. Wasteful? Yes! Easy to debug? Yes! Beautiful and precise normals? Hell yes! I’d say with a score of +1, it’s a go!
Right, we have two enormous textures to render normals to (one for opaque, one for alpha), what else can we do?
Well, Nebula was aiming for a very broad variety of consoles, ranging from DS to PC to PS3. I’m just shooting from the hip here, but that might be the reason to why they implemented the light-prepass method of performing deferred shading. The pre-pass method requires two geometry passes, one for rendering normals and depth, and the second for gathering the then lighted buffer, together with the diffuse, specular and emissive colors. That’s nice, but it requires two geometry passes (which can get really heavy with lots of skinned characters). The other, more stream-lined method is to render normals, depth, specular and diffuse/albedo to four textures using MRT (multiple render targets), generate light using the normals and depth, and then simply compose everything using some sort of full screen quad. Yes! That sounds a lot better! The only problem is that we need four render targets, something which can only be done on relatively modern hardware, but not for some consoles.
Anyway, the deferred shading method does not incorporate a method to deal with alpha. That does NOT mean you can’t light alpha deferred!
The solution is to render all alpha objects to their own normal, specular, albedo and depth buffer, use them for lighting separately (requires another light pass using the alpha buffers as input), and then in a post-effect, gather both opaque color and alpha color, then interpolate between them! Easy peasy! The way I do it is:
/// retrieve and light alpha buffers
float4 alphaLight = DecodeHDR(AlphaLightTexture.Sample(DefaultSampler, UV));
float4 alphaAlbedoColor = AlphaAlbedoTexture.Sample(DefaultSampler, UV);
float3 alphaSpecularColor = AlphaSpecularTexture.Sample(DefaultSampler, UV);
float4 alphaColor = alphaAlbedoColor;
float3 alphaNormedColor = normalize(alphaLight.xyz);
float alphaMaxColor = max(max(alphaNormedColor.x, alphaNormedColor.y), alphaNormedColor.z);
alphaNormedColor /= alphaMaxColor;
alphaColor.xyz *= alphaLight.xyz;
float alphaSpec = alphaLight.w;
alphaColor.xyz += alphaSpecularColor * alphaSpec * alphaNormedColor;
/// retrieve and light solid buffers
float4 light = DecodeHDR(LightTexture.Sample(DefaultSampler, UV));
float4 albedoColor = AlbedoTexture.Sample(DefaultSampler, UV);
float3 specularColor = SpecularTexture.Sample(DefaultSampler, UV);
float4 color = albedoColor;
float3 normedColor = normalize(light.xyz);
float maxColor = max(max(normedColor.x, normedColor.y), normedColor.z);
normedColor /= maxColor;
color.xyz *= light.xyz;
float spec = light.w;
color.xyz += specularColor * spec * normedColor;
alphaColor = saturate(alphaColor);
color = saturate(color);
float4 mergedColor = lerp(color, alphaColor, alphaColor.a);
A simple lerp serves to blend between these two buffers, and the result, mergedColor, is written to the buffer.
Sound good eh? Well, there are some problems with this as well! First of all, what about the background color? Seeing as we light everything deferred, thereby also light the background, wherein the background lighted will serve as our final result in the gather method stated above, we will get an unexpected result. Well, what we will get is some sort of incorrectly lighted background which changes color when the camera moves (because the normals will be static but the angle to the global light will change). So, how do we solve this? Well, by stencil buffering of course! Every piece of geometry draws to the stencil buffer, and thus, we can quite simply just ignore to light and gather any pixels outside our rendered geometry, but without having to render our geometry twice! And so, by simply clearing the buffer which the gather-shader writes to, to our preferred background color, we can have any color we like!
So that’s solved then, alpha and opaque objects with traditional deferred shading with custom background coloring, sweet!
Oh, and I also added bloom, easy enough, render bright spots to a downsized buffer, blur it to an even more downsized buffer, blur it again, and again, and then sample it, et voila, bloom!
So, conclusion, what did we win from this, and what did we lose? We got better normals and lighting to the cost of some graphics memory. We removed half our draw-calls by removing a complete set of geometry passes. We managed to optimize our lighting per-pixel by stencil-buffering, which in turn yielded the ability to use a background color. We managed to incorporate alpha into all of this, without any hustle or expensive rendering. All in all, we won!
Also, here are some pictures to celebrate this victory:
Deferred rendering with alpha. This picture shows a semi-transparent character (scary) with fully transparent regions (see knee-line) and correct specularity
Bloom!
Lighting using the A8R8G8B8 texture format (low quality)
Lighting using the A32R32G32B32 texture format (high quality)
PSSM and CSM
I’ve been hard at work during the last 4 days with global light shadowing. One might think this should be extremely simple, but there is a hitch with global ligths, they have NO position! So how does one render shadows from a source when there is no source?
The solution is found in two different algorithms, one called PSSM (Parallel-split Shadow Maps) and CSM (Cascading Shadow Maps). Firstly though, I should explain what a shadow map actually is. A shadow map is a light-perspective rendered buffer of the scene, which basically means that every light source which needs to cast shadows has to render every object visible to the light as a depth-based buffer. The shadow map basically just stores depth saved from a lights point of view. This of course means that a non-shadow casting light source is significantly faster than the shadow-casting ones.
Now when we’ve gotten that out of the way, we can go back to global lights. As you may or may not have realized, in order to render a shadow map you need a position of a light source, and a direction in which it emits light (except for the point light, which only has a position). The problem is that a only has the direction, so there is no point of view from where one can render the scene. To resolve this, there is a very non-intuitive way of doing it, and that is to split the camera view-frustum into different sections (see Cascades and Parallel-splits). The light source is then rendered from outside the scene bounding box, just to ensure every shadow-casting object gets into the buffer. In order to handle rendering the entire scene into the shadow buffer without needing an enormous buffer, one renders the different splits into different textures, but using different projection transforms to do so. So the first buffer surround the closest area of the camera, the next split or cascade overlaps a larger area and so forth, until it reaches the maximum limit which is based on a given distance. This way, shadows very very far away will be rendered with a very low resolution, seeing as the viewport for that area is very big, so each item doesn’t get that much space.
Nebula used to use this algorithm in version 2.0, without deferred rendering, so re-implementing this algorithm in Nebula 3 with DirectX 11 is a bit of a challenge. I was so in the dark I actually thought I’d just been lucky with everything I’ve ever done right, because how hard I tried I couldn’t get the shader to switch between the shadow maps based on the depth to the pixel. I used a float array to send split distances from the CPU to the shader, but the comparison ALWAYS failed. After about 3 days of getting that part to work, I realized I just had to try comparing the values without a variable, but instead hard-code them. To my amazement, that worked fine, perfectly fine. By this point, I honestly started to consider if Nebula didn’t properly set float arrays to the shader. I debugged the pixel shader in PIX, and the values where completely fine. The funny thing was that the compiled shader description clearly showed that the float array, consisting of 5 floats, was 80 bytes in size. 80! Last time I looked, a float was 4 bytes, 5 * 4 = 20, not 80. That was when I realized what was wrong, for some reason, the compiler (or driver) seemed to think that an array of floats was equivalent to an array of float4, but could nonetheless fetch the values without using swizzling. When I changed the array into an ordinary float4 (obviously loosing a value), the depth test worked perfectly! I’m going to be a bit careful, because this could be simple ignorance, similar to the hull-domain shader problem, but there might be some sort of compilation problem going on here. Seeing as an array of float apparently had the same buffer size as a float4 array of equal size, and because fetching values always returned the incorrect (I believe I got infinite, but it’s somewhat hard to tell on a shader-level) values.
Right, back to PSSM. It currently looks like utter crap. It looks so bad infact, that I’m not even going to show you, because of shame and all that. There are two problems at the moment. The first one is that all geometry gets rendered with the camera inverted, so everything gets rendered backwards, resulting in very strange shadows, the most funny effect being that the object which casts shadows is shadowed by itself. The other problem (may or may not be related to the first one) is that in the seams between the two different split maps is clearly visible because of the radial pattern when comparing depths in screen space. If you’ve had the patience to read this far with the knowledge that you won’t be seeing any pictures, I salute you! Oh and, we’ve started this years game project, which is being made in Nebula, both of which can be followed here: http://focus.gscept.com/gp2012-1/ and here: http://focus.gscept.com/gp2012-2/.
HBAO and ESM
Lately, I’ve been hard at work with the stuff I like the most, shading! First, I reimplemented the old Nebula exponential shadow maps (which looks great by the way) in DX11 for pointlights and spotlights. The direction light is a little trickier, but I will be all over that shortly. I also took the chance to remove the DSF depth buffer into a more high-precision depth-buffer. The old DSF buffer stored 8 bits normal id, 8 bits object id and 16 bits depth, whilst the new buffer stores 32 bits pure depth, removing all the halo-problems.
Bringing shadows to life sure wasn’t easy, but it was well worth the while. Nebula had a limit to only use 4 shadow casting lights per frame, I thought I could boost that to make it use 16 (newer hardware can handle it). I’ve also begun working on the global light shadow algorithm, and I thought I’d start with making the PSSM method work. The reason why I chose PSSM is because major parts of it have already been implemented, but also because it seems like a very valid concept.
To handle AO, and setting variables for the AO, I decided to make a new server for it, which was fitted right next to the light and shadow servers (seeing as it has to do with lighting). The method for computing screen-space ambient occlusion is called HBAO, which basically samples the depth buffer using an offset texture (random texture). It uses the current sample point along with the sampled point to calculate not only depth difference, but also the total difference in angle, giving off a strong occlusion effect if the angle between two surface tangents differ a lot. More of the algorithm can be found here: http://www.nvidia.com/object/siggraph-2008-HBAO.html.
The introduction of the AOServer also allows for setting the AO variables live (whenever there is an interface available). Here are two pics showing the awesome new graphics.
The picture on the right shows a real-time AO pass, and the right shows a scene with 3 shadow-casting lights. Pretty nice right!
Also found a couple of things I want to do with Nody. First, I changed so one can create different types of render targets. If one wants to write to a 1, 2, 3 or 4-channel texture, one should have no problem doing so. I’m also considering being able to add and manipulate render targets between frame shaders. For example, the AO-pass uses the DepthBuffer from the main frame shader, and the main frame shader uses the AO-buffer from the HBAO frame shader. I also want to be able to add an MRT or an RT directly to the output node, and then decide how many channels one wants to use. When this is done, one should not be able to add new render targets or MRTs. This is to avoid silent errors which might occur if the render targets have their positions switched. Also, attaching render targets to the output nodes should also allow you to pick from ANY frame shader, instead of just the current one.
It would also be awesome to have the ability to use compute-shaders in Nody, as well as nodes which lets you code everything freely.
Plug-in, Nebula-out. Get it?
I’ve been hard at work getting a Nebula 3 plugin for Maya to have all the features we’d want. Why you may ask, isn’t the pipeline awesome as it is? Well, the answer is no, no it isn’t. It works, but it is far from smooth, and the Nebula plugin aims to address that. The plugin is currently only for Maya, but there will be a Motion Builder version as well. The plugin basically just wrap FBX exporting, and makes sure it’s suitable for Nebula by using a preset included with the Nebula distribution. It also runs the FBX batcher which converts the fbx-file to the model files and mesh files. It can also preview the mesh if it’s exported, and will export it if it doesn’t exist already. This allows for immediate feedback how the model will look in Nebula. It also tries preserve the shader variables, but it’s impossible to make it keep the material. That’s because DirectX doesn’t support setting a vertex layout with a vertex shader with a smaller input than the layout. This is a problem because converting a skin from static to skinned will cause Nebula to crash, seeing as the material is preserved between exports. So the plugin offers a way to get meshes, including characters, directly to Nebula, which is very nice indeed.
I’ve also been working with getting a complete Motion Builder scene into Nebula, and I actually got Zombie to work with all features. This means the skin, along with more than one animation clip (yay) can be loaded into Nebula seamlessly by simply saving the Motion Builder file and running the exporter. I will probably make a Nebula 3 plugin for Motion Builder as well, so we can have the exact same export and preview capabilities as in Maya.
I know I have been promising a video showing some of the stuff we’ve done, but I just haven’t had the time! Right now, I will start working on the documentation for our applications so there are clear directions for anyone who wish to use them (mainly our graphics artists here at Campus) . The plugin already redirects from Maya to three different HTML docs, each of which will describe the different tools. That’s nice and all, except for the fact that the HTML docs are completely empty.
Characters continued
So I ran into a problem with characters when I tried to load the FBX example Zombie.fbx. It turns out the skeleton in that particular scene isn’t keyed directly, but indirectly using a HIK rig. So when I tried to read the animations from the scene, I got nothing, every curve was a null pointer. I’ve tried back and forth just getting the character in the scene, but the HIK rig doesn’t really NEED to follow the actual skeleton, so there is no exact way of knowing it will fit. Instead, I go into Maya and bake the simulation. This won’t bake the animation to be per-vertex, but instead it will make sure every joint is keyed identically to all it’s effectors. The reason why I don’t really want to read characters in total is not because of simple rig-to-joint connections, but also because of effectors. So a skeleton might be linked to another HIK rig, but the animations are not identical, just slightly identical. Baking the simulation will make sure that every effector along with every possible related skeleton gets keyed in the skinned skeleton.
I really hope MotionBuilder can do the same, because otherwise we are going to be stuck using a single animation layer until the Maya FBX exporter gets support for exporting multiple takes.
I’ve also been working on getting the model files to update when one exports a previously existing Maya scene. The thing is that if one has spent lots of time modifying textures and shader variables, and then decides to tamper with the base mesh or possibly the entire scene, the model file will still retain the information previously supplied. This is a way to compensate for the fact that we can’t set shaders, textures and shader variables in Maya, but have to do it in an external program. When this is done, only meshes with identical names to those already existing in the model file will retain their attributes, all others have to be changed in the material editor.
Characters
I’ve been hard at work getting characters to work, and now they do, well sort of…
There seem to be two problems with characters at this present moment, the first is that Maya FBX exporter have no way of setting the skin and skeleton to bind pose before exporting (which is super important because the skeleton can be retrieved in bind pose, but the mesh can’t). If one is to not export while in bind pose, the mesh will not be relative to the skeleton, resulting an incorrect mesh deformation. The second problem is that I can’t load the animations from the FBX examples such as Zombie.fbx. The mesh works, the skeleton works, but the animation clips doesn’t get loaded for some reason. I have still to investigate this. The good news is that I actually can have an animated character! This means that the skin fragmentation, skeleton construction, animation clips (albeit only from Maya at the moment) and scene construction with a character actually works!
Now, if you found this thread because you were tearing your head off trying to understand how Maya stores joints in FBX, here is the deal. First of all, your KFbxPose is useless, seeing as you want your joints while they are in bind pose to begin with. The only thing you need is your KFbxNode for each joint, which is easily retrievable using a recursive algorithm to traverse your joints. When you got this, all you want is to get the PreRotation (using KFbxNode::GetPreRotation(KFbxNode::eSOURCE_SET) ) and current orientation using the KFbxNode::LclRotation.Get(). Your PreRotation corresponds to the Joint Orientation in Maya, and this rotation will be the basis for your joint. Now, we have two vectors, where X, Y and Z correspond to the degree of rotation around each axis. Note the use of degrees, if you want radians, this is where you want to convert it. The PreRotation (or Joint Orientation) consists of three angular values, rotation around X, Y and Z, but they are not made up of a free form transformation. To get a rotation matrix for these angles, you need to construct a rotation matrix for X, Y and Z (using the axis 1,0,0 for X, 0,1,0 for Y and 0,0,1 for Z) using the axis-angle principle, where your angle is your rotations value. Multiply these three matrices together and we get our final rotation matrix for the joint.
Example code show what I just explained (Note: n_deg2rad converts degrees to radians):
KFbxVector4 preRotation = joint->fbxNode->GetPreRotation(KFbxNode::eSOURCE_SET);
// first calculate joint orientation
matrix44 xMat = matrix44::rotationx(n_deg2rad((float)preRotation[0]));
matrix44 yMat = matrix44::rotationy(n_deg2rad((float)preRotation[1]));
matrix44 zMat = matrix44::rotationz(n_deg2rad((float)preRotation[2]));
matrix44 totalMat = matrix44::multiply(matrix44::multiply(xMat, yMat), zMat);
That is not enough however. We also want your bone rotation at the moment of binding, which we get by getting the LclRotation as previously mentioned. Then apply the same principle to those values…
KFbxVector4 bindRotation = joint->fbxNode->LclRotation.Get();
// then calculate the bind value for the bone
matrix44 bindX = matrix44::rotationx(n_deg2rad((float)bindRotation[0]));
matrix44 bindY = matrix44::rotationy(n_deg2rad((float)bindRotation[1]));
matrix44 bindZ = matrix44::rotationz(n_deg2rad((float)bindRotation[2]));
matrix44 bindMatrix = matrix44::multiply(matrix44::multiply(bindX, bindY), bindZ);
Now we have both matrices, the bone matrix and the joint matrix. In games, we don’t really care about bones, seeing as we want a united joint matrix which we can use in our skinning shader. Anyhow, when we got our matrices, we also want to combine these to get the actual joint bind pose…
// multiply them
bindMatrix = matrix44::multiply(bindMatrix, totalMat);
If you want quaternions as rotations, which is the case with Nebula, just convert the matrix to quaternion, otherwise keep it like this. This solution gives you the joints where they are NOT multiplied with their parents matrices, and this is because Nebula wants them unrelated. Nebula calculates the inverted bind pose for each joint when loading them, and simply multiplies the current joint with the parents inverted bind pose when evaluating the skeleton. Another way of solving this would otherwise be to get the KFbxPose, but the pose only gives you matrices which are premultiplied by all parents, which means the skeleton will be in world space, detached from their parents, which in turn means Nebula wont be able to multiply them with their parents matrices. So, use this method if you want to skin in realtime using the algorithm JointMatrix = JointPoseInverted * (JointPose * JointParentPose).
So the conclusion is that Nebula wants the joints in local space (not multiplied by their parents), so that Nebula can multiply the parents afterwards. The reason behind this is because the skeleton can have simultaneous animation clips running at the same time, which means the parent joints might be affected by two animations simultaneously, and thus the parent matrix cannot be pre-multiplied. This is probably the way most game engines would handle skeletons, seeing as there is minimal re-computation for maximal flexibility.
I will post a video proving that it actually works when I’ve made it work for multiple animation clips (the Zombie.fbx problem).
EDIT: I found on the FBX discussion forums that all of this can be done with a single function call, called KFbxNode::EvaulateLocalTransform… Thankfully, I’ve learned a lot about how it’s really done underneath the hood so I’m not bitter about it… Well OK, maybe a little bit…
Animations
The time I’ve had this past week as been focused on animations and characters in Nebula. There is a big difference between exporting characters in the new installment of Nebula compared to the old. The biggest feature is the fact that one can have several characters in one Maya scene, and have them exported as several individual characters! There is a pretty big difference between characters and ordinary static objects in Nebula, mainly in their model files (.n3). You see, a model file describes a scene, which is usually initiated with a transform node describing the global bounding box for the entire scene. This node then holds all meshes in the scene, with all their corresponding values for material, texture and variables. However, characters are much different! They have another parent node, called CharacterNode, which describes the character skeleton. All meshes described within the CharacterNode are counted as skins to the skeleton, which in turn means they have to be skinnable! This means that having both characters and static objects in the same scene is impossible with the current design. One might as why I don’t just add a root node which contains both a CharacterNode with all its skins, and then have all the other nodes parallel to that node. Well, you see, Nebula has to decide whether or not a MODEL is a character or a static mesh. So combining both static meshes and characters would cause big problems. This also means every single skeleton needs its very own model. Currently, the batcher decides whether or not a Maya scene should be a character in Nebula, or a static mesh. There wouldn’t be a problem if one would just take all static objects into one model, and have every character in their separate ones, except if it wasn’t for giving them a proper name! So one has to chose if they want to make an ordinary static object scene, or a character scene, so that’s that!
And of course, the biggest problem is getting the skeletons, animation curves and skinning to work properly, seeing how many variables there are that can go wrong. Currently I think I’ve managed to get the skeleton working properly, seeing as I can have a box using three joints, unanimated, and it looks correct. However, as soon as I apply an animation, it breaks. The image to the left shows how it looks after animation, and the right one before animation.
I also realized that Nebula only accepts skins which use 72 or less joints, which means that more complex models needs to be split into smaller fragments, where each fragment can use 72 or less joints. I should have this done by the end of the week unless something very time consuming turns up.
I’ve also been collaborating with my colleagues and we’ve started wrapping our programs together, mainly by designing a central class for handling settings. For example, if I set the project directory in Nody, it should be remembered by all toolkit applications so that one doesn’t need to reset it everywhere if one is to change the working directory.
Content
The past two weeks have all been centered around content, and how to get content into Nebula. We’ve been working on a format called Alembic, which provides easy-to-export plugins for Maya. We just recently realized though, that the Maya plugin doesn’t connect skins and skeletons, so there is no way of know what skin goes to what skeleton, bah! Instead, Maya animates every single vertex individually, so a 5 megabyte .mb-file becomes a 50 megabyte .abc file! Not only is it space inefficient, it’s also an enormous setback in performance when animating. Anyhow, we decided to revert back to the FBX, because we realized it would be much easier to dig into FBX (mainly because there has been some work done by a couple of my old class mates). They made an exporter which would allowed you to make a character in Maya, export it to FBX, and then use it in Nebula. And while that is rather nice, the application was quite difficult to expand on.
So we made a new one! It basically uses all the same things at the moment, excepts in a way more modular way. For example, if one has several meshes in one Maya scene, the FBX batcher will export every single mesh node to its own file, and if there is only one mesh, it will create a file with the same name as the counterpart. Also, the batcher will create a model (basically scene graph fragments) which holds every single mesh in your Maya scene as a separate mesh, thus allowing you to modify them by attaching different materials, variables etc. The only thing that is left to do with it is to allow parenting of objects. Seeing as Nebula models already handles parenting by having a node hierarchy, one could just as easily make sure the meshes come in the exact same hierarchy in the model as they do in Maya. The only problem with this is that there might be some unexpected behavior when animating. This remains to be seen, however I fear that parenting with characters will be a feature to add in the very close future.
Right, content, where was I. Yes, meshes, ok, we need meshes, and we have meshes. Although it wasn’t trouble free. Nebula has a set of tools which greatly help with mesh importing, the MeshBuilder, MeshBuilderVertex and MeshBuilderTriangle. These three classes is all you need to represent a mesh. Getting data from FBX was also trivial, but getting data into the MeshBuilderwasn’t. Since a vertex can have several UV-coordinates, and the FBX-models are compressed in such a manner so that they have an index list and a data list for very piece of vertex information, it would require extensive parsing just to know how many vertices one would need! Instead, the MeshBuilder has a function called inflate, which makes sure every triangle has its own unique set of vertices. Thus, one can traverse the index lists and set the data with ease, and then just remove the redundant vertices right? WRONG! Retrieving bitangents and tangents from FBX resulted in every single vertex being unique, which in turn resulted in the MeshBuilder removing 4 vertices when cleaning up. The removal resulted in a destruction of the mesh, and this was of course not acceptable. So instead we decided to get the UVs and Normals, which are only unique for some vertices, and then deflate (remove redundancies) and calculate the bitangents and tangents ourselves. No more exploding meshes = mission accomplished.
This wasn’t all however. It turns out Nebula saves meshes in a compressed format, where positions are stored raw, normals bitangents and tangents are saved as single ints, and the texture coordinates are saved as one int. They use a packing technique where the RG and BA components of the vector describes where in tangent space the normal is pointing, which in turn gives you a decent precision. Texture coordinates are saved as two 16 bit unsigned shorts, resulting in every piece of texture coordinate is only the size of an int. Texture coordinates needs to have more precision than normals, but certainly doesn’t need to be 32 bit per component. Right, I thought you might want to see some proof, so I grabbed a picture for you.
This shows two models made in Maya and exported using the FBX batcher. The artifacts you might see on the inside of the “sphere” is a glitch caused by the SSAO shader. The character is dressed with the eagle texture, so that explains her being transparent in some areas.
It’s alive!
So I’ve been working on Nody, trying to get it to be as flexible and as intuitive as possible. Thus far, I can modify shaders (which in turn actually rewrites the shader code) and have them presented to me in real-time, which is pretty neat. What I wanted to do when that worked was to be able to set variables such as textures, directly from Nody, so that one can preview how a shader would look on a specific model with a specific set of variables. That’s what got me into the texture compression part (see last post).
Nody will not serve as a texturing tool, seeing as it’s purpose is to create shaders, and that’s it. The reason for this is because Nody is supposed to work on a per-shader level. Nebula uses 3 different levels, resource, template and instance. The first, resources, is the most general of these three, it can be a mesh, a shader, a texture, a sound file, or any other type of resource one might need. The second level describes for example a model, which is a collection of resources, and resource states, such as shader variables, texture attachments, sound attachments, animations, skeletons etc. The third level is instance, which is what is you actually use in your game. Nody is a level 1 tool, meaning its purpose is to handle a resource, in this case a shader. A colleague of mine is currently working on a level 2 tool which will be a part of the level editor. This tool is called the material editor, and it allows a user to switch materials (not the shaders within the material), textures, variables etc. It’s basically meant to change the model-file, which is used as a template for all instances. On the instance level, very little is changed in resource-manners. One might want to change a certain variable, but that is pretty much as far as you go. One might want to be able to have variations without really changing the model file, and that is fine, as long as one keeps track of the variable name and sets it correctly.
When picking textures in Nody, Nody will present your working directory, where you have your image files in raw format. Whenever a texture is picked, Nody will look for the presence of an exported version, and if it doesn’t exist, calls the texture batcher to export it.
Although, one might want to be able test out their shader to every extent before deciding it’s exactly what they want, and that means variables and textures has to be testable online. With online I mean without having to restart either application. I haven’t really had the time to make a video yet, but I’m working on getting one out, so you can see how powerful this tool is. It’s really cool too