Making of Varga by Paul Tosca, Romania Web: www.paultosca.com

4.4 Further Tweaking of Normal maps in Photoshop

This paragraph is for those of you who want to better understand normal maps and it may seem a little technical (it requires some basic knowledge of math, vectors and stuff like that ).

4.4.1 Some general info TS (tangent space) normal maps

First, I would like to talk a little about normal maps , how they are computed/displayed. I am more interested in tangent space normal maps ( world or object space normal maps are not suitable for characters or for objects that will deform) so for now on when i will say normal map you will assume i am talking about tangent space unless stated otherwise.

If you are in the same situation i was when i first started with normal maps (tangent space in particular) probably the following info will help. At the beginning i was really frustrated by the "Seam" problem :) : a normal map computed within a software will not display properly in another one , and i am talking here about the seam that will appear in the geometry where you will have an uv shell border.I have struggled to find out what the problem was ... thought i am doing something wrong , read the help of each apps then started searching on the internet and after some research i finally got the idea about the problem , how and why it will occur ...but not how to fix it though:)

The main problem resides in the way each application computes/displays tangent space normal maps.

Tangent space is represented by three vectors: normal (that will always be perpendicular to the surface ) ,and two other vectors perpendicular to the normal called tangent and bi-normal (or bi-tangent : i have seen that the names are used interchangeably though some say that the correct name will be bitangent since we are talking about a surface ... for a curve in 3D , the tangent space we will have one tangent and two normals , normal and binormal .... for surfaces we will have one normal and two tangent vectors : tangent and bitangent but the naming does not matter much as long as we know what vectors we are talking about). These three vectors can be defined for each point on a surface and taken all together they define a coordinate frame.

Usually tangent space per triangle will be computed in the following way:

- the normal vector will always be perpendicular to the surface (triangle) (it will be the blue one and it will be stored in the blue channel of the normal map)

- the tangent(the red one stored in red channel) and binormal(the green one stored in green channel) vectors will be oriented in accordance with the UVs :tangent will be left to right(or right to left) in UV space (so it will be U coordinate in texture space ) and binormal will be down to up (or up-down) (so it will be the V coord in texture space) ;probably you have seen in many dialogs for normal maps the options to flip Red or Green channel , these options stand for the orientations of the tangent(left-right/right-left) and binormal (up-down/down-up) .. some apps use different orientation than others; you can flip the channels inside photoshop also, for instance to flip the Red channel you will press Ctrl+1 ( to go to the red channel ) and Ctrl+I (to invert it) then Ctrl+~ (control tilda , that will be left to 1 key -) to go back to full RGB image (same for green Ctrl+2,Ctrl+I,Ctrl+~).

Now lets say that i will cut the UVs along the edge and will rotate one shell 90 degrees CCW like in the image below. Because of the rotation in UV space the tangent space for the first triangle will be different than in the first situation (the RGB triad for the first triangle has rotated also 90 degrees along blue axis)

Because of this , the orientation of the uv shells will affect the colors you will see in the normal map;if you rotate the uvs of your model ( or the uvs for some shells ) lets say 90 degrees ClockWise and recompute the normal map the colors in the new normal map will be completely different ( not because the entire image will be rotated by 90 degrees ... you can rotate it 90 CCW in photoshop and compare it with the first version to check .... they will be different )

Tangent space per vertex will be then computed and after that you can compute the tangent space vectors for any arbitrary point on the surface by interpolating vertex tangent space vectors; usually more complex calculations are done so that the tangent space will be invariable regarding to tesselation (L shape problem) : for instance in previous versions of maya the computed normal map was dependent to tesselation ... so if you would have a triangulated model and if you would select one triangle edge and flip it then the normal map would no longer be displayed properly due to recalculation of TS after flipping the edge ... in the newer versions of maya this was solved ( usually special tools are used to prepare the geometry for per-pixel lighting by creating proper tangent basis at each vertex, NVMesh Mender is one of them)

Now ,for computing a normal map the raytracing algorithm will cast a ray from the current point of the low poly mesh to the hign poly and at the point of intersection with the high will compute the normal for the high surface in world space (WS) then this normal is transformed in the tangent space (TS) of the low poly version and stored in the normal map.So you see that the normal computed is dependent to the tangent space used for the low poly version and the tangent space normal map must be created using the same normal/tangent/binormal as the game uses (or the application you will use to display the normal maps) ,otherwise the normals will be misinterpreted and you will see discontinuities on border uv shells , hence the problems you will have when trying to display a normal map computed with one app in another app.

It will not be a problem if different apps will use different methods for tangent space vector calculations , the problem is that they are not making that info available so that someone interested could write a plugin/tool that will remedy the problem ...the only app i know so far to made the formulae public is Mudbox , on their online help pages you can find how they are calculating tangent space vectors.

Now you know why the problem occur but there is little to do about it ... at least you should know that you aren't doing anything wrong , the app is to blame :) .

I will show an example and explain how the normal map will be computed and how to interpret it channel by channel... this will help if you want to alter the normal maps in photoshop , overlay multiple normal maps , paint them directly in photoshop etc ...

I will bake the normals from the plane and cylinder below to a single quad plane highlighted in green.As i explained earlier the tangent space will be : normal pointing up( blue one) tangent from left to right ( 3-4 red one) and bitangent/binormal down-up (3-1 the green one) . The cylinder has all the edges hard so the normals will change in steps, it will be easier to analyze.I have numbered the faces on the cylinder from 1 to 7.

Now lets suppose we are looking at the objects directly from front , it will look something like in the image below.

At the top of the image you have a color scheme: high poly is white,the normals for high poly are cyan,low poly is magenta, the normals for low poly are yellow (since we have a quad plane the normals will have the same orientation everywhere on the surface),

Below the color codes you see the TS (tangent space) triad: tangent (red left to right ), normal (blue down to up) and binormal (green and perpendicular to the screen pointing from you to the screen); notice in the figure above that all the faces from the high version are parallel with the green arrow (binormal in tangent space) ,excluding the triangles that will cap the cylinder and that will not be visible in the normal map anyway.

Now ,for every face in the high poly I can decompose the normal vectors (cyan colorcoded) in TS in their tangent (red colorcoded) and normal (blue colorcoded) components ,the binormal component will be zero for all of them since the faces are parallel with the binormal vector = green arrow hence the flat color in the green channel of the normal map.

The green interrupted guides were drawn to help visualize better the transitions between faces and to see how they correspond with the actual geometry.

The gray arrows pointing from low poly to high are the searching rays used during the computation of normal map.Now lets consider some points from the low poly (A,B,C,D,E,F,G) and see what actually happens.

As a side note , usually vectors are normalized (with unit length) and if you decompose a vector into TS components (or other system coordinate )each component will have a real value between -1 and 1 however this will be converted to positive integer and stored into RGB channels of the normal map as follows:

- for tangent ( in the red channel ) negative values (-1,0) will be mapped linear to 0- 127 ,0 will be 128 (neutral) and positive values (0,1) will be mapped linear to 129-255

- same will apply for binormal in the green channel

- for the normal vector the values can be only positive (0,1) so they will be mapped to 128-255;most of the time the normal of the high will not deviate a lot in comparison with the normal vector in TS for the low poly hence the blue look of normal maps in TS ( the blue channel will be mostly white);because of this TS normal maps might be compressed since it uses fewer colors than OS (object space)/WS (world space) normal maps,also since TS vectors can't point backward the normal component will be always positive and when normalized it can be coded using only the tangent and binormal (the normal will be computed based on the other two)

For point D we shoot a ray and when intersecting the high version the normal will be that of face number 4 that will be parallel with the normal TS vector and following the dotted lines you can see that the red channel will be 128 since the tangent component will be zero, and the normal component will be 1 mapped to 255 (full white) in blue channel.

For point F , the searching ray will intersect face number 6 and we can see that the normal vector of this face (cyanish colorcoded) will be decomposed into two components : tangent component (red)with a positive value around 0.78 mapped to 228 in the red channel and binormal component (blue) with a positive value of 0.625 mapped to 208 in the blue channel.

For point B , the searching ray will intersect face number 2 and we can see that the normal vector of this face will be decomposed into tangent component (red)with a negative value around -0.78 mapped to something like 22 in the red channel and binormal component (blue) with a positive value of 0.625 mapped to 208 in the blue channel.