BFP's Forging Notes

Is software engineering compatible with making games?

2012-01-17T14:36:00.005+01:00

This is one of the hundred of questions that I ask myself many times.

After some mumbling and experience, I came to the understanding that what's really misleading is the equality software engineering == O.O.P. == design patterns.

This is what it appears when reading forums and speaking with peoples around.

That's wrong. Completely wrong.

First of all: software engineering is NOT Object Oriented Programming.

Software engineering is a tool to find solutions to problems.
Object Oriented Programming is one way of solving the problem, but it is another tool.

Second: Object Oriented Programming is NOT design patterns.

The real trouble here is that people is lazy and many times not focused / precise enough to describe the problem.

A PATTERN is a SOLUTION to a known PROBLEM.

So, what's a problem?

The word problem cames from greek, and means "What cames BEFORE a project".

The real error here is the definition of the problem.

DEFINING A BAD PROBLEM WILL LEAD TO BAD SOLUTIONS.

That can be sound silly or foggy, but really is a common error I see around (and also in myself).

So going back to the equality, patterns are solutions to problem.
Our mind, that always needs to categorize and label stuff to organize data, works with associations: when we find something similar to what we already know, under a certain percentage of similarity, we assume that is equal.

This is normal and common: you know the experiment that if we read words with only the initial

and fnail wrdos we can understand them the same.

We already (normally) encounter that word, so we can understand which one is.

Using patterns is not the only way of using OOP: they are completely different domains.

Software engineering is an analysis and design mindset.

What we're really missing is to DESCRIBE CORRECTLY PROBLEMS.

Also, we have not so many patterns that are usable for the gaming industry.

That's for two reasons.

1) There are not so many software engineers/architect that have enough time to study and design a so large software like a game and an engine, with the production constraints we have.

2) Believing in this inequality, a lot of peoples believes that software engineering is not for realtime applications.

But that's hope.

Even in a strange way, some new patterns/way of programming is coming out.

Data Oriented Programming they say.

There are no explicit patterns, written in an academic way, but there are patterns.

In that case, what happened is describing the problem in a correct way.

For example, in a lot of patterns there is no mention of hardware constraints or performances.

Data oriented programming thus is becoming a new way of coding: coding thinking about data and performances.

But data oriented does not exclude object oriented.

They can co-exist!

We must be open-minded: still we need to go one level up, thus being independent of Object or Data oriented paradigms.

We must focus on each problems, and why they're there.

Object oriented programming is born to solve the problem of creating high-maintainable code used by different persons during a long time-span.

Data oriented programming is born to solve the performances issues and platform bindings that

object oriented was leaving behind.

Each problem its own solution.

Need 1000 calls per frame on consoles? Data oriented, cache friendly, multithreaded and no virtuals are the keyword.

Need 1-10 calls per frame? Object oriented is good.

Looking from this angle, the two paradigm CAN and MUST co-exist, because they're not strict and they're really a way of dealing a certain class of problems.

Position reconstruction from depth (3)

2012-01-09T18:34:00.008+01:00

Hi guys,

just wanted to share the code I've used to study the position reconstruction problem.

This code lets you switch easly between linear depth and post-projection depth, and I'll show also a way to check if the reconstruction is correct.

Following is the code to convert between Camera/View space to PostProjection space and viceversa.



// This must be done on the CPU and passed to shaders:
float2 getProjParams()
{
 //#define PROJ_STANDARD
 #ifdef PROJ_STANDARD
 float rangeInv = 1 / (gFar - gNear);
 float A = -(gFar + gNear) * rangeInv;
 float B = -2 * gFar * gNear * rangeInv;
 #else // We get rid of the minus by just inverting the denominator (faster):
 float rangeInv = 1 / (gNear - gFar);
 float A = (gFar + gNear) * rangeInv;
 float B = 2 * gFar * gNear * rangeInv;
 #endif

 return float2(A, B);
}

// Input: 0..-far - Output: -1..1
float postDepthFromViewDepth( float depthVS )
{
 float2 projParams = getProjParams();

   // Zn = (A * Ze + B) / -Ze
   // Zn = -A - (B/Ze)
   float depthPS = -projParams.x - (projParams.y / depthVS);

 return depthPS;
}

// Input: -1..1 - Output: 0..-far
float viewDepthFromPostDepth( float depthPS )
{
float2 projParams = getProjParams();

   // Ze = -B / (Zn + A)
 float depthVS = -projParams.y / (projParams.x + depthPS);
 
 return depthVS;
}

Next I'll show some helper functions to encode/decode the depth in different spaces.


///////////////////////////////////////////////////
// POST PROJECTION SPACE
///////////////////////////////////////////////////

// Returns post-projection depth
float decodeProjDepth( float2 uv )
{
 return tex2D( depthMap, uv ).r;
}

// Returns viewspace depth from projection (negative for left-handed)
float decodeViewDepthFromProjection( float2 uv )
{
 float depthPS = decodeProjDepth( uv );
 return viewDepthFromPostDepth( depthPS );;
}

// Returns depth in range 0..1
float decodeLinearDepthFromProjection( float2 uv )
{
   float depthVS = decodeViewDepthFromProjection( uv );
   // Left handed coords needs the minus,
   // because the depth is negative
   // and we are converting towards 0..1 domain
   return -depthVS / gFar;
}

// Returns post-projection depth
float encodePostProjectionDepth( float depthViewSpace )
{
 return postDepthFromViewDepth( depthViewSpace );
}

///////////////////////////////////////////////////
// VIEW/CAMERA SPACE
///////////////////////////////////////////////////

// Returns stored linear depth (0..1)
float decodeLinearDepthRaw( float2 uv )
{ 
 return tex2D( depthMap, uv ).r;
}

// Returns viewspace depth (0..-far)
float decodeViewDepthFromLinear( float2 uv )
{
 return decodeLinearDepthRaw( uv ) * -gFar;
}

// Returns linear depth from left-handed viewspace depth (0..-far)
float encodeDepthLinear( float depthViewSpace )
{
 return -depthViewSpace / gFar;
}

With simple defines we can control and switch between using a linear depth or a post-projection (raw) depth buffer to check that our calculations are fine:


#ifdef DEPTH_LINEAR
#define encodeDepth encodeDepthLinear
#define decodeViewDepth decodeViewDepthFromLinear
#define decodeLinearDepth decodeLinearDepthRaw
#else
#define encodeDepth encodePostProjectionDepth
#define decodeViewDepth decodeViewDepthFromProjection
#define decodeLinearDepth decodeLinearDepthFromProjection
#endif

Now all the reconstruction methods, both the slow and the one that uses rays:



// View-space position
float3 getPositionVS( float2 uv )
{
   float depthVS = decodeLinearDepth(uv);
 
 //float4 positionPS = float4((uv.x-0.5) * 2, (0.5-uv.y) * 2, 1, 1);
 float4 positionPS = float4( (uv - 0.5) * float2(2, -2), 1, 1 );
 float4 ray = mul( gProjI, positionPS );
   ray.xyz /= ray.w;
 return ray.xyz * depthVS * gFar;
}

float3 getPositionVS( float2 uv, float3 ray )
{
 float depthLin = decodeLinearDepth(uv);

 return ray.xyz * depthLin;
}

float3 getPositionWS( float2 uv )
{
 float3 positionVS = getPositionVS( uv );
 float4 positionWS = mul( gViewI, float4(positionVS, 1) );
 return positionWS.xyz;
}

float3 getPositionWS( float2 uv, float3 viewDirectionWS )
{
 float depthVS = decodeViewDepth(uv);
#if defined(METHOD1)
   // Super-slow method ( 2 matrix-matrix mul )
   float4 pps = mul( gProj, float4(getPositionVS( uv ), 1) );
 float4 positionWS = mul( gViewProjI, pps );
 positionWS /= positionWS.w;
 
 return positionWS.xyz;
#elif defined(METHOD2)
 
   // Known working slow method
 float3 positionWS = getPositionWS( uv );
#else return positionWS .xyz;
 
   // Super fast method
 viewDirectionWS = normalize(viewDirectionWS.xyz);

   float3 zAxis = gView._31_32_33;
   float zScale = dot( zAxis, viewDirectionWS );
   float3 positionWS = getCameraPosition() + viewDirectionWS * depthVS / zScale;

 return positionWS;
   #endif // METHOD1,2,3
}

Last but not least, I'll show you the code to encode the depth and perform a simple calculation to understand if we've done right:


// This is only with an educational purpose,
// so that we can switch towards storing
// viewspace or postprojection depth.
// In the GBuffer creation, the depth stored like this:

depth = float4( encodeDepth(IN.positionViewSpace.z), 1, 1, 1);

// A very cheap and easy way to detect if
// we've worked correctly is to add a
// point-light and light a bit more
// the rendering with something like:

#ifdef POSITION_RECONSTRUCTION_VIEWSPACE
 float4 lightPos = mul( gView, float4(100.0, 0.0, 0.0, 1.0) );
 float3 pixelPos = getPositionVS( uv, viewDirVS );
 #else
 float4 lightPos = float4(100.0, 0.0, 0.0, 1.0);
 float3 pixelPos = getPositionWS( uv, viewDirWS );
   #endif

Note the different rays, viewDirVS and viewDirWS. They are calculated as MJP showed a lot of time and two different ways, one for meshes and the other for fullscreen quads.

I think that's all for now, I'll attach a screenshot of the simple test I've used to test the reconstruciton. Note that the light is the same in all the conditions, view/world space reconstruction, linear/postprojection depth storage.

Enjoy!!!

Position reconstruction from depth (2)

2012-01-09T17:47:00.008+01:00

Happy new year guys!
I want to finish my summary for position reconstruction.
The missing part is the position, that can be in ViewSpace (or CameraSpace) and in WorldSpace.

So far we know the relationship between Post-Perspective and ViewSpace depth of the fragment. If not, roll back the previous post.

The methods are taken directly from the amazing MJP, kudos to his works!
My target is again left-handed coordinate system like OpenGL, that needs some more attention
(really a minus sign in the correct place make a HUGE difference!).

The methods I'm using are the one that pass down a ray to the fragment shader.

VIEWSPACE POSITION

Ray Generation
For a fullscreen quad we take the ClipSpace position and multiply by the inverse of the projection matrix to obtain a ray in ViewSpace.

float4 ray = mul( gProjectionInverse, positionCS );
ray /= ray.w;
OUT.viewRayVS = ray;

Position reconstruction
Here we need the LinearDepth between 0 and 1, that depends upon your choise of storage.
The formula is:

viewRayVS * linearDepth

If you are reading from the depth-buffer, then you'll need to convert it to view-space and then
divide by the far value.
Pay attention here. If you are using a left handed coordinate system, your ViewSpace depth will be ALWAYS negative.
So the left-handed passage will be: rawDepth -> viewSpaceDepth(0..far) -> division by -far
Right handed: rawDepth -> viewSpaceDepth(0..far) -> division by far

WORLDSPACE POSITION

Ray generation
The generation here is:

ViewDirectionWS = positionWS - cameraPositionWS
and the camera position in WorldSpace can be found as the last column of the inverse of the view matrix.
In HLSL if you want to access this, you can create an helper method:

float3 getCameraPosition()
{
return gViewInverse._14_24_34;
}

Position reconstruction
In this case the reconstruction is longer.
Around the web you can find the solution like get the viewspace position and then multiply by the inverse of the view matrix.
It is ok, but can be a lot faster.
Matt in his blog suggest the solution (here) to scale the depth on the camera zAxis:

float3 viewRayWS = normalise( IN.ViewRayWS );
float3 zAxis = gView._31_32_33;
float zScale = dot( zAxis, viewRayWS );
float3 positionWS = cameraPositionWS + viewRayWS * depthVS / zScale;

here we're talking about the real viewSpace depth, that can be converted from the post-projection depth by using:
float depthVS = ProjectionB / (depthCS - ProjectionA);
This is already the depth in view space (that is always negative for left-handed systems).

When storing the linear depth between 0..1, we'll need to convert it back to viewspace.
In case of left-handed system, we stored the linear depth like that:

float depthLinear = depthViewSpace / -Far;

So to have again the depth in ViewSpace, we have:

float depthViewSpace = depthLinear * -Far;

Hope this is useful guys.
Credits goes to MJP for his amazing work, and this is just a way to summarize all the possible problems in reconstruction.
The problem I found around the web is in defining the real DOMAIN of variable and spaces, and I hope that this contributes to have more clearer ideas about how to handle depth reconstruction and space changes without fear.

In the next post I will simply write down all the methods in the shader I used to test all this stuff!

Position reconstruction from depth (1)

2011-11-28T14:38:00.011+01:00

Hello gents,
this post is just a quick recap about the possible ways to reconstruct position from the depth buffer that I found around: almost all the credits goes to http://mynameismjp.wordpress.com/.

Let's define (again) the problem:

Reconstruct pixel position from the depth buffer.

Applying a personal way of seeing code, let's put in evidence Data and Transformations.
In my experience, I came up with a simplicistic idea about coding:

Coding is a sequence of Data transformed into other Data.

I know it is very simplicistic, and low level (we're not taking in account any architecture) but this is a low-level view of the problem.
And more views of the same problem can shed more light on the true nature of the problem itself (as in life in general).

In this problem we have two data: Pixel Position and Depth buffer.
The transformation is reconstruction.

To understand further, we can define the Domains of the datas.

Pixel position can be either in World Space or in View Space
Depth buffer can be encoded either in linear or in post-perpsective z (raw depth buffer)

So either the transformations will be from Depth buffer to Pixel position and can be:

Linear depth buffer to View Space position
Linear depth buffer to World Space position
Post-Perspective depth buffer to View Space position
Post-Perspective depth buffer to World Space position

To finish, we have two other transformations:

Encode to Linear depth buffer
Encode to Post-Perspective depth buffer

the post-perspective is hidden by the hardware, and it is what it's inside the real depth buffer.

The linear one maps the eye/camera/view space z to the domain 0..1.

To really finish this introduction to the problem, we must know a little bit about our coordinate system. Moving data from world to view to projection spaces, we must define those domains.

We can just skip World space and concentrate on the other.

If we follow OpenGL or DirectX APIs, we know that they are different in both spaces:

OpenGL uses a right-handed system for the view space
DirectX uses a left-handed one
OpenGL uses a cube between (-1, 1) on x,y,z as projection cube
DirectX uses a cube between (-1,1) on x,y and (0, 1) on z

Using a right-hand system ends up looking at negative z. Keep this in mind.

ENCODING AND DECODING TRANSFORMATIONS

In this section we'll talk about encoding: what we want to encode?

The raw-depth-buffer contains a depth transformed from the view-space depth, and they are encoded in a simple way, depending on your projection matrix.

Let's take only the relevant part of the matrix (the last 2x2 corner), that is:

( A B ( zView
-1 0 ) 1 )

and multiply it with the point in viewspace Pview(zView, 1).
Doing the multiplication has the result:

Pndc = (A * zView + B, -zView )

to became a 3d point (1d here) we apply the division by W:

Pndc = ( A * zView + B / -zView, 1 ) that further simplified became

Pndc = ( -A - (B / zView), 1).

Zndc so is -A - (B /zView).

This is the way in which the depth is encoded in the depth buffer, and the value is between -1 and 1.
Note: if you try to do some maths and put zView = n and zView = f, you'll notice that the values are not mapped correctly between -1 and 1. This is because we're using negative values, so the correct ones are zView = -n and zView = -f.

To find zView, just solve by zView and we'll obtain:

zView = -B / (zNdc + A )

So now we have defined the two transformations:

Projection-Space Encoding: -A - (B / zView )
Projection-Space Decoding: -B / (zNdc + A)

Ok then, it's finished.

Wait...what are thos A and B???

Those values depends again on the choice of your projection matrix.

In OpenGL they are defined as (n is near plane, f is far plane):

A = - (f + n) / (f - n)
B = -2 * n * f / (f - n)

those values can be easly calculated and passed to the shaders (don't bother doing it inside a shader, those are perfect values to be set once in a frame with other frame-constants) to reconstruct depth.

Different is the linear depth encoding. We're still encoding view-space depth, but that became easier. The values in camera/eye/view space are like world-space, but just centered around the camera. For the right-handed systems, we will encode all the negative z, because the camera is looking into the negative z semi-space.
The z values will be in the range 0, -infinite: the projection will take care of getting rid of values that are smaller than the near plane and greater than the far.

Linear depth encoding: -zView / f
Linear depth decoding: zLin * f

Those values are between 0 and 1.

Finally...some CODE!!!

This is POST-PROJECTION DEPTH:

// Calculate A and B
float rangeInv = 1 / (gFar - gNear);
float A = -(gFar + gNear) * rangeInv;
float B = -2 * gFar * gNear * rangeInv;

// Write -1,1 post-projection z
float encodePostProjectionDepth( float depthViewSpace )
{
 float depthCS = -projParams.x - (projParams.y / depthViewSpace);
 return depthCS;
}
// Read -1,1 post-projection z
float decodePostProjectionDepth( float2 uv )
{
 float depthPPS = tex2D( depthMap, uv ).r;
       return depthPPS;
}
// Reconstruct view-space depth (0..far)
float decodeViewSpaceDepth( float2 uv )
{
       float depthPPS = decodePostProjectionDepth( uv );
 float depthVS = -B / (A + depthPPS);
       return depthVS;
}

This is Linear depth

// Encode 0..1 view-space depth
float encodeLinearDepth( float depthViewSpace )
{
 return -depthViewSpace / far;
}
// Decode 0..1 view-space depth
float decodeLinearDepth( float2 uv )
{
 float linearDepth = tex2D( depthMap, uv ).r;
 return linearDepth;
}
// Reconstruct view-space depth (0..far)
float decodeViewSpaceDepth( float2 uv )
{
 float linearDepth = decodeLinearDepth( uv );
 return linearDepth * f;
}

As you can see using a linear depth is easier to encode and decode, but it's more expensive from a memory point of view (you'll need an additional render target), and you're already using a depth buffer so you already have those informations.

Next stop is a service post about reconstruction methods for position, even though they are explained a lot by Matt Pettineo on his blog!

P.S. Fixed a typo in the postProjectionDepth encoding. Fixed a typo in the A and B calculations.

Rendering Architecture (2)

2011-11-23T12:59:00.003+01:00

Hi guys,

a simple follow up about the very low-level architecture I'm using in the latest months, after having a look at DirectX 11.

In the context of rendering, we can assume that every time we render we know exactly which type of renderer (DX9, DX11, LibGCM, X360, ... ) we are using: we don't want to switch renderer on the fly, and on consoles this is impossible to do.

So starting with this in mind, we know and want that in the executable we will build there will be only one renderer.

This can be achieved with a dll/lib in a different projects as you prefer, but the bit I want to talk about is the RenderInterface.

First of all, what is a RenderInterface?

We have another context parameter: we know that during the rendering phase of our game, we need to provide three main informations to the graphics card and they are

Geometry informations
Shading informations
Render states

and we set those informations in various way, for example on DX/X360 we use Set*** command and Draw*** to issue the drawcall.

The geometry informations are relative to vertex buffer, vertex format/declaration and index buffer; the shading ones are the various shaders (depending on the API, vertex, fragment, geometry,...) and the informations to be used by the shaders (constants and textures); the render states are the all the other informations, like render targets, depth/stencil, alpha blending, so all the configurable states that are grouped in directx 10 and 11.

The render interface thus is splitted in two: a RenderContext, that sets all the informations to issue drawcalls only and draws, and the RenderDevice that manage the creation, destruction and mapping/unmapping of low level graphic resources.

This division permit to easly divide what is "deferrable" to what not, so if you want to create your own command buffer or use the DX11 one (good luck) than you already know that the RenderContext is the right guy to call.

Every object that can be renderable will have a render method that will take pass the RenderContext around, so that it can set the data for the draw calls.

The real catch is to use the curiously recursive template pattern to create the interface for both the RenderContext and the RenderDevice, and create the different implementations for each platforms: even though you need to typedef the specific template implementation, you can assume (see above) that for each target you have only ONE type for the RenderContext implementation (RenderContext) alive and thus you can use it.

The methods called in the API-dependent class can be all protected so that you enforce the interface, and inlining all the calls in the RenderContext class will map a call of your render context to a direct call of the method, thus avoiding virtuals and with "static polymorphism".

Even though on PC is not a cost, on consoles ( I really suggest you to try, if you can ) is a bad hit (especially on ps3) to call virtual functions a LOT of times, but let's try to figure out the numbers:

if you have 1000 draw-calls, probably you'll have 4 or 5 RenderContext calls (SetVertexBuffer, SetIndexBuffer, SetVertexShader,SetPixelShader,SetConstants,SetVertexFormats, DrawIndexed, ...) for each draw-call, thus having 4000-5000 virtual calls for each frame. So you end up having 4000-5000 cache misses per frame and all without any apparent reason, and the cost of cache misses on consoles...is varying, but can be from 40 to 600 cycles for each call.

How many cycles are we wasting?

With this system, you have a common interface and no virtuals. No silver bullet, but the problem to find a solution requires a correct definition of the constraints...

BFP

Rendering architecture (1)

2011-10-31T09:59:00.011+01:00

Hi guys!
Too much time I don't update this blog!

I wanted to share with you a way in which I describe rendering for my home-engine, used to study technologies quickly.
I'm doing this way since 2008, so now it's starting to become very mature, but I'm sure will improve in the future!

Let's start from the objective: I wanted to create something flexible and quick to iterate to study all the different rendering techniques that I'm interested into.
Speaking about DX9+ GPUs, the context of the problem has some constraints.
The rendering APIs are a state machine, and to draw we must provide at least those informations:

1) Vertex Buffer
2) Index Buffer
3) Vertex Format
4) Vertex Program
5) Fragment Program
6) Shader Constants
7) Render States
8) Render Targets

We can see the Rendering as a problem on HOW to create those information to be sent to the GPU.
The Rendering nowadays can be seen as a sequence of drawcalls applied to a render target.
So back in time I started to define the concept of Rendering Pipeline as a sequence of drawcalls in a render target.
Each of those sequences became a Stage of the pipeline.
At the end of the day we have

Pipeline
Stage0 -> Stage1 -> Stage2

This sounds a little simplified, and actually it is simple, but it's really powerful.
An example of Stage can be as simple as a ForwardRendering that renders in the framebuffer (that can be seen as a Render Target),
a GBuffer generation, Shadow Creation, SSAO, ...

For each stage we define some RenderTargetInputs and Outputs, to link the different stages together.

If I made the picture clear, each rendering became a series of rendering of stages.
To describe the rendering I started using an XML file like that:

Each stage type corresponds to a class that extends the Stage class in the code.

At the very beginning this approach was good, because I could describe rendering in a mix of

xml/code that was very fast to prototype with!

After some months of coding I saw emerging some interesting patterns:

Each stage has some RenderStates set before rendering
Each stage can draw per material or providing a shader
Each stage that describes a postprocess, draws a single quad and has a shader associated.

Further defining the rendering problem, I found another missing information in the rendering objects that were sent to each stage.

How can I define them?

I ended up with what I called a RenderView, that is a camera plus a list of "render instances". So now each stage renders a particular RenderView and has some renderstates defined, plus an optional shader if it is a postprocess.

The point is...why just not describe all those informations in the XML?

Enter ScriptableStage!

This stage has a renderview, some renderstates, and an optional shader. All defined in the XML. This provides me the flexibility to describe a lot of the rendering techniques around.

After some more technique explorations, I found another emerging pattern:

Each time I want to render, I need to provide 3 groups of informations

Geometry
Shading
Render states

When you define all those informations, you can describe all the rendering you want! Even materials can be seen as a list of Shaders corresponding to stages and can be described through XMLs...

The application logic that sits on top must really provide mostly the geometry and link it with materials (that are shaders + render states).

With this simple description I found a very powerful way of describing the rendering and prototype/explore very fast.

Next post I'll describe also a way to achieve a simple reloading mechanism and avoid virtuals for the renderer itself to be more console friendly ;)

Enjoy!

What about your challenge?

2010-05-08T15:22:00.003+02:00

I know I know...I ended up with only 1 day of programming.
But some days after, I ended doing an interview @ Codemasters.
And the big news is...NOW I WORK AT CODEMASTERS!!!
A big leap into the possibility to grew and study techniques, doing a work that I LOVE with a great company!

But hey...and your challenge?
My challenge is returnig back, even if I decided to restart again from scratch.
Why that?
In the meantime I've read some books around (mainly c++ stuff...) and ended up with some new ideas.
Also, my will is to create a framework to explore all these papers and thesis around the web that take my attention!

But all of this will not happen now, first I have to enter in the apartment and get all the stuff out of the boxes!

Stay tuned for more infos!

Progress report: Day 1

2010-03-19T12:34:00.006+01:00

Yesterday it was my first game-engine development day.
As you know, I'm facing the challange of creating a small game-engine in 10 days.
This can be considered a game to improve speed and understanding of a game engine and the decisions about the design. I found that personally, when coding at home (without time constraint) I need to study too much time before taking a decision on design. After this decision is taken, and some UML written down, I code with all in my mind (and papers) in a good way.
To improve the decision time I choose the way of training myself (as with guitar, breakdancing, motorbikes...).
What is the first principle of training? Practice, practice and PRACTICE.
Reading some books of NLP and Self-Improvement, I found in Anthony Robbins a perfect description of what he call the 'decision muscle':

Decision -> Action -> Results and Feedback.
It is the SAME as a controller in a dynamic system (automation systems...), and applies to all the field of the life.
And decision making is crucial also in engine design.
To improve decision making, I found really useful four things:

Learn more. Learn all. Many times knowing different techniques improves you understanding of other problems: any techniques, and desing has a mentality behind.
Learning the MENTALITY is powerful. Then apply to other fields. Eg: data oriented structure of arrays used in vectorized math and shaders, can be successufully applied to game objects (Pitfalls_of_Object_Oriented_Programming paper is a good example).
Try completely different approaches.
Practice, practice, practice!
Learn from mistakes:
there are no failure, there are only results
(Robbins)

So finally I decided to begun another iteration of my home-engine with new knowledge from books and papers.

FIRST DAY

Total time: 10 hours
Evident results: running up window, multithread engine loop on.

The first day ended with some great insight of what an engine must be (for me).
I've almost finished the Platform Abstraction Layer, on which every other layer will rely.
The first thing I noticed is that you must create a complete environment for other programmers to work with your engine. Many times, due to lack of documentation or time to study, it's better to create restrictions to code use and rails for the other programmers.
For example, if you want the complete control of how classes are accessed, you can create some macros that fordib (declare private) copy constructor and equal operator. Or you can typedef pointer, const pointer, reference and such and use ALWAYS them.
Another example can be the error detection: a bunch of macros like _CHECK( condition ), or _ASSERT( condition ), _ASSERTNOFAIL (condition ) can be EXTREMELY useful to have a consistent way of developing. With that you can redirect all these macros to write down in a global output device you messagges, and this can be in a transparent way to programmers: think of an external window that brings up when you are in debug, with all informations sent everytime a check fail (or success), something missing, asserting.

Consistency is the key.
Decide the guideline in how you want to handle situations, and then use all the c++ features to create this Consistency.

Watching other code I've felt that when you give too much degree of freedom in the code to other using your engine, they will always do what you have not expected.
This leads to the need of "extending" C++, or use it to place constraint and quickly change behaviour.

Constraints.
Flexibility.

These two words commonly are the opposite, but have a variable number of choises, but with precise choises, leads to choose one of the path you thinked of.
Take the following code snippet:

HBOOL WorkerThread::GiveUpSomeWork(WorkerThread* pIdleThread)
{
SpinMutexLock Locker;
_HPKM_CHECK(Locker.TryLock(&m_oTaskMutex));
_HPKM_CHECK(!m_uiTaskCount);

// Grab work
SpinMutexLock LockIdleThread(&pIdleThread->m_oTaskMutex);

// Taskpool has some new tasks, quit.
_HPKM_CHECK(pIdleThread->m_uiTaskCount);

// We have only 1 task, try to split it.
if (m_uiTaskCount == 1)
{
TaskPtr pTask = HNULL;

if (m_apTasks[0]->Split(pIdleThread, &pTask))
{
pTask->m_pCompletion->MarkBusy(HTRUE);
pIdleThread->m_apTasks[0] = pTask;
pIdleThread->m_uiTaskCount = 1;

return HTRUE;
}
}

// Grab half tasks (rounding up)
U32 uiGrabCount = (m_uiTaskCount + 1) / 2;

// Copy this thread tasks to the idle thread list.
TaskPtrPtr ppTask = pIdleThread->m_apTasks;
U32 i;
for (i = 0; i < uiGrabCount; i++)
{
*ppTask++ = m_apTasks[i];
m_apTasks[i] = HNULL;
}
pIdleThread->m_uiTaskCount = uiGrabCount;

// Move remaining tasks down
ppTask = m_apTasks;
for ( ; i < m_uiTaskCount; i++)
{
*ppTask++ = m_apTasks[i];
}

m_uiTaskCount -= uiGrabCount;

return HTRUE;
}

The macro _HPKM_CHECK checks the condition and return HFALSE if the condition is not met.
This in the Release version. In debug or profile version, you can substitute it with other commands that send an event to an output device, or maybe print something in the game console.
As you saw, also TaskPtr type is a typedef. This ensure that we can test and let some classes use smart pointers, provide timings about access with/without smart pointers in a transparent way.

As Engine Programmer, engine is not only a c++ (and a bunch of other languages in other subsystems...) code mess, but a TOOL with which everyone MUST express himself.
For me, this is something that really lacks in many engines, even commercial ones.
Placing many smalls constraint, guides and hints gives everyone the power to use the engine as its full glory.
With macros, templates, defines (really not new stuff...) you have to give CREDITED TOOLS to code with.
You have to redirect almost ALL calls inside your code in a way YOU decide.
Almost EVERY method call must be under your control. Even simple memcpy, strlen, sin...they must be wrapped and even in the case of using the standard functions, you have to decide it.
This TOTAL ABSTRACTION (almost) leads to better code control and later optimization.

Even in coding, DECIDING ALL is the key! Decide that every sin call leads to a modified version, maybe with a table-lookup, or to the standard function. But you have to DECIDE!

Outside of those condiserations, I've worked on the platform abstraction. This includes:

All types redefinition;
Multithread-pooltask implementation (completely abstract);
Timing management;
Engine architecture based on an abstract engine, and external-declared subsystems;
Client definition (under Windows a window that handle OS messagges);

Giving os/api-independent bricks (classes, components) leads to portable software.
The almost-hated virtual table, a good enemy on xbox360 and ps3 intensive operations, can be achieved in very straightforwarding way.
The final result is an almost controlled window that takes its messagges and call its task pool to use every kind of task possible divided on all the worker threads.
I'm implementing some interesting things like parallel data processing (parallel_for...) and want to contiune this way.

Of course, these are all mine thoughs and I know that there are many other way of doing the same thing better.

But you know, I have 9 more days to end the engine!

Demiurge

Challenge!

2010-03-17T23:46:00.004+01:00

I've always found that when you rise a challenge to yourself, and take it seriously, this can lead to intersting results.
I want to work on my decision-time, that I want to improve, and also speed up my coding abilities.
So I decided to rise a challenge: write a multi-threading mini-game engine in the shortest time possible. I'm deciding to give me a time of 10 days. I know for sure that there are days in which I'll not even have my pc with me (next weekend, and next week a couple of days...) but I want to try this.
The goal is to provide a basic framework for windows7, directx9.0c, cg capable of letting me create a small game.

Will I be strong enough to win?
Let's see!

Every day I'll write down on which part I'll work.
The challenge will begin thursday, so stay tuned!

Skinning!

2010-02-17T20:15:00.002+01:00

Finally I've made skinning!

http://www.youtube.com/watch?v=SwAIHFE64v0

This is a small video that shows the skinning with an FBX model! I'm satisfied for that, even if it costs me too much time due to the lack of documentation as of FBX...
But now I understood (with many trials and errors) the way to retrieve animation data, skeleton, meshes and material informations.
Next step it will be a big overall change of the engine in all its parts, because I've seen the current architecture is quite messy and it is not organized in a way I like.
This iteration of the engine, as already mentioned, is intended to cover all the aspect of an engine, and not only rendering (as my previous 2 engines).

Stay tuned for more informations!

Rendering design thoughts

2010-02-12T16:31:00.000+01:00

I opened a really interesting discussion @ gamedev:

http://www.gamedev.net/community/forums/topic.asp?topic_id=561645

I'm brainstorming and sharing my thoughts about rendering, and the mentality behind the renderer I use in my engine.
Take a look!

FBX, skinning and exploration

2010-02-08T17:16:00.003+01:00

Finally I came up coding an importer that can give me the possibility to access data that include mesh, skeleton, animations and materials.
This is a HUGE step ahead in importing asset for the engine, and this lead me to other type of battles and thoughs.
I found FBX lacking of documentation, the only serious source code is made by a guy that created an FBX importer using DirectX10 and very kindly provided the source.
Still, I wanted to use FBX as an INTERMEDIATE format, that is translated in my own internal format, for speed purpose: I'm also using Collada as intermediate format,
but still I have problems in exporting in Max2010 (a beautiful CRASH).

Next days I hope I'll publish some pics (or maybe a video) showing the skinning.
The BIG improvement coming from the skinning integration is the focusing on the problem concerning the render of different kind of geometries...this is really fascinating, because
I know for sure I'll change all my rendering code very soon. Why that? Simply because I want to *explore* other way of rendering, I've always used direct rendering but now I want to change to other solution, and found in deferred rendering (not deferred light rendering, this with light-pre-pass was done on the previous iteration of Hydra) a new way of thinking: create a command buffer, one for each thread, and then add commands from every thread.
A final merge and the rendering, and you have deferred rendering.
I read some posts around the web, and also the emergent presentation about this, and I found really a good way of thinking the rendering as a multithreading process.

The other BIG improvement is the way of thinking "data oriented"...it is really amazing, basically it is a way of thinking based on data ACROSS objects, and not only objects...so objects became a set of properties and data, but not in a constraint way.
It is really 'holistic' way of thinking about the code, and I want to try very different way of doing the same thing.
Strange is the fact that coding is the process of transform informations, nothing more, nothing less. And we are experiencing a BIG process of letting all more complicated.
Multithreading is taking to the road of seeing programming with a different level of detail, no more safety and simple flow, but a more wider use of the machine and its power.

Speaking of exploration...I'm really sad about the announcement about Castlevania:Lord of Shadows. It will be another GodOfWar clone with no more exploration...

Happy new year!

2010-01-14T11:57:00.004+01:00

Finally...2010! Happy new year!
I feel very excited about the new year: it begans with many news, not all good, but changes are coming!
I'm rean completion of the base rendering system of the new engine, I'm working on multithreading and I fell in love with Data-Oriented Programming mentality!
Next days I'll have FBX model ready to draw with every shader again, deferred/forward/light-pre-pass easy to switch and already implemented, then I'll move to finalize the multithreading base and physics base.
I want to experiment with a multithreaded framework, to see how can I achieve good performance and stability and usability!

Stay tuned...new experiments awaits!

Math library; Core Engine development;

2009-12-18T16:55:00.002+01:00

Even if I am not coding really fast, I'm studying different topics in which I find really different 'formae mentis': this is really fascinating!

Actually I written a non-template version of my math library, with some basic support for SSE/VMX code: premature optimization is the root of all evil.
So I'm moving forward to implement other core part of Hydra: camera, input and debug output!
Actually the camera is almost done, with all the code (like plane extraction, frustum vertex in world/view-space) working, the input is almost done and I'm writing the implementation of a directx9-based renderer, that will be used right now only for debug graphics to test that everything with maths works!

Also, I bumped in the Function Pointers World...really fascinating, with simple code that can make you skip some branch to choose what to do on different situations and have some performance gains...
I'll try to use them in the core part of the engine, to avoid virtual keyword and to skip some branches; this engine is not a commercial-one, so I am free to experiment everything keep my attention!

Small snippets

2009-12-11T15:42:00.002+01:00

There are small snippets that can be used very well...
One is a simple software interrupt:

__asm int 3;

The second is more evolved, but if used correctly can wield to know everything about your code:

#define Stringize( L ) #L
#define MakeString( M, L ) M(L)
#define $Line MakeString( Stringize, __LINE__ )
#define Reminder __FILE__ "(" $Line ") : Reminder: "

so if you use the #pragma comment (Reminder ": TODO/HACK/FIXME") under MSVC compilers you will see the message in the output window, and with a double-click on it, you'll be teleported on where the code is written, so you can keep track of all the stuff you have to change...during compilation time!

Really useful, small and smart ;)

P.S. A really simple profiler is born, now I'm writing my math library from scratch, so to see some eye-candy you have to wait some days!
Next steps include:

Basic math library (with future support for SSE)
Input handling (mouse, keyboard)
Camera and Frustum classes
Debug draw (lines, spheres, cubes)
System coherence tests (project/unproject, world/view/screen space tests)

In the meantime I'm designing the structure of the engine with all its new systems/parts, and after those steps I'll began coding all of that!

Stay tuned ;)

Math-o-centric view

2009-12-08T16:10:00.002+01:00

One of the thousands of changes I'm doing in the new Hydra is the math system: my math always based on a template math library without any SSE-kind optimization.
I went into a journey that highlighted the lack of centrality of math code and the lack of SSE.
Is it really necessary to use SSE?
The real advantage of SSE is the use of instructions that works on 128-bit registers, and that can contains, for example, a 4d vector.
The BIG advantage of SSE is to use the mentality used when writing shaders to reorganize data in
Structures Of Array and gain a near 4x speedup!
After some study, I've decided so far to eliminate the template aspect of the math library: floating points is the standard de-facto. The real twist is the centrality of math: to use SSE, it is better to create a place in which all math is done; this ensure the possibility to apply different kind of math calculations (better for multiplatform development) without changing code outside the math system.
I think THIS will be the real power of the new design: all math inside the proper system!
After that big change, SSE optimization is a matter of time.
For example, Frustum Culling: imagine a method that performs a Frustum-AABB intersection

int Camera::intersectAABB(const AxisAlignedBox3f* const aabb)

even if it can be cool to let the camera calculate the intersection (by information expert, it can be either the Camera or the AABB), it is smarter to move this method in the math system, like that:

int Math::FrustumAABBIntersection(const AxisAlignedBox3& aabb, const Frustum& frustum)

this assure that if you want to implement different math (SSE/VMX/SPU-like) you can do it using the preprocessor.

I'll try this different approach, and profile the changes from template-based-sparse to float-centric math, and also to float-centric-sse-math.
I am really CURIOUS about the results!

Stay tuned, next time I'll post some timing and talk about the Core of the Engine!

Starting from scratch...again?

2009-11-24T12:44:00.003+01:00

One of the most intriguing aspect of the game development is the tendency to hide the knowledge.
This is one of the most stupid thing that we human can do: hide our knowledge.
Why that?
Lack of self-confidence, company ripoff of staffs, and many others...I think it would be very useful to study some System Engineering and Systems in general.
Maybe a lecture like 'The web of life' could be REALLY USEFUL.
Sharing of knowledge let other people to improve what you are working on: think of the open source.
One day...the gaming industry will be more open, and maybe not every company will start its own engine because it has no money to buy one! :P

Sorry for the rant...but I like sharing ALL, because I find that everybody can learn you something, even a 'negative person': it teach you what NOT TO DO!

Apart from that...I've started rewritting the engine, to go multithreading and to apply all the knowledge I'm building now. Nothing more than an engine to understand the base of all the game development and to experiment new things...
The third iteration of Hydra (codenamed 'Turgid') will be available also on SourceForge.
I'm pretty excited about it, because I can concentrate on different aspects every time!
The first iteration was about...EVERYTHING! Two years ago, I've know nothing about game engine programming. So I studied all from scratch, trying to create a base renderer.
The second was about shaders: I wanted to develop and to investigate the beautiful world of shaders! It ended in a Light-PrePass renderer with shadow maps, screen space ambient occlusion, lightshafts, hdr...many interesting things!
What about the third? I want to enter the multithreading world, I'm really fascinated about it. Also, I want to extend the engine to became a more robust and complete framework, including also a small editor, some physics, AI and sound! And...continue studying special effects, shaders, DirectX11...
The last thought about the third iteration is about the multi-platform development: the actual design had taken in account the possibility to use OpenGL and to run under Linux and MacOS; about this...time will tell!
I want to update this blog more frequently and take many screenshot of the work in progress...

Stay tuned!

Moving forward: Portals, DX10, Multithreading!

2009-11-13T10:08:00.002+01:00

REALLY busy time, luckly!
We are working hard and beautifully in my company...we are experimenting and learning new things, so it is a great time!
In my spare time, apart from learning playing flamenco and jazz, going gym, reading many books (Robbins, Dills, Dyer...) , I'm studying some interesting things: Portals, Dx10 and Multithreading!
Currently I'm implementing a simple Portal System, and designing a completely rewritten game engine that takes in account the multithreading as the base brick of all the structure...this is really fascinating, because nowadays most applications are single threaded and the multithreading is added AFTER the creation of the project.
I've installed Windows 7 on my laptop, and I have to admit that it seems a good operative system: vista-eye-candy, nearxp performance. That unlocked the doors to Directx10, with all the goodies that cames from...the Directx11 also can be used, but on a Geforce8600m gt there is no room for that!
So basically...time to study. Hope to post some screenshots of the portal system soon!

SSAO!

2009-09-22T03:37:00.003+02:00

After some fights...here comes the SSAO!

This is the beauty of the 'waterfall effect' of the deferred/lightprepass/z-based rendering: one you have coherence between different spaces (world, view, projection, texture...) and you switch between them rather easily, you can begin working on many interesting effects :)
This is a real basic implementation, based on various post on gamedev (thanks guys!!!)

Finally a new connection!

2009-09-21T03:51:00.003+02:00

Long time no see guys!
This is the first day I have an internet connection in my new house. After 3 months actually!
I was really busy at work, but I continued programming.
My new challange is the Deferred Rendering...a real cute rendering architecture!
After many reads (mainly gamedev) I moved to my first implementation: I found really useful to do all the calculations in view space, the view space pixel position reconstruction became really simple.
Also, the G-Buffer is actually composed of only 2 render target, rendered with a MRT: normal and linear depth. The result is a Light-Pre-Pass-Like renderer...I'll show a little result:

10 pointlights on the screen with the known rockwall normal map.
I'm also working on the ScreenSpaceAmbientOcclusion so I can move to create a little demo.
Stay tuned!

Actual State; Near future; Future;

2009-07-27T09:58:00.002+02:00

Some changes have happened in the last month.
I changed city, company, life.
I moved 700km away from my home, in a more prolific environment and with cool guys.
This is the main reason of the slowdown of the blog!

I'm actually coding hard, because I want to study some topics in which I'm weak, and then I want to merge all of them in the new Hydra!
During these days I'm creating some really small demos that covers different aspects of game programming:

- GUI and Font handling;
- Animation;
- Collada import;
- New graphic pipeline;
- Lightmapping;

These different topics are really interesting, and I'm quite happy of the results.
GUI and Font handling it is something simple, but can give you satisfaction and OUTPUT.
Animation is a really tricky topic, I've done a small exporter in maxscript and all sounds good,
but then I've read this article (link) and found that an importing pipeline is a good choice, even if more difficult for some aspects! Then I began using Collada...really good and FREE.

New graphic pipeline. I wanted something more shader-oriented than before, so I switched to Cg...really COOL! The rendering is organized in pipelines...each one has its own stages. Every stages can specify one or more render target and all the textures produced are specified in a common pool of texture of the pipeline. The next step will be involving a node-based editor for the creation of the rendering phase...I found it very easy and powerful, something that can istantaneously change the appearence of the game to see different pipelines and render, to study the feel of the game and tune it up!
With that in mind I begun my lightmapper. Through I started only some days ago, now it output a simple lightmap with a really simple phong shading: actually I'm spending time in creating a good unwrap and to create the radiosity normal mapping brick that I need to complete this step!

Until september I've no connection at home...so updates will be randomic! :(

But...stay tuned!

Secret of Mana

2009-06-17T23:46:00.003+02:00

Yes...I want to know the Secret of Mana.
Long time ago...there was two special games. One was The Legend of Zelda: A Link to the Past, and another one was Secret of Mana (I don't want to mention Chrono Trigger...I'll do it in another post).
What's about SoM? I really enjoyed it, playing with my best friend and sometimes with my sister...all in realtime? What?
Great story, great ambientation...and great gameplay. The weapons you found and the spell you learn grow with the usage. The fights are in realtime, the AI can control the other in the party or you can play together.
One of my greatest dream is to create a game like that for some console one day...with the beautiful music and all the magic that Secret of Mana has.

But now the mana is weak...and later games fail in public reception.
One day, I'll forge a new Mana and all will be good for the Universe...

Updates, ideas, thoughts

2009-06-04T01:41:00.002+02:00

Finally I'm finishing the animation demo I've scheduled. After some research I found a good way to handle animations with blending and transitions. The results are good, the code is simple and I've both CPU and GPU skinning implementations.
Next will be the 2D demo...sure I want to develop something simple yet powerful!

On the other way I'm researching on both toolset development and multithreading. The latter is a difficult and big subject to study, but also so fascinating! I've not found many resources around, I think not so many people use multithreading in a correct way...
On the toolset research, I'm creating an architecture that is both GUI and Engine indipendent.
Why so? Is it possible? Yes yes...it is possible. I wanted to create something that can grow with me: I've always believed the power of toolset, but I know also that much time is needed to create something robust and useful. This little/big idea let me create my own editor and develop it during time...I'm really excited about this software design!

Last but not least...the last guardian (sorry for the repetitions :P). Amazing. Wonderful. I want to make games with such magic...the real state of the art. If you haven't seen it...click here .

Stay tuned!

Plans

2009-05-05T13:32:00.003+02:00

The best way to create something is to actually DO it.
Because of this, I began planning some smalls tech-demos for my always-in-development engine Hydra: these demos will give me knowledge about different topics, and incrementally give Hydra more features.
The first, small demo I've done was about culling and the new shader oriented pipeline...I've posted some screens not too much time ago (see this and this) and that debug lines are a giant step towards engine's features development. This is something known to many, but the separation between code and data, and the need of data to be displayed, can be overwhelmed by using debug methods (draw 3d lines, 2d lines, circles, spheres, boxes, cylinders).

Also the use of custom exporter (my case is maxscript) lead to another problem: how to export data in a cross-platform manner and in an efficient way. The solution is simple: Collada. It is the perfect intermediate format that can be translated in platform-dependent code, and that exports almost everything graphical thus physics objects. The use of Collada was suggested by the problem of exporting animation (SRT, biped, skinning) data from 3d studio max...what if I found a model created in Maya? And the old release of 3d studio? Using custom-made scripts it is great, maxscript is very simple yet powerful (creating a custom exporter need a small amount of time, personal experience), but when I was developing the script I had to create something adapted to Hydra...abstracting, introducing a new layer was the answer.

The second demo, as you can imagine, is about animations exported using Collada. I'm thinking of doing first CPU accelerated skinning, I've got a bunch of GPU-accelerated shaders that performs skinning, even if I'm not so confident about the real power of GPU...it becamed too much used, and cores are multiplying!

Third demo will see a simple 2d framework with menus, font management and console.

The fourth and the fifth will be the harder part...radiosity lightmapping and scene management. Lightmapping will be only an algorithmic problem, scene management will be a design problem.
The question about how to describe scene, and let different subsystems (graphic, AI, sound) interacts is way too bit complicated to be explained in this post. Also, there are many design flaws in every different implementation...just Keep It Simple and Stupid.

Hatching love

2009-05-04T07:08:00.002+02:00

Some years ago I began cultivating a passion for drawing (now possible only in my sparetime) that evolved in some 3d modeling...one of my favourite technique was "hatching", in which volumes are "described" by simultaneous lines that intersect creating grey-scaled images and giving the idea of volume. Even now I'm not a great modeler/drawer, but I find this fantastic shader to be linked to my vision about drawings (see comics like Berserk or Hokuto no Ken to see it in action).
So based on my first and only semi-serious 3d model, Zoddo from Berserk, I finally came up with this fantastic shader, even if the model has serious normals problem (that affects lightning :( ):

I wander what can be the feeling with a 3d world shaded in this way...

Parallel access: SSAO, Light Pre Pass

2009-04-28T00:53:00.003+02:00

During these days I'm changing many things to Hydra: mainly there is a graphic revamp...then I'll be ready to make some AI/gameplay experiments!
I'm developing the lightmapping/radiosity part...this is far from being complete; the new graphic pipeline is almost ready, even if there are some issues here and there...
But tonight I wanted to develop some useful shaders with Fx Composer...

SSAO

Light Pre-Pass

The power of Hydra is that it has 1to1 Fx Composer/InEngine shader graphics: I'm using CgFx library to parse and create the shaders, I found really easy to use but yet powerful. So creating shaders for Hydra is just a matter of time: open FxComposer, create the project, compile...then simply copy the file onto the shader directory, create a material with it...DONE!

There were two great techniques I was impressed by: Screen Space Ambient Occlusion and Light Pre-Pass renderer.

The first was developed by Crytek in 2007, adding a great realism to the scenes because of the use of the ambient term in the lightning equation (always set to a constant-flat value). This calculation is done entirely in realtime, and even if it is something not much performant on many machines, there are plenty of them that can handle this beautiful effect...

The second technique is a renderer design created by Wolfgang Engel : it is a mid point between forward and deferred rendering...really really impressive! It lets you separate light from geometry affected by it, simply rendering the normals and depth to a buffer, then render all the lights needed, then use the forward rendering paradigm using the light buffer for lightning calculation. Even if I have some problems with it, and I've modeled only the diffuse term of the lightning equation, I'm really satisfied tonight.

Take care...

Radiosity lightmapping and new Renderer design

2009-04-04T19:45:00.002+02:00

Hello everybody,
is a long time since my last post. I was busy at work and in some ideas for my engine, that are still in a primordial stage.

After being fascinated by the Radiosity Normal Mapping Technique by Valve, I've decided to try the way of a lightmapper. I know it's something difficult, but being something I've never do, I want to give it a try!

The result is a good bunch of links about radiosity and lightmapping in this thread I've opened (here I am JorenJoestar) on Gamedev:

http://www.gamedev.net/community/forums/topic.asp?topic_id=528414

Also here there are some more informations for radiosity on GPU.

In the meantime I'm designing a new Renderer that it's really flexible. There are still some shadows on the design, so I've came up opening a post on gamedev to receive some critics and some ideas. I've decided to give it a "submission date" after which I'll begin the coding: it's annoying to be stucked with all the things I have to do!

More on that soon...

Culling and scene

2009-02-15T15:34:00.006+01:00

Speaking of culling, I've finished frustum culling and rendering using an octree as spatial structure. I've began also working on occlusion culling, and found a very intersting paper:

Coherent Hierarchical Culling Revisited

You can download the paper from there. It seems a very good techniques that uses Hardware Occlusion Queries with many render-state friendly optimizations, and seems also a real improvement to the basic technique.
But before moving to occlusion culling, I'm studying the design of the scene management, so when I've done with it I can create a real scene to test the CHC++ and also some simpler techniques of occlusion culling (using occlusion planes/frustum entierely on CPU). I found very interesting the use of occlusion queries, but if the scene you are about to render uses heavly the GPU it can impact the performances too much.
With the advent of the multicore cpus the old-fashioned-cpu culling can be of great impact on the performances.
In the meantime, I'll attach two simple shots of some random-placed meshes (the same as the previous post) this time entirely culled and rendered:

The first is the scene seen from the camera, the second shows the frustum and the object rendered.

Octree and HydraIO!

2009-02-13T02:30:00.002+01:00

Hello again...now that I finally closed the section about the previous engine, it's time to talk about the new one!
The name HydraIO, that means "Hydra2" read as binary code, and is the natural successor of the first one.
This version will be more complete and will have more features than Hydra, but it's still under heavy development. The architecture is totally new, the level of abstraction of the resources is different but let me implement all the resources also with consoles in mind, finally the shaders are totally data driven and uses Cg effects as standard. The old assembly used on the first Hydra is dead!
Actually the engine let me load models from 3d studio max, apply a cgfx effect totally data-driven, and now I'm working on culling.
This is the first shot of HydraIO:

a really simple Octree in a random based set of static meshes, next I'll attact a debug shader that let's me know if an object has been culled or not. In the meantime, I want you to know a simple trick about the criteria to be matched to end dividing the octree. In many papers I've read, there's always mentioned the criteria based on the number of polygons: if you are under a threshold, there's no way to not split the octree.
Another simple critera is to let you decide the aabb minimum size, so if you are slicing too much the octree doesn't divide: this was a real salvation for avoid infinite cycles!
Staticmeshes are meshes that have their points transformed in worldspace directly, thus all their shader will have only the multiplication: view * projection, and that are fitted in the octree.

Hydra History #3: the twist in the tale

2009-02-09T22:25:00.004+01:00

Then it comes the light.
But first, the tangent and bitangent problem...it causes some really annoying (but funny) parallax effects!

After solving this issue, I began fighting with another great effect: the shadows! After a while, and maaaany problems, I've figure out that the road to the demo lead me to a simple shadow map, with the light cast by the player...
This is the scene without the relief, but with the improved water.
Oh yeah, the water!

The other great effect to improve...after some trial, I've ended with a water like that:

I've used a simple environment map to enhance the visual appeal. The noise is terrible by the way, but tweaking it in assembly was a real PAIN!

This shots shows the technique for reflection and refraction, taken from GPU Gems2 Ch19.
Using also the copy of the framebuffer in directx, you can skip the second render of the scene that contains the refraction mask in the alpha channel!

The final set of effects was near completion, but with some problems. Still, this is not a demo that satisfied myself.

So I ended up writing an importer from quake3 arena bsp maps...

And added some basic physics with Nvidia PhysX.

On august I ended it and in september I began working in the game industry...but still there are many things I want to know and that I'm studying.
This was my first engine from scratch, now I've began a second one (called Hydra IO ) that is different in architecture and more powerful.
Soon I'll post some shots of it, but right now I'm working on the core...

Hydra History #2 and an interesting article

2009-01-20T22:22:00.019+01:00

On the path to the new engine, I faced the challange of the resource manager design and this turned in an interesting discussion on gamedev. This is the link:

Engine design and resources

Talking about Hydra, the first release was a great success for me. It was a graphic engine wrote from scratch (int main...) and during its lifetime it teached me much things.

This the second shot of my engine. At the time it was only in wireframe, and I was developing an importer from the 3ds format, due to my previous work with 3d studio max.

After that, I've began coding shaders but using asm shaders...this was actually a pain. The renderer has both Direct3D and OpenGL support, so I had to create both shaders two times.
To solve this problem, I've began coding in Cg and then converting to asm every time. This process was really slow and the shaders I've implemented were not optimized and much time was spent to fine-tune them (code, export in asm, modify gpu registry bindings...).
To support some sort of scene hierarchies, I've implemented Bounding Volume Hierarchy:

Those two models were under the same node, so the resulting volume must contain both the volume of the models.
This screenshot used OpenGL renderer.

Having a simple scene hierarchy system, here comes the time to develop something more interesting.
I have always love water, both in the reality and in 3d rendering (even using max), so the natural choise was to study how to achieve the water.
I've seen what's missing in my renderer, so I create functions for both OpenGL and DirectX to support culling planes and render to texture. I created a tube under max and added some noise to create a cave, exported and then I've tried to create the water.
This was my first attempt:

This image seems like coming from ten years ago, but it was a success at the time. The reflection was done via render to texture and culling plane, and the result for debug purpose can be seen on the lower left.
The refraction was poor, and there were no noise and no light.

After this preliminary water, it comes the time to face another subject I love: shadows. This was the REAL pain. I've passed nights and many posts on gamedev to achieve a simple shadowmap, due to the texture format, due to the poor pipeline: cg->asm->hydra. It takes long time to see the shadow map working, I don't know even how much!

Finally the long waited shadow map! Ugly but hey man...I've learnt many other things!

And then I wanted to give another touch...using normal maps.
I've noticed a family of graphic effect that derive from normal maps like parallax occlusion mapping, relief mapping, cone step mapping and stuff like that.
So I've began studying them, creating shader for normal, simple parallax and relief with fake soft shadows using FxComposer. It ended up with the overused rockwall:

Another problem here was the lack of informations like Tangent and Bitangents.
So I create a function to fill the vertex buffer with those informations...using the math explained in this article: Tangent Space Calculation

The end of the story will be in part #3...stay tuned!

Demiurge

Hydra history #1

2009-01-08T13:32:00.002+01:00

A years ago, I've began coding a graphic engine called Hydra. It was very important for me, because it learned me much things about different fields, like graphics, engines, directx/opengl, cg, hlsl, windows and linux.
After some months of nighttime development, I've created a demo and a video about it:

http://it.youtube.com/watch?v=Bx_WeMigxRA&feature=related

Even if it's not an impressive demo, it is based totally on the renderer made by me. Started from scratch. And this leaded me to continue to study and research about graphic and engines.

And leaded me to finally be a game programmer!

Long time...

2008-10-28T22:49:00.002+01:00

It is a long time since my last post.
Many things have changed, and many were improved!
First of all...I've done it! I'm working in the game industry! In september I've began working @ Gd-Mind, a small game company located here in Rome. The peoples are great, Tiziano Lena (Lead Prog) and Gabriele Barlocci (Senior Prog) are quite great masters...and day after day they teach me many new things.
The Hydra Engine finished with a small demo, but during the entire process it teached me many stuff about engines and now I'm thinking of starting a new engine from scratch, the old architecture is quite too complex. The engine included NVidia PhysX for physics and lua for scripting, because of the will to expand to a game engine and not a mere graphic engine. But this lead me to think about a new and more simple architecture, and choose to support only a graphic API.
Stay tuned for more info about the next steps...

The power of design...

2008-03-26T12:49:00.003+01:00

The step is done. Friday I've done the design of the new architecture, that will allow a multi-platform/multi-render development of my engine, all based on libraries. So I wrote down all the subsystems, the new engine tree, and the code came up from my fingers during friday night.
The result is an SDK, useful to create demos and applications focusing on the apps and not on the renderer or the OS.
I feel very satisfied about it, it is modular and let me concentrate on the next steps: developing an OpenGL renderer and a OpenGL/Linux application, to let the engine be developed with my two main OS.
I'm working also on an embedded version of the engine in WxWidgets, so I can create some useful tools to build shaders, scenes, and stuff like that.

Casper: today he will become Software Engineer. With the max possible rank. Congratulations!

P.S. I'm Demiurge. The second is Casper...and the next?

First milestone!

2008-03-19T11:00:00.002+01:00

The time has come to close the first milestone of my little engine.
Some basic features, but still important (like a full math library, geometry and texture importer from 3Ds files,
render to texture and basic shader, basic camera and scene managment).
And now a new need.
The need to move to multiplatform/multirenderer architecture, the need to see it in every way possible.
I'm still confused about the solution to this problem, cause the windows development was fast and easy with visual studio 2005 and DirectX9, but more satisfaction is yet to come.

It's time to design - how to build over different platforms.
It's time to design - how to let my brother work on our dream.

Welcome Caspar!

Render to texture

2008-02-01T12:06:00.000+01:00

Finally the times come.
The times to develop one of the most useful feature of an engine: render to texture!

Here is the link to my simple scene...rendered 2 times: one to create the texture and another to
create the real scene!
http://i110.imagethrust.com/images/
2YVi/view-image/hydrarendertotexture.html

I like it...it begins to take shape...

BREAKING NEWS!

It's time to grow up...I'm pleased to announce that Casper, my best friend and brother, joined me in this adventure.
Say Hi to Casper!

Bad Fool Prototype

Gentlemen, start you engines!

2008-01-24T17:20:00.000+01:00

Be prepared...
This is only a trial...
To share some notion,
to learn new stuff.
To put answers and
find new questions.

A solid backup of

ideas and solutions.