Threading

May 1, 2011 at 9:14 PM

Hi guys

First of all, good luck for your project that already looks very nice !

Note: There is a "bug" because when a thread delete a block there already commands to light it or build it. It should clean build & lighting queues before. Anyway that's not the point : )

I saw your threading architecture for the "NewTake". Some advices while it's still in refactoring, hope it will help
About generation you have (or plan to have) 4 threads, one for generating, one for lighting, one for building and one for deleting. This is not very scalable. I have 8 cores in my computer, others only have 1 or 2.
You could use a single queue for commands, and create as many threads as you want, picking in these commands. Threads do not need to be specialized in a precise task, all they have to do is run in parallel.
So user (or engine code) could choose how many threads to create at launch, and it would not affect code structure.
Moreover, this would ensure that commands are made in order (Generate>Light>Build>Delete) so no block can be deleted if not Build, etc.

Each thread could behave like:

lock(cmdlist)
{
  cmd = cmdlist.Dequeue()
  switch(cmd.Type) {
    case CmdType.Generate: DoGenerate(cmd);
    case CmdType.Light: DoLight(cmd);
    case CmdType.Build: DoBuild(cmd);
    case CmdType.Delete: DoDelete(cmd);
  }
}

What do you think of it ? Am I missing something ? I could help to refactor it if you need a hand.
Anyway, I'm trying some new terrain generator to generate a town, and it's already quite funny !
Good luck again !

Developer
May 1, 2011 at 9:43 PM

Hi,

You're correct about the separation of tasks into separate threads. It was only added yesterday temporarily to investigate some threading issues. It will be removed again soon. Threading is still under development, and you might see some commits that are purely for testing something also as the pre-alpha moves towards alpha.

Thanks for the tip also. If you want to attempt an approach with it, we could always look at whether the code could be used in the main code itself.

The town generator idea seems good. If you feel like sharing some pictures of it, that would be interesting also to see.

Coordinator
May 2, 2011 at 1:37 AM

About generating a town,here s one on my todo-techcraft bookmarks ;)   

http://recreationstudios.blogspot.com/2009/12/procedural-cities-in-xna.html

So i would really like to see your progress !

About the threads :  go, do your own renderer (bad name by the way)  on your fork, we need some fresh look on this part.  I ll test it and pull it in the main code when it will be ready ( as jacoo said ! )

What im currently trying to achieve is getting rid of the remove_range: If we always remove a chunk when adding a new one, there is no need to scan for chunks to be removed. But I did not invest a lot of time in this the last days, and my vector maths skills are not the best, too much years of business oriented code ;) . So any help would be really appreciated !

Coordinator
May 2, 2011 at 3:01 AM

Newcomers will appreciate the new class simpleRenderer :  no threading, no infinite world, an easy to use basis for new developments.

This may be of use for testing light optimizations and trying to find why currently fullscreen mode has so bad fps on laptops !

http://techcraft.codeplex.com/SourceControl/changeset/changes/4a332a673e33

May 4, 2011 at 11:31 AM

Some thoughts after investigation the engine a little:

- the modular approach seems quite nice (generators, renderers, controllers, tools...)
- The "Renderer" (like ThreadPoolWorldRenderer I made) includes blocs generation and rendering them, maybe you could split them.

I tried to rewrite some code to make a simpler threaded processor, but I ran on multiple multi thread issues when I put more than one thread to build/light/generate (I put 6 on my i7). These include:

- Chunk code is not multi-thread-save, in particular _N, _E, _S, _W members, I had to remove it
- Lighting code access "N, E, S, W" multiple, the value can change weanwhile, this leads to crashes
- Vertex buffer generation not MT-safe
- Some generators are not MT-safe (TerrainWithReivers uses shared members)

So if I continue on this way, I will have to rewrite quite a lot of code, on Chunk class in particular. I'm not happy with generation as block lighting is usually made more than once (due to neighbors blocks appearing)
I suppose others are working on these, so maybe I'll stop here for now. Tell me what you think of the code I made and we'll see if I continue.

enomi : about chunks creating & removing, I made a stack of unused chunks and I reuse them. Like this no "new" made if you d'ont change render view distance
jacoo2 : no progress on town generation as I spent time on the threaded generation, I'll try to start this now.

Coordinator
May 4, 2011 at 6:25 PM
Edited May 4, 2011 at 6:25 PM

Hi, thanks for your good work and input. Did you forget to push your
commits to your fork ? I cant see any new code there !