Some new implementations

Mar 30, 2011 at 3:32 AM

Hi,

I've been following this project for a while and I wanted to say I love the new project direction.  I implemented some new features that might interest you:

Infinite world
Chunks generate and render based on the players position.

Multi-Threaded
I had some frame rate hiccups from generating chunks on the fly ( mainly due to building vertices ) so I put vertex building on a separate thread and had it run constantly.  I also only load the chunk closest to the player per frame so that the locks don't hold up the draw calls.  I keep 60 fps constantly with this method.

Server Client Model (Still in progress)
I added a networking engine that I had been working on for a different project.  The server or host holds a separate world from the clients and sends them chunks serialized into a binary format (protobuf) based on their position.  The server saves every so often into an xml file which contains data for the loaded chunks and players.

Small Changes

- Added a SelectedBlock class which represents which block the player is currently targeting.

- Using the SelectedBlock class, I added a mining mechanism similar to minecraft.  It takes X swings to break a block depending on how tough the block is and a crack decal renders over the block as you dig.

- Added a better terrain generator with mountains, trees, caves, water, beaches surrounding the water, and special tile placement such as gold and iron ores - it looks great!

 

I'm still working on it so the code is very far from commented.  I should have some ready in a couple days if you're interested.

Coordinator
Mar 30, 2011 at 4:06 AM

Of course I am extremely interested ! And I m pretty sure all our silent followers are too ;)

Did you achieve to render the world around the camera / playerPos instead of  world coordinates, to avoid float instability when coordinates get large ? 

In fact I ve been working on the code tonight, and i saw the problem first hand by positioning the world origin at 10000*16,0,10000*16 : the camera jitters a lot.

I ve done few refactorings / clean ups tonight, but now I do not want to commit all these , merging your code is way more a priority !   

Shoot me the code as is in a zip at enomi dot enomi at gmail dot com !  I ll read it tomorrow morning :)  Or commit on hg, I m making you a commiter now !

 

Coordinator
Mar 30, 2011 at 4:32 AM

Oh and just few more things :

- Most important :  I did not take the time to put back the frustum culling from the old code ! 

- There is a bug when digging chunk borders, the nearest chunk(s) should be marked dirty to fill the gaps ( or maybe an other trick if rebuilding up to 4 chunks at a time when digging a corner impers performance ) 

- about server side, I would like to have it running on linux too, I hesitate between writing it in Java ( leveraging things like apache mina , and 64 bit jvm for having lots of memory ) or maybe in mono.  I m open to any thing !  Regarding protocol design,  I intended to only send modified blocks on the wire and let clients generate terrain blocks, if carefully designed even trees generation and the likes can be deterministic , based on a seed.

Hope you ll send the code soon, I ve got hundreds of gameplay ideas, performance optimizations to try, some sort of unit testing to implement... lots of things to do !

 

Mar 31, 2011 at 4:12 AM

Hey enomi,

I didn't render the world around the camera yet as I have other priorities but it will be easy, just use a world transformation matrix instead of moving the camera so that the player is always at 0,0.  This will require some physics rewriting.

I like your protocol design, that's basically what I've done.  There's a random seed generated by the server that everyone gets and the only chunks that are saved and sent are the ones that have been modified by a player.  A linux option would be great.  The server i have made is for the xbox / windows but multiplatforming would be a sound goal.  Xbox does not let you connect to servers outside of xbox live :(

I have already put in code to set chunks dirty around the block that was dug - the only problem is with vertex building on a separate thread it flashes a little because the draw call happens first sometimes :( - ill need to immediately build the vertices from the main thread to keep this from happening but for now, aesthetics are not my concern.

I didn't even realize the frustum culling was missing! doh!  I was so happy with my fps that I didn't notice.  I'll toss that in if you haven't already.

Sorry it's taking so long for me to upload it, I'll work on uploading it tomorrow - I've been pretty busy today.

 

P.S. We should coordinate some goals for each dev so we can merge our code easier and so we all aren't writing the same things.  I think I'm going to merge the original water rendering code next if you haven't already.

Mar 31, 2011 at 5:30 AM

I forgot to say I also added the option for transparency in solid blocks (leaves for instance).  Just add this line in the solidblock shader:

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
    float4 texColor = tex2D(BlockTextureSampler, input.TexCoords);
   
    if(texColor.a == 0)
    {
        clip(-1);
    }

    float4 ambient = AmbientIntensity * AmbientColor;       
    float fog = saturate((input.Distance - FogNear) / (FogNear-FogFar));   
    float4 color =  texColor * input.Shade * ambient;
   
    return lerp(FogColor, color ,fog);
}

 

Note: you will need to render the block faces that borderline these transparent blocks

Mar 31, 2011 at 12:50 PM

why are you recreating triangles? why not use instancing or just move old triangles

Coordinator
Mar 31, 2011 at 1:58 PM
Skurcey wrote:

why are you recreating triangles? why not use instancing or just move old triangles

because it works ;)  the loop checking faces visibility for one chunk runs so fast that messing with independant triangles seems to be a waste of development time.

Working just on the model ( block array) and letting it rerender is way more easy ( = less prone to bugs ! )

There may be room for improvement, but changing this sort of things requires some measurements before !I m not working for 5 hours to gain .0002 second on a loop when ne next loop takes 0.2 s !

about instancing I m not sure I follow you , but you can read this article , http://blog.eckish.net/2011/01/ this guy did test vertexbuffer  vs instancing for minecraftish terrain,and vertexbuffer was the clear winner !

Coordinator
Mar 31, 2011 at 3:06 PM
Edited Mar 31, 2011 at 3:23 PM

Great news Durrban.

I only did some refactoring to isolate different coordinates systems ( world.viewableChunks / chunk.index  vs Chunk.postion vs vertice positions ) in preparation for the rendering around camera thing.

I m not commiting it , it will be more easy to merge or redo this after your own big commit !

The transparency trick is clever than having a separate vertexarray just for transparent blocks, I hope it performs well and does not make the card switch render states between each transp and solid block (no idea if clip(-1) does this !).

- Leave the frustum culling for now, it depends on the "rendering around camera"

- For water rendering , have a try but be carefull : the old code made some sort of out of memory when you let the water flow more than 10 blocks ( at least on my laptop).

I would really like more realistic water flow : the amount of water would be fixed, when it flows water blocks should move not be created out of nowhere. (Source blocks are the exception)

When this will be done I d like "pressurized lava" flowing up =  volcanoes = very dangerous to dig too low !  

Of course all this would be gankers paradise on a public multiplayer server but makes a very interesting single/coop players world  .

 

And I ll enter some work items to coordinate goals. [ DONE - click on advanced view on the issue tab ]

Apr 1, 2011 at 5:53 AM

Also Great News,

I've successfully rendered the world around the camera.  I'm currently sitting at 1.6 Million x 1.6 Million with no camera artifacts whatsoever!  I've implemented frustum culling which works as well.  There are some minor physics anomalies but it can be fixed easily.  Got water working but no flow yet.  Made a "CharacterPhysics" class that can deal with player and enemy physics.

To speed things up, I had to draw away from your separation method :(  I am recoding everything to use parameters and not the TechCraftGame singleton then it will be ready.

Coordinator
Apr 1, 2011 at 2:38 PM

Nice !

I Hope to see the code soon to understand your concern about "separation method". Send me a zip if you encounter any problem with hg mercurial ( enomi dot enomi at gmail do t com) or even if you do not have the time to finish everything, i can make the commits for you if it can speed up the process.

About our dig at border/corners bug, one solution may be to implement multiple chunks in Y direction, having 16x16x16 chunks instead of 16x128x16. This sort of things requires taking measurements in release optimized mode ( not debug ) . And  it may be a good idea to use the /arch:SSE2 compile flag too  !

Apr 1, 2011 at 10:07 PM

Awesome!

Coordinator
Apr 2, 2011 at 4:03 AM

Gooooo Durrban ! Check it in ! We want to see all those marvelous things !

Coordinator
Apr 5, 2011 at 3:18 AM

So finally I did merge jacoo s fork in the main code.  But we still do not have all of your features, so Durrban, if you come back, we are still interested. Maybe you can help us fix the current code !  

Coordinator
Apr 7, 2011 at 1:13 AM

The threading in this version is very broken...you'll probably see my comment on IRC in the morning ... i got the boundary rendering working i think (i'm pretty sure, but can't be certain 'cos of other issues) basically the chunk building and vertex buffer building run on different threads but aren't synchronised properly so sometimes it builds a vertex buffer from a partially built chunk. I'm going to rewrite things using the .net4 task factory threading stuff that raistlin mentioned on Irc the other day. I've been ripping my hair out 'cos i was almost certain that my boundary code was correct but was getting all sorts of messed up rendering.....i hate multithreaded programming  - debugging other peoples multithreaded programming is my idea of hell, but the new .net 4 task stuff that raistlin mention looks ideal for what we need - i spent a fair bit of time researching it today after he mentioned it last night. I'm out tomorrow night so it'll be friday before i get it done (unless i can get it working during my lunch hour hehe), but i'm on the case and will get it all working nice by the weekend!

If you want to try what i've done re the boundary issues in an older version replace BuildVertexList in ChunkRenderer and BuildBlockVertices in VertexBlockRender with these versions

http://dl.dropbox.com/u/9584488/BuildVertexList.txt

http://dl.dropbox.com/u/9584488/vertexblockrenderer.txt

I've commented it up so you should be able to see what is going on 

S.

 

Coordinator
Apr 7, 2011 at 3:49 AM

I commited Bamyazi's code as a separate chunkrenderer :  BoundariesChunkrenderer.  Just uncomment a line in worldrenderer init method to use it.

As I m not the best person for helping on XNA multithreading, I added some keyboard functions to help debugging / profiling :

Escape : clean exit, aborting the threads on exit (works also with escape key, window close, alt f4).

F4 : resizes the window to 160*100 , useful for looking at your profiler window when your screen is small, press f4 again to restore initial resolution.

Space : release the mouse cursor (press again to restore camera movement)

 F7 for wireframe is still there too, now in Game1.cs

Developer
Apr 7, 2011 at 3:53 AM

Great.

I knew about the threading issue when added in. It's only partially implemented, yes. The syncing was/is something that I was thinking of adding in once the subterrain vertex lists were resolved. There was also a discussion about chunk pooling. The only immediate purpose of adding in the thread and queue to the chunks (worldrenderer) was to reduce the lag.

I noticed the vertex buffer and partial chunks, as commented in a recent commit.

I can get started on the task factory if you want?

 

Coordinator
Apr 7, 2011 at 4:20 AM
Edited Apr 7, 2011 at 4:23 AM

I just re added the FPS counter with F3 to toggle between VsyncFixedtimestep / noVsyncAndNoFixedTimestep. The App now starts in vsynch mode.

You can get started on the task factory if you have time, or wait for bamyazi, as you want !

I wont commit til tomorrow now.

 EDIT : And now I pushed the commits too ;)

Coordinator
Apr 7, 2011 at 8:04 AM

Hi jacoo2 - sounds great if you know what the issues are and want to go ahead since i can't work on it today - i'll be back on the case tomorrow if there's any problems with the boundary stuff. I check in on Irc when i get home tonight and see how its going.

S.

Coordinator
Apr 7, 2011 at 1:09 PM

frameratecounter.cs is missing from the source

Coordinator
Apr 7, 2011 at 2:50 PM

Sorry I cant commit from where I am now, proxy problems ( I tried and failed ! )

Just copy paste framerateCounter.cs from the root of techcraft4.0 (old project) into a new "profiling" folder and change the "using Newtake.debug" to  using "NewTake.profiling" in game1.cs.

debug* resources are excluded from hg, so nothing should be named debug even folders.

Coordinator
Apr 8, 2011 at 4:24 AM
Edited Apr 8, 2011 at 4:24 AM

I added a single threaded implementation using "your boundary stuff" , and it does 55-60 FPS when generating the world, it can even fly like crazy at 130 FPS when pressing F3 for unlocked timestep and no vsync (and it 's still on my i5 laptop with embedded gpu ).

The current multithreaded implem is still there, its just one line to comment/uncomment in game1.cs .

I leave the code alone for now, you can go crazy with the new concurrent model of csharp4 :)

Coordinator
Apr 8, 2011 at 7:37 AM
Edited Apr 8, 2011 at 9:31 AM

Looks like there's still issues with the boundaries stuff too - i'll have time from this evening to spend on it...would be nice to get this out of the way 'cos it's holding up lots of other cool stuff now :)

edit - I've fixed the chunk renderer - working version is here http://dl.dropbox.com/u/9584488/SolidBoundsChunkRenderer.cs

i'll do the .net4 multithreading this evening.

Coordinator
Apr 8, 2011 at 1:48 PM

cool you ve done the

//TODO always using solidBlock at boundaries is fast ( no world.blockat) but wrong when world.blockat would send something else that blocktype.NONE

And in your code there is still no world.blockat, only few sparse matrix access to get neighbouringChunks ,
 wich we can optimize later if necessary with something like having members Chunk North, South, West, East; in class Chunk.
( and Chunk.Norris of course ;)
Coordinator
Apr 9, 2011 at 4:30 AM

It's 4am and i've got sucked into hacking around with this &$%! engine all night :) I've got multithreaded chunk building pretty much working as best i can - still using DrawUserPrimatives but now with very basic

batching which gave a moderate improvement in performance. The are some rendering issues due to chunks being built on multiple threads but i think i can overcome them with some trickery in the way chunks 

are submitted to be built (it's cheating but it'll probably work). The biggest performance hit now is the WorldRenderer.Update method which with a world size of 27x27 chunks which is visually similar to what

i can see in Minecraft it loops through 729 blocks and causes a small but noticable glitch on the main rendering thread. If we can resolve that then on my laptop at least it seems to perform at least as well as 

Minecraft itself. I tried briefly moving the update onto a separate thread but it caused too many problems that i couldn't be bothered to fix at this time of night..it would probably cause no end of problems too.

There was some discussion on IRC earlier about how this could be improved as is so i think that may be the best way forward. I'll wrap things up and post the code in the morning....sleep beckons!

Coordinator
Apr 9, 2011 at 4:35 AM

cool, its 23:35 here and im already tired. ill commit something cool in the singlethread classes in few minutes, good night !

Coordinator
Apr 9, 2011 at 5:17 AM
Edited Apr 9, 2011 at 5:18 AM

So here is the cool thing : one more singlethread optimization, good for multithread of course :

As we only draw block faces near empty ones ( so we only draw potentially visible blocks ), we can limit the y loop in BuildVertexList  to be only between the lowest and highest empty blocks in chunk .    

These bounds are stored in chunk.highestNoneBlock  and chunk.lowestNoneBlock during the terrain generation ( and i implemented it in digging too as a last minute feature ;) 

 

BuildVertexList was taking 18 ms to run on average

Now, on the current terrain without caves, it runs at ... 4.6 ms average ! ;)

Coordinator
Apr 9, 2011 at 10:24 AM
Edited Apr 9, 2011 at 2:18 PM

Current state of multithreaded madness available here http://dl.dropbox.com/u/9584488/TechCraft_Threadtest.rar

I've tried to stay within the code structure but i did have to do one or two hacks so consider this experimental. Once i've got it the remaining glitches ironed out i'll implement it cleanly (unless anyone wants 

to pull it togeather today - the rendering issues are very minor) - i'm off to the beach since it's a gorgeous day here.

Sounds like your optimization would fit in easily enomi - nice one - we're getting there. The chunk batching code will apply to single threaded easily too.

S.

edit - just tried deploying this implementation to xbox and unfortunately it won't work because the compact framework doesn't support the task.factory stuff. Personally i'm not that bothered at the moment we can do another threading model to support xbox/wp7 at a later date - enomi's code structure mean that it won't be too much of a problem, but i thought i'd post an update to get peoples feedback on things. 

I also just tried FortressCraft on the xbox which has just been officially released on Indie games and while it's visually very cool it is FULL of bugs at the moment still. They do have an awesome effect where chunks rise up into the sky as they load which looks very cool, but the game locks solid for me every 30 seconds or so which kindof spoils all the nice effects which is a shame.

Coordinator
Apr 9, 2011 at 2:44 PM

I just tried your last code on my desktop athlon 6000+ dual core,  GF9800 GT . I see no rendreing issue, but  I only get 30 FPS, even with FlatReferenceTerrain ! 

FPS goes slightly up when pressing F3 to loose the fixedtimestep. Maybe we re doing too much things in draw instead of update.

 ( add  chunk.generated = true; at the end of flatReferenceterrain to make it usable)

BTW I tried fortresscraf too, promising but definitely not a finished product ! Collision detection was disturbing and water kept appearing and disappearing.

but its one of the rare indie games where I  hit the 8 minutes demo time limit ;)

Coordinator
Apr 9, 2011 at 2:56 PM

Yeah - i tried all sorts of things to improve the frameright - If you look in the ChunkBatcher code try swapping the rendering methods - it may be that we can improve things there somehow since it does quite a lot of 

array copies, however from my tests DUP still came out top. I'm still playing around so will keep you updated, can you send me your updated chunk rendering optimizations and i'll try with them too. Might be worth 

getting someone with a decent profiler to profile it and post the results.

Coordinator
Apr 9, 2011 at 3:49 PM

Its checked in. NOthing else to send

On 2011-04-09 9:56 AM, "bamyazi" <notifications@codeplex.com> wrote:
> From: bamyazi
>
> Yeah - i tried all sorts of things to improve the frameright - If you look in the ChunkBatcher code try swapping the rendering methods - it may be that we can improve things there somehow since it does quite a lot of array copies, however from my tests DUP still came out top. I'm still playing around so will keep you updated, can you send me your updated chunk rendering optimizations and i'll try with them too. Might be worth getting someone with a decent profiler to profile it and post the results.
>
>
Coordinator
Apr 9, 2011 at 8:02 PM
Edited Apr 9, 2011 at 8:13 PM

I'm a bit lost now on the multithreaded stuff to be honest - i'm building a clean implementation of all 3 current versions based on your last checkin..and keeping the code structure which i'd hacked up in my experiments

A pure single threaded version 

A queued threaded version 

A multithreaded task based version

so we can compare them properly since you appear to get very different performance than me on your machine. If anyone else can help with the threaded versions please do as i'm really out of my comfort zone on asychronous stuff...Chunk feckin Norris or not :)

Coordinator
Apr 9, 2011 at 9:28 PM

Thats crazy I get 60 fps mostly constant on the i5 laptop ! Your code is laptop optimized by accident !

I suspect the mem copies and DUP are very fast on these kind of laptops and on on the xbox !

This is very interesting cause most games suck on these laptops ( minecraft for example) .