Tell me anything about the project you'd like to me to put in the post.
- when you started Voxelman and why you chose D
- which of D's features have been a real benefit and how
- which features (or bugs) have caused you frustration or difficulties and how
- what your ultimate goals are for the project.
Now, as the project grew up and my skills matured, my motivation has much less oscilations than at the beginning. I wanted to make a project like this for a couple of years (I beleive since I first played Minecraft Classic). I was thinking of cool things I can do and improve, clever optimization tricks and all the low-level stuff. But never written any real code.
And somewhere in 2011-2012 I started the project. I started with creating an SDL window, got some triangles on the screen. Then I did cubes and single chunk. It was a simple singlethreaded thing. It was all done with fixed camera and I had only rudimentary camera controls. It took me quite some time to get camera controls right. It was a tricky mess of matrix multiplications that only sometimes worked. Later I changed it to use quaternions (which may have been be unnessesary).
TODO: wrtie about anchovy library
I dont really remember how I found D. I was in need of some statically typed compiled language, other than C++. I remember trying to do engine in C++, but immediately stuck due to my low level of knoweldge. I was frustrated about all the source file organisation, need of forward declarations, header separation and include system. In D it was as simple as writing code. I buyed cheap 10 inch tablet just to read Andrei's book, because my 3.2" PPC was to small to read the whole thing. I enjoed reading every single bit of it.
This year (2016) I was finally able to simply download ldc and build my project. I was happy to see drastic performance boost. Previously it used older frontend, and later some bugs prented me from using it.
The ultimate goal is to provide a platform where people can easily create plugins, share them, play/create worlds together and share them. Ideally a complete project build should have engine source and tools (launcher, source editor, compiler). Players should be able to initiate connect to any server in server list, then launcher will download missing plugins, compile new executable and start engine with the list of plugins The problem here is compiler redistribution on Windows platform.
- It can be big (not really a problem + it can be auto-downloaded)
- You need winsdk
- You need linker
- You need C standard library that comes with VS
- That leads to dependency to Visual Studio which is unacceptable for regular players.
Currently a build of voxelman is <3MB in size. And I think that this is a good property to have.
- Networking
- World saving and loading
- Infinite world loading/generation
- Multiple world dimensions
- Fast editing
- Multithreaded world gen and mesh gen
- Server-client architecture (both built-in and dedicated server is avaliable)
- Semi-convenient launcher. Can be used to do world management, connect to servers, start dedicated server, compile project with different options, launch game with choosen world.
- Support for plugin packs. A game will load a pack and filter all compiled-in plugins based on a list of plugins in a pack.
- Immutability usage
- meshing
- saving
- editing
- undo/redo (not yet implemented)
Each dimension is broken into chunks. Chunk is 32^3 array of blocks. Each chunk can have a set of data layers (currently blocks and block entities) Each layer is essensially an immutable snapshot. It can be of different storage types (uniform - all blocks are the same, compressed array or full array - layer stores an array of data)
Those layer then can be freely transmitted between thread, with reference counting done in main thread. And when no longer needed they are deleted.
Now, when chunk is received on client side it can be sent to worker thread and geometry will be generated. Snapshots are sent to IO thread when save point occurs, and they can still be used in main thread, sent to client, processed by other worker threads.
One can easily use old snapshot, while several new ones are in use.
Whenever layer is being modified, data is copied into "write buffer", changes are made, and at commit point at the end of frame, all write buffers are committed to chunk storage.
You can see similarities between chunk layer system and entity component systems. They both have entity (entity id vs chunk world position). And each entity has a set of components (components of entities vs layers of chunks)
One of the goals was the ability to perform minimal amount of work on save. Since my chunk data is immutable, I can simply send all modified chunk snapshots to IO thread and they will be saved. Other things still need to be serialized in the main thread.
Currently I have semi-hackish plugins. All plugins inherit from IPlugin interface. Then, each plugin registers itself in shared static constructor in a global table of plugins. Global table has lists for server and client plugins. Then engine adds those plugins to plugin manager based on plugin pack provided.
Plugin manager implements initialization sequence. When starting initialization you have lots of dependencies, so you need to run things in specific order.
- dlib - very solid library by Timur Gafarov. I mostly use it for its great math library. It also has means of working with images, audio, filesystem and other utilities
- derelict-enet - bindings to enet udp networking library
- derelict-glfw3 - bindings to glfw3 library. I use glfw3 for window creation, mouse and keyboard access and opengl context setup
- derelict-gl3 - bindings to OpenGL
- cbor-d - my own implementation of CBOR, which states for Concise Binary Object Representation. It is used for binary serialization in the engine. This is basically an improved msg-pack format.
- imgui - immediate mode gui library. Used with DerelictImgui bindings to cimgui wrapper. Used both in game and in launcher.
- sdlang-d - SDL aka Simple Declarative Language. Used for config files.
- LZ4 - compression library. Very fast.
- LMDB - mmapped nosql db, very fast, small and reliable.
- SQLite - second world db backend. After transition to LMDB used as fallback on linux.
- modules, no forward declarations
- Slices (aka arrays) - atomic part of any program. First thing: they are perfect design, pointer and length are bundled together. On a lazy side of things: they are dynamic. You can append to them, concatenate, change length.
- foreach - 99% of loops in my code are those guys.
- delegates - the idea of storing function pointer together with context pointer is just as essensial as bundling pointer with length into array slice. I was unpleasantly surprised when I saw what C# delegates look like.
- hash maps (aka Associative Arrays)
- templates are beatiful. You simply add another set of parenthesis and you are done.
- GC
- High-level stuff is fully in GC memory. I call something high-level if it has only one instance, so I use interfaces/classes for high-level parts. Low-level things are mostly stack allocated, use structs (which are POD in D), and the most performance sensitive and memory consuming parts use manual memory management (Mallocator). This includes chunk storage and chunk meshes.
- CTFE and codegen, introspection. I have code gen in entity component system that generates iteration queries (similar to SQL select queries), serialization
99% of code uses arrays and built-in hashmaps.
- shared
- Mutex cannot be used as shared
- You need to cast out shared in order to access shared variables from member functions
is() expressionsyntax feels like regular expressions for D templates, very powerful and concise, but hard to understand (espessially more complex forms of it). It was hard to understand what some of those mean. Documentation needs separate section foris()expression, with examples for literally every case.- //Really lots of complicated stuff like dll's, which are quite underpowered and not fully implemented in a sense of class hierarchies.
Of course, nothing will change unless I report a bugzilla issue, do it myself
- Mappings and Registries - The project has lots of things to synch between client, server and data base
- Entities with properties (that can be optional) - I already have this in my ECS implementation as well as in chunk management system. I think such approach can be used for UI widgets.
I started my multithreading by using send/receive functions from std.concurrency.
I found out that I need to send messages variable in length. For example, when loading or saving chunks, you need to send all the layers to another thread. This involved allocation of array for all the layers. Also, shared attribute was involved.
This situation led me to the implementation of lock-free message queue, where each message is just a stream of bytes. You write variables on one end and read them from another. This is obviously single producer, single consumer queue. Other disadvantage was the use of fixed-size circular array. You need to make sure that queue doen't fill up.
This was a point where I found a good book that explains how atomics work. C++ Concurency in Action: Practical Multithreading. This is one of the places in documentation where you feel a lack of pointers on where to find relevant information on specific topic.
So the new solution doesn't require any allocations and is actually faster than built-in one.
Later I added some notification system via Semaphore, so that worker threads wait when out of work.
- Minecraft (now with mods)
- Factorio
- Management like (Dwarf fortress, Gnomoria, Castle story, Stonehearth, Timber and stone, Prison Architect, Rimworld)
- Astroneer