nimrod component/entity system

entitty.nim

## entitty has been moved to the fowltek package 

macro_output.nim

Unicast macro result: 
proc die*(entity: var TEntity) {..} = 
  echo "message ID is ", messageID("die")
  echo "vtable is $# len" % $ len(entity.typeInfo.vtable)
  if not entity.typeInfo.vtable[messageID("die")].isNil: 
    cast[proc (entity: var TEntity) {.noConv.}](entity.typeInfo.vtable[
        messageID("die")])(entity)
  
Unicast macro result: 
proc takeDamage*(entity: var TEntity; amount: int) {..} = 
  echo "message ID is ", messageID("takeDamage")
  echo "vtable is $# len" % $ len(entity.typeInfo.vtable)
  if not entity.typeInfo.vtable[messageID("takeDamage")].isNil: 
    cast[proc (entity: var TEntity; amount: int) {.noConv.}](entity.typeInfo.vtable[
        messageID("takeDamage")])(entity, amount)
  
block: 
    let msg_id = MessageID("debugStr")
    let comp = componentInfo(THealth)
    if messageTypes[msg_id]: 
      if comp.multicast_messages.hasKey(msg_id): 
        echo "Overriding the implementation of multicast message `debugStr` for ", 
             comp.name
      comp.multicast_messages[msg_id] = cast[pointer](proc (entity: var TEntity; 
          result: var seq[string]) = 
        result.add "$#: $#".format(ComponentInfo(THealth).name, entity[THealth]))
    else: 
      if comp.unicast_messages.hasKey(msg_id): 
        echo "Overriding implementation of unicast message debugStr for ", 
             comp.name
      comp.unicast_messages[msg_id] = (weight: 0, func: cast[pointer](proc (
          entity: var TEntity; result: var seq[string]) = 
        result.add "$#: $#".format(ComponentInfo(THealth).name, entity[THealth])))
block: 
    let msg_id = MessageID("debugStr")
    let comp = componentInfo(TPos)
    if messageTypes[msg_id]: 
      if comp.multicast_messages.hasKey(msg_id): 
        echo "Overriding the implementation of multicast message `debugStr` for ", 
             comp.name
      comp.multicast_messages[msg_id] = cast[pointer](proc (entity: var TEntity; 
          result: var seq[string]) = 
        result.add "$#: $#".format(ComponentInfo(TPos).name, entity[TPos]))
    else: 
      if comp.unicast_messages.hasKey(msg_id): 
        echo "Overriding implementation of unicast message debugStr for ", 
             comp.name
      comp.unicast_messages[msg_id] = (weight: 0, func: cast[pointer](proc (
          entity: var TEntity; result: var seq[string]) = 
        result.add "$#: $#".format(ComponentInfo(TPos).name, entity[TPos])))
block: 
    let msg_id = MessageID("debugStr")
    let comp = componentInfo(TVel)
    if messageTypes[msg_id]: 
      if comp.multicast_messages.hasKey(msg_id): 
        echo "Overriding the implementation of multicast message `debugStr` for ", 
             comp.name
      comp.multicast_messages[msg_id] = cast[pointer](proc (entity: var TEntity; 
          result: var seq[string]) = 
        result.add "$#: $#".format(ComponentInfo(TVel).name, entity[TVel]))
    else: 
      if comp.unicast_messages.hasKey(msg_id): 
        echo "Overriding implementation of unicast message debugStr for ", 
             comp.name
      comp.unicast_messages[msg_id] = (weight: 0, func: cast[pointer](proc (
          entity: var TEntity; result: var seq[string]) = 
        result.add "$#: $#".format(ComponentInfo(TVel).name, entity[TVel])))
block: 
    let msg_id = MessageID("debugStr")
    let comp = componentInfo(TSpriteInstance)
    if messageTypes[msg_id]: 
      if comp.multicast_messages.hasKey(msg_id): 
        echo "Overriding the implementation of multicast message `debugStr` for ", 
             comp.name
      comp.multicast_messages[msg_id] = cast[pointer](proc (entity: var TEntity; 
          result: var seq[string]) = 
        result.add "$#: $#".format(ComponentInfo(TSpriteInstance).name, 
                                   entity[TSpriteInstance]))
    else: 
      if comp.unicast_messages.hasKey(msg_id): 
        echo "Overriding implementation of unicast message debugStr for ", 
             comp.name
      comp.unicast_messages[msg_id] = (weight: 0, func: cast[pointer](proc (
          entity: var TEntity; result: var seq[string]) = 
        result.add "$#: $#".format(ComponentInfo(TSpriteInstance).name, 
                                   entity[TSpriteInstance])))
block: 
    let msg_id = MessageID("debugStr")
    let comp = componentInfo(TBoundingBox)
    if messageTypes[msg_id]: 
      if comp.multicast_messages.hasKey(msg_id): 
        echo "Overriding the implementation of multicast message `debugStr` for ", 
             comp.name
      comp.multicast_messages[msg_id] = cast[pointer](proc (entity: var TEntity; 
          result: var seq[string]) = 
        result.add "$#: $#".format(ComponentInfo(TBoundingBox).name, 
                                   entity[TBoundingBox]))
    else: 
      if comp.unicast_messages.hasKey(msg_id): 
        echo "Overriding implementation of unicast message debugStr for ", 
             comp.name
      comp.unicast_messages[msg_id] = (weight: 0, func: cast[pointer](proc (
          entity: var TEntity; result: var seq[string]) = 
        result.add "$#: $#".format(ComponentInfo(TBoundingBox).name, 
                                   entity[TBoundingBox])))
block: 
    let msg_id = MessageID("takeDamage")
    let comp = componentInfo(THealth)
    if messageTypes[msg_id]: 
      if comp.multicast_messages.hasKey(msg_id): 
        echo "Overriding the implementation of multicast message `takeDamage` for ", 
             comp.name
      comp.multicast_messages[msg_id] = cast[pointer](proc (entity: var TEntity; 
          amount: int) = 
        entity[THealth].hp -= amount
        if entity[THealth].hp <= 0: 
          entity.die()
        echo "Entity took damage, now at ", entity[THealth].hp)
    else: 
      if comp.unicast_messages.hasKey(msg_id): 
        echo "Overriding implementation of unicast message takeDamage for ", 
             comp.name
      comp.unicast_messages[msg_id] = (weight: 0, func: cast[pointer](proc (
          entity: var TEntity; amount: int) = 
        entity[THealth].hp -= amount
        if entity[THealth].hp <= 0: 
          entity.die()
        echo "Entity took damage, now at ", entity[THealth].hp))
block: 
    let msg_id = MessageID("debugDraw")
    let comp = componentInfo(TPos)
    if messageTypes[msg_id]: 
      if comp.multicast_messages.hasKey(msg_id): 
        echo "Overriding the implementation of multicast message `debugDraw` for ", 
             comp.name
      comp.multicast_messages[msg_id] = cast[pointer](proc (entity: var TEntity; 
          R: PRenderer) = 
        let s = $ entity[TPos]
        let p = entity[TPos]
        R.stringRGBA(p.x.int16, p.y.int16, s, 255, 0, 0, 255)
      )
    else: 
      if comp.unicast_messages.hasKey(msg_id): 
        echo "Overriding implementation of unicast message debugDraw for ", 
             comp.name
      comp.unicast_messages[msg_id] = (weight: 0, func: cast[pointer](proc (
          entity: var TEntity; R: PRenderer) = 
        let s = $ entity[TPos]
        let p = entity[TPos]
        R.stringRGBA(p.x.int16, p.y.int16, s, 255, 0, 0, 255)
      ))

You are getting closer, still several issues:

1. As far as I understand, newComponent is some kind of component registration function that must be called once by the user for each component. You don't need to do this, because this call could be easily produced as a side-effect from instantiating ComponentID(T)

proc addComponent(T: typedesc): int
  result = numComponents
  inc numComponent
  allComponents.add(PComponentInfo(name: T.name, size: sizeof(T), id: id))

proc componentID(T: typedesc): int =
  var id {.global.} = addComponent(T)
  return id

2. use a hashtable for the TypeInfo cache. getTypeInfo should not perform a linear search.

I have a couple questions still

there are 3 types of messages
1) statically dispatched to a certain component:
if has(Component): get(Component).doMessage(args)

This is just calling a function on a component?

2) dynamically dispatched to a single component
let (offset, proc_ptr) = entity->typeinfo->vtable[messageID]
proc_ptr(entity->data[offset], args)

What the utility of this? What kind of message would only one component respond to

3) dynamically dispatched to every component that implements them (multicast messages)
let runlist = entity->typeinfo->multicast_vtable[messageID]
for msg in runlist: msg.proc(entity.data[msg.offset], args)```

proc foo(e: var TEntity, x: string) =
  let runlist = e.typeInfo.multicast[messageID("foo")]
  for msg in runlist:
    cast[proc(e: var TEntity; x: string){.noConv.}](msg.func)(e, x)
    ## imo the message should operate on entities so that different components can be accessed 

I am lost on how/where to store pinned components

type TEntityManager = object
    typeInfosTable: TTable[seq[int], PTypeInfo]
    pinnedComponents: seq[pointer] ## this is a sequence the size of `allComponents` \
      ## in pinned component slots there is a `ptr TTable[KeyType, pointer]` 
      ## The pointer is to the instantiated component 
      ## EntityManager.get(ComponentType, Key) will fetch or try to load 
      ## according to pinnedcomponentinfo.loadImpl \ 

      ## ^ that was my first pass, i realized the flaw in this implementation
      ## but im not sure how to store/load or even instantiate the pinned components
      ## id almost rather just manage heavy things like meshes outside of the 
      ## entity system and have the mesh component be a pointer to your data

The implementation of a message is function defined over a component.
The role of the system is to dispatch the message call to the right component (or set of components).

I indeed missed one important detail in my previous description. In our system, we had a special "keyword" (C++ define) named go_this (GO stands for game object). This keyword can be used inside a component function to get you a pointer to the entity:

void Health::TakeDamage(int damage) {
   m_Healt -= damange;
   HealthChanged(go_this, m_Healt);
}

I guess, with your current set of terms, getEntity(component) will be a good name for the go_this concept.

Here how go_this works:
In the PEntityData blob, the objects are laid out like this:
[[go_this_ptr][Component1][go_this_ptr][Component2][go_this_ptr][Component3]]

so, you can get the go_this pointer, by simply doing something like cast[ptr Entity](offset(addr(component), - sizeof(pointer)))
It's possible to avoid the evil pointer tricks if you just mandate that components types should have a common TComponent base type, but I saw this as an unnecessary step in C++.

What is the purpose of the unicast messages?

I can think of 2 reasons for their existence.

1. A little bit of additional performance.
   Since all of the message usage code looks the same regardless of the message type:
   entity.someMessages(some, args)
   .. it's very easy to change the message type (you just have to modify the single line where the message is declared). This creates an opportunity for a simple guidance for users to always use the cheapest dispatch method until they need the more flexible functionality. Start with static dispatch, if you need multiple polymorphic implementations switch to unicast, if you need multiple handlers, switch to multicast.

2. Messages that naturally don't fit the multicast model.
   There are messages that just give some specific data from an object (e.g. GetPosition). It doesn't make sense for them to be multicast and it will even make more sense that the system will give an error if you indeed try to build an object with 2 components claiming to implement this message. My system had an additional feature allowing the components to define "implementation priority (or implementation bid)" when declaring that they implement certain message. The component with the highest bid wins and gets to implement the message in the EntityType with conflicting set of components. The absence of highest bid was considered an error.

You don't need to store pinned components anywhere. Just allocate them and put pointers to them in the entityData. You don't need to declare them in a special way too. I just used a single generic type TPinned[Health] that is used instead of the regular Heath type when creating objects with pinned Health (Health here is just an example).

Mind you, pinned components are not really needed until you want to support run-time mutation of the entities (adding and removing components) so you are focusing on them a bit too early.

Otherwise, pinned components are special only in the way messages gets dispatched to them. I used a thunk function that was getting inlined most of the time:

# this is quite pseudo-code at the moment, but it can be made to work in general:
proc pinnedDispatcher(msg_name: expr[string], pinned: ptr TPinned[T], args...) =
   `msg_name`(pinned.obj_ptr, args...)

Our Model component was indeed just storing a pointer to a shared Mesh object and some other material data without being pinned (pinning was really more about dealing with memory moves required by entity mutations). This pattern for having shared data between components (or type specific helper types that are stored inside the components) happens all the time.

Multicast messages are usually void, but there are two common patterns for returning values from them.

One easy method is that one of the parameters serve as return value accumulator (e.g. a var sequence that the component implementations will append items to).

When the algorithm for combining the return values must be more complicated, or you want to interrupt the processing of the multicast message before reaching all components, my system allowed you to specify a "combiner" algorithm. It was very similar in mechanism to the combiners in various signals/slots libraries. See boost.signal for example:
http://www.boost.org/doc/libs/1_53_0/doc/html/signals/tutorial.html
(Search for "Signal return values")

Nice progress, fowl. I have to study a bit more carefully what are you doing inside the macros, but I can see certain things already. You don't need to have this kind of branching inside the message dispatchers:
if not entity.typeInfo.vtable[messageID("die")].isNil:

Instead, it's smarted if you populate the vtable with a "default" implementation that raises an error or silently ignores the message. My system also allowed the user to supply his own default implementation for a given message. The default implementation should take an entity as a parameter (instead of component). To achieve this you need to use a trampoline function. Here is how it works:

1. In the vtable, you store a function that uses offset 0.
2. When it's called, it reads the go_this pointer that is inevitably stored at that location and then it just forwards all the parameters to the user supplied default implementation (stored in the EnityManager by MessageID).

I've been busy this week at work, but I'll look at the problems you've reported this weekend (or sooner).

I've noticed you are not a big user of getAst or quote from the macros module. Why so? They are both nicer to look at IMO and much faster to execute by the compiler (because most of the AST crunching happens only once during the semantic analysis instead of every time in "evals").

The same argument for branching in unicast messages applies for multicast as well.

There is no reason why the dispatched code can't be just

for entry in entity.typeInfo.multicastTable[MulticastMessageID("msgname")]:
   cast[proc_type](entity.procptr) (offset_ptr(entity.entityData, entry.offset), args... )

If there is only one implementation, it still will be faster to have a for loop over 1 element array/seq than to have a branching in the dispatcher code to detect that.

I'm a bit inconsistent in my explanations about the vtable and the multicastTable. The reason for this is that in my system there were no such distinction (I was just using C++ unions for the record types). It's easier to explain the system with two tables and it doesn't hurt the performance at all, but please note that if there are two tables, there must be also two MessageID functions: MessageID vs MulticastMessageID