notes on adding vertex skinning to mgrl

vertex_skinning_notes.org

soft body animation, round 2

gist

shower musings:

The infulencing bones as they are calculated for the current frame are stored in a vec4 texture lookup. The lookup is 4px in height and Npx wide, where N is the number of bones in the model. Thus, RGBA is the row, and H in the column.

As for attributes, the model has per vertex the following:

• 8x vec2 array

The 8 limit is arbitrary, but could be increased maybe if testing proves that it is too limiting. For each vec2, vec2.x is the x coordinate in the matrix lookup texture. This should be normalized against the texture’s width at export time such that it is always a value in the range of 0.0 to 1.0. The vec2.y value is the influencing alpha value.

There should then be a uniform bool to determine if the object is a softbody animation.

example

The vertex shader code would work something like this:

uniform mat4 world_matrix;
uniform mat4 view_matrix;
uniform mat4 projection_matrix;
attribute vec3 position;

const int max_vertex_groups = 8;
attribute float softbody_matrix[max_vertex_groups];
attribute float softbody_alpha[max_vertex_groups];
uniform sampler2D softbody_lookup;
uniform bool softbody_animation;

vec3 softbody(float x, float a) {
  if (a == 0.0) {
    return vec3(0.0, 0.0, 0.0);
  }
  else {
    mat4 matrix = mat4(texture2D(softbody_lookup, vec2(x, 0.0)).rgba,
                       texture2D(softbody_lookup, vec2(x, 0.25)).rgba,
                       texture2D(softbody_lookup, vec2(x, 0.5)).rgba,
                       texture2D(softbody_lookup, vec2(x, 1.0)).rgba);
    // not sure if the above is valid lol
    return (matrix * vec4(position, 1.0)).xyz * a;
  }
}
   
void main() {
  vec4 xyzw;
  if (softbody_animation) {
    vec3 tmp = vec3(0.0, 0.0, 0.0);
    for (i=0; i<max_vertex_groups; i+=1) {
      tmp += softbody(softbody_matrix[i], softbody_alpha[i]);
    }
    xyzw = vec4(tmp, 1.0);
  }
  else {
    xyzw = vec4(position, 1.0);
  }
  gl_Position = projection_matrix * view_matrix * world_matrix * xyzw;
}

unknowns

• [ ] can individual attributes be array types?

  answer: no idea! lol! but we can work around it easily if they can’t be.

• [X] can texture lookups happen on the vertex shader? (might need an ext?)

  answer: maybe! According to WebGL Stats, if we limit to common GNU/Linux browsers and Firefox on Android, 96% of users have at least four vertex texture image units are available. For 93%, 16 are available. However, like with other extensions, if we use any other android browser, the supported cases drop to 83%.

  As a result, this probably means that we’ll need to have some kind of capabilities check to determine softbody support.

what happens in the exporter

The exporter should be relatively simple. After generating the armature data (possibly before the vertex data is generated), the matrix bandwidth is now known, and therefor the matrix lookup index attribute is normalized against that so as to be in the 0.0-1.0 range.

The vertex groups are stored lean, so while there is a maximum of 8 per vertex hardcoded into mgrl, the exporter only stores the ones that actually exist in their respective data blocks.

what happens per frame

– model loading –

a. a texture is registered for matrix data to be streamed to. it is a non-power of two texture, and regardless, there should be no interpolation. Additionally, this texture needs to be a floating point rgba texture.

– cpu side –

1. each bone matrix is read and flattened into a singular array. This array is uploaded to a particular texture target.
2. the softbody animation uniform is set true
3. the model’s vbo is bound, causing the vertex group related attributes to be set. the model is drawn.

– gpu side –

1. the vertex shader iterates on the vertex group attributes. if alpha is greater than 0.0, then the four texture lookups are called to re-construct the matrix and the coordinate is transformed, multiplied by alpha, and the added to an accumulator.
2. the accumulator is transformed into screen space and sent to the fragment shader.

rational

The competing approach would have been to put the influencing matricies themselves into uniform vars. This could have been done one of two ways:

1. register 8 of them, and split the object into sub VBOs for differet vertex groups. This could easily approach 50 VBOs per a humanoid character, probably much more. Also would need some creativity working between this really being a per vertex thing but VBOs being per face. Its a huge mess, and in retrospect, lots of VBO calls is probably a really bad idea for performance.
2. determine a reasonable maximum, and register that many. The problem here is two fold. I don’t want to say you can only have N bones in a softbody, and I also don’t want to say that a very large value of N uniform slots are forever and always reserved for a feature that might never be used anyway.

The texture lookup method on the other hand eliminates the problems in number 2 and also allows for non-constant access; so we can actually use a uniform index to perform the lookup, which probably wouldn’t be the case with #2, so that also makes this potentially much faster.

Streaming a small texture per frame should be ok for performance if I understand this all correctly. After that, it is just twiddling two uniform vars and drawing.