Tribes 1 Model Formats.txt

Tribes 1 Model Formats
----------------------

Tribes 1 uses file formats which are similar to file formats used in the Torque Game Engine, with the major difference being more use of tags and versioned objects serialized by class name (working on the same basic principle of class instantiation in torques' console system). Also a lot more emphasis on paletted textures.

Currently this document only covers enough to render .dts, terrain (.dtf,.dtb), and interior (.dig,.dis,.dil) shapes present in Tribes 1. For tribes 2 file formats, your best bet is to check out the earlier torque code.

We'll refer to each field as [type] [name or tag value]. If successive fields need to be at a specific offset indicated by a field, that will be noted with "@nameOffset:" where appropriate. In addition if there is a list of typed data, it will be listed as "type[size]:" followed by the fields of that type.


Common types
------------

* Tag

	uint32 storing a tag, usually followed by the size of the proceeding data.

* ChunkSize

	uint32 where the last bit indicates if the file size should be dword aligned, in which case the amount of data you need to read is ((ChunkSize & ~0x80000000) + 3) & ~3.

* ChunkList

	A successive list of chunks which will be read until a number of chunks have been read. The number of chunks needs to be read somewhere in the second chunk, otherwise processing will be aborted.
	
* Chunk

	A simple tuple of [Tag, ChunkSize] where ChunkSize is the sum of the size of the stored fields.

* Point2I

	int[2] representing a point.

* Point2F

	float[2] representing a point.

* Point3F

	float[3] representing a point.

* Box3F

	float[3] for the min point followed by float[3] for the max point.

* Quat16

	uint16[4] representing a quaternion. To decode you divide each component by 0x7fff.

* QuatF

	float[4] representing a quaternion.

* Mat3F

	float[3][3] representing a matrix in row-major format.

* String

	uint16 length followed by the string data. If the last bit of the length is set, the actual size will be increased for dword padding i.e. the size to read is ((length & ~0x80000000) + 3) & ~3.

* VersionedPersBlock

	A serialized object. Stored as follows:
	
		Tag 'PERS'
		uint32 chunkSize
		String className
		int32 version

* LZHData

	Compressed data encoded via LZ Huffman (see reference implementation by written by Haruyasu Yoshizaki for details)


Volume Format
-------------

Tribes '.vol' files are stored as follows:

	Tag 'PVOL'
	uint32 stringBlockOffset
	for each file:
		Tag 'VBLK'
		ChunkSize encodedSize
@stringBlockOffset:
	Tag 'vols' (@stringBlockOffset)
	ChunkSize volsSize
	char[volsSize] stringData
	Tag 'voli'
	ChunkSize voliSize (should be multiple of entry size)
	for each file entry:
		uint32 id (always 0, not used in tribes)
		int32 filenameOffset (relative to @stringBlockOffset+8)
		int32 fileOffset (offset to relevant VBLK data)
		uint32 size
		uint8 compressType (0 = none, 1 = RLE, 2 = lzss, 3 = lha / LZHData)

File data is typically stored uncompressed, however older Dynamix games like Red Baron II actually use the compression. All compressed data is compressed in chunks of 500 bytes.

For RLE, the following pseudocode should suffice:

	until input consumed:
		read byte
		if byte & 0x80:
			read repeat_byte
			emit repeat_byte byte & ~0x80 times
		else:
			read byte bytes
			emit bytes
	
For LZH, using the standard algorithm without a starting size should work. 

You may find that older archives instead have the following format:

	Tag 'VOL '
	Tag 'volh'
	Tag 'vols'
	uint32 unknown
	char[vols size-4] stringData
	Tag 'voli'
	for each file entry:
		int32 id (filename relative to stringData; -1 == empty)
		uint32 offset (relative to +9)
		uint32 size
		uint8 compressType (0 = none, 1 = RLE, 2 = lzss, 3 = lha / LZHData)
		uint8 padding
	At each file offset:
		tag 'VBLK'
		[file data; uncompressed size = fileEntry.size]


Palette Format
--------------

Tribes makes use of palettes. Normally a mission will load a single '.ppl' file which contains multiple palettes. Earlier games may make use of a simpler palette format with only 1 single palette in them. In addition microsoft palette files can be read.

	Tag 'PL98'
	uint32 numPalettes
	int32 shadeShift  (shadeLevels = 1<<shadeShift)
	int32 hazeLevels
	int32 hazeColor
	uint8[32] allowedMatches (bit vector which marks colors that can be matched against)
	Data[numPalettes]:
		uint32[256] colors
		int32 index (batches up with index entry in bitmap)
		uint32 type (noremap=0, shadehaze=1, translucent=2, colorquant=3, alphaquant=4, additivequant=5, additive=6, subtractivequant=7, subtractive=8)
	[remap data]
	uint32 weightPresent
	if weightPresent != 0:
		float[256] colorWeights
		uint32 weightStart
		uint32 weightEnd
	
Remap data size and content varies depending on the types of palettes used. Each maping table should be associated with the relevant palette data. Unless you are writing a software renderer however, the remap data is largely not useful. But it still needs to skipped past to properly read the palette.

	for each data:
		if type is shadehaze:
			uint8[256 * shadeLevels * hazeLevels] shadeMap
		if type is translucent or additive or subtractive:
			uint8[256 * 256] transMap
	for each data:
		if type is translucent, shadeHaze, additive or subtractive:
			uint8[256] colorIndex
			float[256] colorRed
			float[256] colorGreen
			float[256] colorBlue
	for each data:
		if type is noremap:
			uint8[256] colorIndex
			float[256] colorRed
			float[256] colorGreen
			float[256] colorBlue


Phoenix Bitmap Format
---------------------

Tribes uses its own bitmap format, which is similar to what you would get if you serialized a bitmap in torque... only it's mostly limited to paletted textures. It also supports loading microsoft bmp files.

In the case of microsoft bitmap files tribes will assign a palette index to them from bfReserved2 if `bfReserved1 == 0xf5f7 and bfReserved2 != 0xffff`, in which case the colors from that palette will be used instead of those provided by the bitmap.

Tribes '.bmp' files are serialized as follows:

	Tag 'PBMP'
	uint32 numPalettes
	ChunkList
		if Chunk = 'head':
			uint32 version
			uint32 width
			uint32 height
			uint32 bitDepth (should always be 8)
			uint32 attribute (normal=0x0, transparent=0x1, fuzzy=0x2, translucent=0x4, ownmeme=0x8, additive=0x10, subtractive=0x20, alpha8=0x40)
			[numChunks = version & 0x00ffffff]
			[stride = ((width * bitDepth >> 3)+3)&~3]
		if Chunk = 'DETL':
			int32 mipLevels (maximum = 9)
		if Chunk = 'DATA':
			uint8[Chunk.size] data  (mip0, mip1, mip2, ...)
		If Chunk = 'piDX':
			int32 paletteIndex
		If Chunk = 'RIFF':
			[microsoft palette data]

When loading bitmaps the colors used will be taken from a palette data entry equal to the paletteIndex value. It's also possible for a palette to be present in the bitmap data, in which case that can be used.

If loading bitmap data into OpenGL, you'll want to use the alpha channel if transparent or translucent is set. For transparent images, alpha should be 0 or 255. You'll also want to set the following modes depending on the attribute flags:

	if (transparent)
		glEnable(GL_ALPHA_TEST);
		glAlphaFUnc(GL_GREATER, 0.65f);
	if (additive)
		glBlendFunc(GL_SRC_ALPHA, GL_ONE);
	else if (subtractive)
		glBlendFunc(GL_ZERO, GL_ONE_MINUS_SRC_COLOR);
	else
		glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);


Material Lists
--------------

These are stored either in '.dml' files or at the end of '.dts' files.

	Tag 'PERS'
	uint32 chunkSize
	String className 'TS::MaterialList'
	uint32 numMaterials
	Material[numMaterials]:
		uint32 flags
		float alpha
		uint32 index
		uint8[3] rgb
		if version < 2:
			uint8[16] filename
		else:
			uint8[32] filename
		if version == 1 or 2:
			uint32 type
			float elasticity
			float friction
		if version == 1 or version > 3:
			uint32 useDefaultProps (assumed to be 1 if not present)


DTS Shapes
----------

The format for the root shape object in '.dts' files is as follows:

	Tag 'PERS'
	uint32 chunkSize
	String className 'TS::Shape'
	int32 version
	uint32 numNodes
	uint32 numSequences
	uint32 numSubSequences
	uint32 numKeyframes
	uint32 numTransforms
	uint32 numNames
	uint32 numObjects
	uint32 numDetails
	uint32 numMeshes
	if version >= 2:
		uint32 numTransitions
	if version >= 4:
		uint32 numFrameTriggers
	float radius
	Point3F center
	if version > 7:
		Point3F minBounds  (can be estimated with center + radius and refined with points)
		Point3F maxBounds
	Nodes[numNodes]:
		if version <= 7:
			int32 name
			int32 parentNode
			int32 numSubSequences
			int32 firstSubSequence
			int32 defaultTransform
		else:
			int16 name
			int16 parentNode
			int16 numSubSequences
			int16 firstSubSequence
			int16 defaultTransform
	Sequences[numSequences]:
		if version >= 5:
			int32 name
			int32 cyclic
			float duration
			int32 priority
			int32 firstTriggerFrame
			int32 numTriggerFrames
			int32 numIFLSubSequences
			int32 firstIFLSubSequence
		else if version >= 4:
			int32 name
			int32 cyclic
			float duration
			int32 priority
			int32 firstTriggerFrame
			int32 numTriggerFrames
		else:
			int32 name
			int32 cyclic
			float duration
			int32 priority
	SubSequence[numSubSequences]:
		if version <= 7:
			int32 sequenceIdx
			int32 numKeyFrames
			int32 firstKeyFrame
		else:
			int16 sequenceIdx
			int16 numKeyFrames
			int16 firstKeyFrame
	KeyFrame[numKeyframes]:
		if version < 3:
			float pos
			uint32 key (transform index - keyMask=0x3FFFFFFF, vis=0x80000000, valid=0x40000000)
		else if version <= 7:
			float pos
			uint32 key (transform index or mesh frame)
			uint32 matIndex (matMask=0x0FFFFFFF, vis=0x8000, visMatters=0x40000000, frameMatters=0x10000000, matMatters=0x20000000)
		else:
			float pos
			uint16 key (transform index or mesh frame)
			uint16 matIndex (matMask=0x0FFF, vis=0x8000, visMatters=0x4000, frameMatters=0x1000, matMatters=0x2000)
	Transform[numTransforms]:
		if version < 7:
			QuatF rot
			Point3F pos
			Point3F scale
		else if version == 7:
			Quat16 rot
			Point3F pos
			Point3F scale
		else:
			Quat16 rot
			Point3F pos
	Name[numNames]:
		char data[24]
	Object[numObjects]:
		if version <= 7:
			int16 name
			uint16 flags (0x1 = invisible by default)
			int32 meshIndex
			int16 nodeIndex
			Point3F offset  (relative to node)
			int16 numSubSequences
			int16 firstSubSequence
		else:
			int16 name
			uint16 flags (0x1 = invisible by default)
			int32 meshIndex
			int32 nodeIndex
			uint32 offsetFlags
			Mat3F offsetRot
			Point3F offset  (relative to node)
			int32 numSubSequences
			int32 firstSubSequence
	Detail[numDetails]:
		int32 rootNode
		float size
	if version >= 2:
		Transition[numTransitions]:
			if version < 7:
				int32 startSequence
				int32 endSequence
				float startPosition
				float endPosition
				float duration
				QuatF transformRot
				Point3F transformPos
				Point3F transformScale
			else if version == 7:
				int32 startSequence
				int32 endSequence
				float startPosition
				float endPosition
				float duration
				Quat16 transformRot
				Point3F transformPos
				Point3F transformScale
			else:
				int32 startSequence
				int32 endSequence
				float startPosition
				float endPosition
				float duration
				Quat16 transformRot
				Point3F transformPos
	if version >= 4:
		FrameTrigger[numFrameTriggers]:
			float pos
			int32 value
	if version >= 5:
		int32 defaultMaterials
	if version >= 6:
		int32 alwaysNode
	Mesh[numMeshes]:
		VersionedPersBlock object (always 'TS::CelAnimMesh')
	int32 hasMaterials
	if hasMaterials:
		VersionedPersBlock materialList (always 'TS::MaterialList')


DTS Meshes
----------

Meshes are persisted in the main DTS shape file in the mesh list. The only type of mesh is 'TS::CelAnimMesh', formatted as follows:

	Tag 'PERS'
	uint32 chunkSize
	String className 'TS::CelAnimMesh'
	int32 version
	int32 numVerts
	int32 vertsPerFrame
	int32 numTextureVerts
	int32 numFaces
	int32 frames
	if version >= 2:
		int32 textureVertsPerFrame (for lower versions, should be assumed = numTextureVerts)
	if version < 3:
		Point3F scale  (this was moved into the frame data for versions >= 3)
		Point3F origin
	float radius
	PackedVertex[numVerts]:
		uint8 x
		uint8 y
		uint8 z
		uint8 encodedNormal (index into the encoded normal table, same as the one in torque)
	Point2F[numTextureVerts]
	Face[numFaces]:
		int32 vertIndex0
		int32 texIndex0
		int32 vertIndex1
		int32 texIndex1
		int32 vertIndex2
		int32 texIndex2
		int32 matIndex
	Frame[numFrames]:
		int32 firstVert
		if version >= 3:
			Point3F scale
			Point3F origin

To unpack the vertices, multiply each element by the frame scale and add on the frame origin, or in the case of older shapes, use the scale and origin present in the mesh data. Frame data for meshes is controlled at runtime by keyframe data.


Rendering Shapes
----------------

Tribes will use the "Detail" data and the "alwaysNode" field to determine what to render. It's assumed the first node is always the "bounds" node, and the whole shape is moved by the inverse transform of that node to center the object.

In pseudocode:

	select detail level based on size of projected radius
	renderNode(alwaysNode)
	renderNode(selectedDetail.rootNode)
	
	renderNode(node):
		calculate node transform
		render node objects relative to transform
		renderNode(node.children)

Shapes have a default pose defined by the "defaultTransform" field in each Node which can be used in the absence of sequence data. Node transforms can simply be calculated by accumulating the transforms of the parent nodes, though keep in mind the translation component should be calculated by `parentPos + (parentRot * localPos)`.

All object meshes should be rendered relative to the associated node transform, and offset by the "offset" field in the object.

To incorporate sequence data, you need some sort of thread object which keeps track of the following:

	* Current sequence
	* Playback position
	* Any transitioning state (if transitioning between sequences)
	* Object states (mesh vert frame, mesh texvert frame, visiblity state)
	* Node visibility states

For each animated node and object, there is a list of subsequences associated with each sequence where there is an animation track.

Animation tracks for objects change the mesh vert frame, texture vert frame, and visibility.
Animation tracks for nodes change the node transform and visibility.

For nodes, simply select the closest keyframes and interpolate between them. You'll want to interpolate rotations between keyframes the same way as torque does (refer to the interpolate method on its QuatF class).

For objects, simply change the frames or visibility as you pass the keyframe.

Interiors
---------

Interiors in tribes are split up into 4 files:
	- A ".dis" file which indexes everything required
	- One or more ".dig" files which store geometry
	- One of more ".dil" files which store lighting information
	- One ".dml" file which is a serialized material list

DIS Index
---------

	Tag 'ITRs'
	uint32 chunkSize
	int32 numStates
	State[numStates]:
		uint32 stateNameIdx
		uint32 lodIdx
		uint32 numLods
	int32 numLods
	Lod[numLods]:
		uint32 minPixels
		uint32 geomNameIndx (index into names list)
		uint32 lightStateIdx
		uint32 linkableFaces
	int32 numLodLightStates
	LodLightState[numLodLightStates]:
		uint32 bits
	int32 numLightStates
	LightState[numLightStates]:
		uint32 bits
	int32 nameSize
	char[nameSize] names
	int32 materialListIdx (index into names list)
	bool linkedInterior

Each Lod defines geometry for a particular detail level, shown at the `minPixels` level. This must be loaded from the file indicated by `geomNameIndx` from the same VOL file.

Similarly a filename for a DML file is specified via `materialListIdx`, and a DIL file via `stateNameIdx`


DIG Geometry
------------

	Tag 'PERS'
	uint32 chunkSize
	String className 'ITRGeometry'
	int32 version
	int32 buildId
	float textureScale
	Point3F minBounds
	Point3F maxBounds
	int32 numSurfaces
	int32 numBSPNodes
	int32 numSolidLeafs
	int32 numEmptyLeafs
	int32 numPVSBits
	int32 numVerts
	int32 numPoint3Fs
	int32 numPoint2Fs
	int32 numPlanes
	Surface[numSurfaces]
		uint8 flags
		uint8 materials
		uint8 tsX
		uint8 tsY
		uint8 toX
		uint8 toY
		uint16 planeIdx
		uint32 vertIdx
		uint32 pointIdx
		uint8 numVerts
		uint8 numPoints
	BSPNode[numBSPNodes]
		uint16 planeIdx
		int16 front
		int16 back
		int16 fill
	BSPLeafSolid[numSolidLeafs]
		uint32 surfIdx
		uint32 planeIdx
		uint16 numSurfaces
		uint16 numPlanes
	BSPLeafEmpty[numEmptyLeafs]
		uint16 flags
		uint16 numSurfs
		uint32 pvsIdx
		uint32 planeIdx
		Point3F minBounds
		Point3F maxBounds
		uint16 numPlanes
	uint8[numPVSBits]
	Vertex[numVerts]
		uint16 pIdx
		uint16 tIdx
	Point3F[numPoint3Fs]
	Point2F[numPoint2Fs]
	Plane[numPlanes]
		float x
		float y
		float z
		float d
	int32 highestMip
	uint32 flags


DIL Lighting Info
-----------------

TODO


Rendering Interiors
-------------------

Tribes will render an interior first based on the active `State`, then dependent on the current suitable `Lod` level from the range specified by `lodIdx...(lodIdx+numLods)`.

To render any particular `Surface`, you should use `Vertex` records specified from the range `vertIdx...(vertIdx+numVerts)`. These will specify a `tIdx` textureCoord which should be taken from the `Point2F` list, and a `pIdx` position from the `Point3F` list.

Texture coords should additionally be scaled by the `tsX` and `tsY` values, and offsetted using the `toX` and `toY` values using the following algorithm:

	scale.x = (float)((int)surface.tsX+1) / texture_width
	scale.y = (float)((int)surface.tsY+1) / texture_height
	offset.x = (float)surface.toX / (float)texture_width
	offset.y = (float)surface.toY / (float)texture_height

The surface normal is specified by the `planeIdx` Plane. Finally, the actual material is taken from the associated material list using the `materials` property.

TODO: figure out how lighting info is mapped, and how BSP nodes are evaluated.

Terrain Files
-------------------

Like torque, tribes terrains are stored as heightmaps with grid block square flags. Unlike torque, the actual data is stored in volume files saved by the "ted" program. These have either the ".ted" or ".vol" extension.

DTF : Terrain Index
-------------------

	Tag 'GFIL'
	int32 version
	uint32 mlname_size
	char[mlname_size] mlname (material list filename)
	uint32 last_block_id
	uint32 detail_count
	uint32 scale
	Box3F bounds
	Point2I origin
	float[2] range
	Point2I size
	if version > 0:
		int32 pattern (this relates to the intended block layout)
	int32[size[0] * size[1]] block_map
	uint32 list_size
	BlockEntry[list_size]:
		int32 ident
		uint32 name_size
		char[name_size] name

Unlike torque, tribes stores the heightmap data in one or more block files, which the index 
file references. These are eventually laid out in a grid according to the size values.

The index specifies which material list to use, with the block images being stored in a 
partner ".vol" file.
"pattern" refers to the block pattern type which defines how the blocks are actually laid out. 
This can be 0 (one to all), 1 (unique), 2 (mosaic).

Typically tribes terrains are sized 3x3, with each block mapping to the same entry (0). The actual 
block file is a 256x256 block (which is technically a 257x257 heightmap).

Blocks should be laid out row-major.

To calculate the block size multiplier (scale_shift), use:

	(detail_count - 1) + scale

To calculate the relative block position:

	x = (index % size[0]) << scale_shift
	y = (index / size[0]) << scale_shift


DTB : Terrain Block
-------------------

	Tag 'GBLK'
	int32 version
	char[16] block_ident
	int32 detail_count
	int32 light_scale
	float[2] range
	Point2I size
	if version == 0:
		float[size[0] + 1 * size[1] + 1] height_map
	else if version < 4:
		ROMap height_map
	else:
		uint32 uncompressed_size
		LZHData height_map (compressed form of version 0 height_map)
	if version < 4:
		GridBlock[size[0] * size[1]]:
			uint8 matFlags (similar to torque terrain material flags)
			uint8 matIndex (index of material in material list)
	else:
		uint32 material_map_size
		LZHData material_map (compressed form of version < 4 data)
	if version >= 2:
		PinMapEntry[0...11]:
			uint16 map_size
			char[map_size] pin_map
	if light_scale >= 0
		if version < 4:
			uint16[((size[0] << light_scale) + 1) * ((size[0] << light_scale) + 1)] lightmap
		else:
			uint32 lightmap_size
			LZHData lightmap (compressed form of lightmap)
	[Rest of data relates to the HiRes lightmap which doesn't appear to be used]

The actual data here is similar to that present in torque terrains, except the images for the terrain squares are precomposed and 
referenced by the material list in the index file.

To normalize the height map:

	normalized_height = (value - range[0]) / (range[1] - range[0])

matFlags is of the form:
	
	RXYEEE
	(R = Rotate, X = X Flip, Y = Y Flip, E = Empty level; 0 = no transform)

TODO: Explain what these actually do.

To decompress ROMap data:

	Read first row, float[size[0]+1]
	For row 1...(size[1]):
		Read scale float
		Read leading height float
		Read offset chars[size[0]-1]
		Map each offset:
			result = float(offset) * scale
		Read trailing height float
		(Thus: [leading + mapped offsets + trailing])
		Add resultant row
	Read trailing row, float[size[0]+1]

NOTE: If you have trouble with the LZH data, it's possible to skip it by looking 
for the next appropriate size value in the stream.

height_map data should be sized "size[0] + 1 * size[1] + 1".
material_map data should be sized "size[0] * size[1] * 2".
lightmap data should be sized "((size[0] << light_scale) + 1) * ((size[0] << light_scale) + 1) * 2".


-------------------

And that's it for now. Hopefully this should be enough to get anyone started with prodding around these assets!