Skip to content

Instantly share code, notes, and snippets.

@awesomez

awesomez/vox_to_obj_AUTO.py

Forked from shivshank/vox_to_obj_exporter.py
Last active February 2, 2023 04:44
Show Gist options
  • Save awesomez/cf9e29392a3fb910970b66a6d639a337 to your computer and use it in GitHub Desktop.
Save awesomez/cf9e29392a3fb910970b66a6d639a337 to your computer and use it in GitHub Desktop.
Download ZIP
Exports from MagicaVoxel VOX to OBJ. Can preserve all edges for easy editing in a program like Blender.
Raw
vox_to_obj_AUTO.py
"""
AUTOMATIC drag and drop support for windows (NO PROMPT!)
1. Copy script to directory you want your files copied to.
2. Select the files you want to convert.
3. Drag & drop onto this script to convert .vox to .obj!
Files will be exported to directory of this script.
automatic mod by awesomedata
This script is designed to export a mass amount of MagicaVoxel .vox files
to .obj. Unlike Magica's internal exporter, this exporter preserves the
voxel vertices for easy manipulating in a 3d modeling program like Blender.
Various meshing algorithms are included (or to be included). MagicaVoxel
uses monotone triangulation (I think). The algorithms that will (or do)
appear in this script will use methods to potentially reduce rendering
artifacts that could be introduced by triangulation of this nature.
I may also include some features like light map generation for easy
importing into Unreal Engine, etc.
Notes:
* There may be a few floating point equality comparisons. They seem to
work but it scares me a little.
* TODO: use constants instead of magic numbers (as defined in AAQuad),
(i.e., ..., 2 -> AAQuad.TOP, ...)
* A lot of assertions should probably be exceptions since they are
error checking user input (this sounds really bad now that I've put
it on paper...). So don't run in optimized mode (who does that
anyways?).
* I am considering adding FBX support.
"""
import math
class AAQuad:
""" A solid colored axis aligned quad. """
normals = [
(-1, 0, 0), # left = 0
(1, 0, 0), # right = 1
(0, 0, 1), # top = 2
(0, 0, -1), # bottom = 3
(0, -1, 0), # front = 4
(0, 1, 0) # back = 5
]
LEFT = 0
RIGHT = 1
TOP = 2
BOTTOM = 3
FRONT = 4
BACK = 5
def __init__(self, verts, uv=None, normal=None):
assert len(verts) == 4, "face must be a quad"
self.vertices = verts
self.uv = uv
self.normal = normal
def __str__(self):
s = []
for i in self.vertices:
s.append( str(i) + '/' + str(self.uv) + '/' + str(self.normal))
return 'f ' + ' '.join(s)
def center(self):
return (
sum(i[0] for i in self.vertices)/4,
sum(i[1] for i in self.vertices)/4,
sum(i[2] for i in self.vertices)/4
)
def bucketHash(faces, origin, maximum, bucket=16):
extents = (
math.ceil((maximum[0] - origin[0])/bucket),
math.ceil((maximum[1] - origin[1])/bucket),
math.ceil((maximum[2] - origin[2])/bucket)
)
buckets = {}
for f in faces:
c = f.center()
# TODO
def optimizedGreedyMesh(faces):
# TODO
edges = adjacencyGraphEdges(faces)
groups = contiguousFaces(faces, edges)
return faces
def adjacencyGraphEdges(faces):
""" Get the list of edges representing adjacent faces. """
# a list of edges, where edges are tuple(face_a, face_b)
edges = []
# build the list of edges in the graph
for root in faces:
for face in faces:
if face is root:
continue
if facesAreAdjacent(root, face):
# the other edge will happen somewhere else in the iteration
# (i.e., the relation isAdjacent is symmetric)
edges.append((root, face))
return edges
def contiguousFaces(faces, adjacencyGraphEdges):
""" Get the list of connected components from a list of graph edges.
The list will contain lists containing the edges within the components.
"""
groups = []
visited = dict((f, False) for f in faces)
for face in faces:
# if the face hasn't been visited, it is not in any found components
if not visited[face]:
g = []
_visitGraphNodes(face, adjacencyGraphEdges, visited, g)
# there is only a new component if face has not been visited yet
groups.append(g)
return groups
def _visitGraphNodes(node, edges, visited, component):
""" Recursive routine used in findGraphComponents """
# visit every component connected to this one
for edge in edges:
# for all x in nodes, (node, x) and (x, node) should be in edges!
# therefore we don't have to check for "edge[1] is node"
if edge[0] is node and not visited[edge[1]]:
assert edge[1] is not node, "(node, node) should not be in edges"
# mark the other node as visited
visited[edge[1]] = True
component.append(edge[1])
# visit all of that nodes connected nodes
_visitGraphNodes(edge[1], edges, visited, component)
def facesAreAdjacent(a, b):
""" Adjacent is defined as same normal, uv, and a shared edge.
This isn't entirely intuitive (i.e., corner faces are not adjacent)
but this definition fits the problem domain.
Only works on AAQuads.
"""
# note: None is == None, this shouldn't matter
if a.uv != b.uv:
return False
if a.normal != b.normal:
return False
# to be adjacent, two faces must share an edge
# use == and not identity in case edge split was used
shared = 0
for vert_a in a.vertices:
for vert_b in b.vertices:
if vert_a == vert_b:
shared += 1
# hooray we have found a shared edge (or a degenerate case...)
if shared == 2:
return True
return False
class GeoFace:
""" An arbitrary geometry face
This should only be used for arbitrary models, not ones we can
reasonably assume are axis aligned.
"""
def __init__(self, verts, uvs=None, normals=None):
self.vertices = verts
assert len(verts) in (3, 4), "only quads and tris are supported"
self.normals = normals
self.uvs = uvs
def toAAQuad(self, skipAssert=False):
q = AAQuad(self.vertices)
if self.normals is not None and len(self.normals) > 0:
if not skipAssert:
for i in self.normals:
assert self.normals[0] == i, \
"face must be axis aligned (orthogonal normals)"
q.normal = self.normals[0]
if self.uvs is not None and len(self.uvs) > 0:
if not skipAssert:
for i in self.uvs:
assert self.uvs[0] == i, \
"face must be axis aligned (orthogonal)"
q.uv = self.uvs[0]
return q
class VoxelStruct:
""" Describes a voxel object
"""
def __init__(self):
# a dict is probably the best way to go about this
# (as a trade off between performance and code complexity)
# see _index for the indexing method
self.voxels = {}
self.colorIndices = set()
def fromList(self, voxels):
self.voxels = {}
for voxel in voxels:
self.setVoxel(voxel)
self.colorIndices.add(voxel.colorIndex)
def setVoxel(self, voxel):
self.voxels[voxel.z*(256**2) + voxel.y * 256 + voxel.x] = voxel
def getVoxel(self, x, y, z):
return self.voxels.get(z*(256**2) + y * 256 + x, None)
def _index(self, x, y, z):
return z*(256**2) + y * 256 + x
def getBounds(self):
origin = (float("inf"), float("inf"), float("inf"))
maximum = (float("-inf"), float("-inf"), float("-inf"))
for key, voxel in self.voxels.items():
origin = (
min(origin[0], voxel.x),
min(origin[1], voxel.y),
min(origin[2], voxel.z)
)
maximum = (
max(maximum[0], voxel.x),
max(maximum[1], voxel.y),
max(maximum[2], voxel.z)
)
return origin, maximum
def zeroOrigin(self):
""" Translate the model so that it's origin is at 0, 0, 0 """
origin, maximum = self.getBounds()
result = {}
xOff, yOff, zOff = origin
for key, voxel in self.voxels.iteritems():
result[self._index(voxel.x-xOff, voxel.y-yOff, voxel.z-zOff)] = \
Voxel(voxel.x-xOff, voxel.y-yOff, voxel.z-zOff,
voxel.colorIndex)
self.voxels = result
return (0, 0, 0), (maximum[0] - xOff,
maximum[1] - yOff,
maximum[2] - zOff)
def toQuads(self):
""" --> a list of AAQuads """
faces = []
for key, voxel in self.voxels.items():
self._getObjFaces(voxel, faces)
return faces
def _getObjFaces(self, voxel, outFaces):
if voxel.colorIndex == 0:
# do nothing if this is an empty voxel
# n.b., I do not know if this ever can happen.
return []
sides = self._objExposed(voxel)
if sides[0]:
f = self._getLeftSide(voxel)
self._getObjFacesSupport(0, voxel.colorIndex, f, outFaces)
if sides[1]:
f = self._getRightSide(voxel)
self._getObjFacesSupport(1, voxel.colorIndex, f, outFaces)
if sides[2]:
f = self._getTopSide(voxel)
self._getObjFacesSupport(2, voxel.colorIndex, f, outFaces)
if sides[3]:
f = self._getBottomSide(voxel)
self._getObjFacesSupport(3, voxel.colorIndex, f, outFaces)
if sides[4]:
f = self._getFrontSide(voxel)
self._getObjFacesSupport(4, voxel.colorIndex, f, outFaces)
if sides[5]:
f = self._getBackSide(voxel)
self._getObjFacesSupport(5, voxel.colorIndex, f, outFaces)
return
n = AAQuad.normals[i]
# note: texcoords are based on MagicaVoxel's texturing scheme!
# meaning a color index of 0 translates to pixel[255]
# and color index [1:256] -> pixel[0:255]
u = ((voxel.colorIndex - 1)/256 + 1/512, 0.5)
outFaces.append(
# this is most definitely not "fun"
AAQuad(f, u, n)
)
def _getObjFacesSupport(self, side, color, faces, outFaces):
n = AAQuad.normals[side]
# note: texcoords are based on MagicaVoxel's texturing scheme!
# meaning a color index of 0 translates to pixel[255]
# and color index [1:256] -> pixel[0:255]
u = ((color - 1)/256 + 1/512, 0.5)
outFaces.append(
# fact: the parameters were coincidentally "f, u, n" at one point!
AAQuad(faces, u, n)
)
# MagicaVoxel does -.5 to +.5 for each cube, we'll do 0.0 to 1.0 ;)
def _getLeftSide(self, voxel):
return [
(voxel.x, voxel.y + 1, voxel.z + 1),
(voxel.x, voxel.y + 1, voxel.z),
(voxel.x, voxel.y, voxel.z),
(voxel.x, voxel.y, voxel.z + 1)
]
def _getRightSide(self, voxel):
return (
(voxel.x + 1, voxel.y, voxel.z + 1),
(voxel.x + 1, voxel.y, voxel.z),
(voxel.x + 1, voxel.y + 1, voxel.z),
(voxel.x + 1, voxel.y + 1, voxel.z + 1)
)
def _getTopSide(self, voxel):
return (
(voxel.x, voxel.y + 1, voxel.z + 1),
(voxel.x, voxel.y, voxel.z + 1),
(voxel.x + 1, voxel.y, voxel.z + 1),
(voxel.x + 1, voxel.y + 1, voxel.z + 1)
)
def _getBottomSide(self, voxel):
return (
(voxel.x, voxel.y, voxel.z),
(voxel.x, voxel.y + 1, voxel.z),
(voxel.x + 1, voxel.y + 1, voxel.z),
(voxel.x + 1, voxel.y, voxel.z)
)
def _getFrontSide(self, voxel):
return (
(voxel.x, voxel.y, voxel.z + 1),
(voxel.x, voxel.y, voxel.z),
(voxel.x + 1, voxel.y, voxel.z),
(voxel.x + 1, voxel.y, voxel.z + 1)
)
def _getBackSide(self, voxel):
return (
(voxel.x + 1, voxel.y + 1, voxel.z + 1),
(voxel.x + 1, voxel.y + 1, voxel.z),
(voxel.x, voxel.y + 1, voxel.z),
(voxel.x, voxel.y + 1, voxel.z + 1)
)
def _objExposed(self, voxel):
""" --> a set of [0, 6) representing which voxel faces are shown
for the meaning of 0-5, see AAQuad.normals
get the sick truth about these voxels' dirty secrets...
"""
# check left 0
side = self.getVoxel(voxel.x - 1, voxel.y, voxel.z)
s0 = side is None or side.colorIndex == 0
# check right 1
side = self.getVoxel(voxel.x + 1, voxel.y, voxel.z)
s1 = side is None or side.colorIndex == 0
# check top 2
side = self.getVoxel(voxel.x, voxel.y, voxel.z + 1)
s2 = side is None or side.colorIndex == 0
# check bottom 3
side = self.getVoxel(voxel.x, voxel.y, voxel.z - 1)
s3 = side is None or side.colorIndex == 0
# check front 4
side = self.getVoxel(voxel.x, voxel.y - 1, voxel.z)
s4 = side is None or side.colorIndex == 0
# check back 5
side = self.getVoxel(voxel.x, voxel.y + 1, voxel.z)
s5 = side is None or side.colorIndex == 0
return s0, s1, s2, s3, s4, s5
class Voxel:
def __init__(self, x, y, z, colorIndex):
self.x = x
self.y = y
self.z = z
self.colorIndex = colorIndex
def genNormals(self, aaQuads, overwrite=False):
# compute CCW normal if it doesn't exist
for face in aaQuads:
if overwrite or face.normal is None:
side_a = (face.vertices[1][0] - face.vertices[0][0],
face.vertices[1][1] - face.vertices[0][1],
face.vertices[1][2] - face.vertices[0][2])
side_b = (face.vertices[-1][0] - face.vertices[0][0],
face.vertices[-1][1] - face.vertices[0][1],
face.vertices[-1][2] - face.vertices[0][2])
# compute the cross product
face.normal = (side_a[1]*side_b[2] - side_a[2]*side_b[1],
side_a[2]*side_b[0] - side_a[0]*side_b[2],
side_a[0]*side_b[1] - side_a[1]*side_b[0])
def importObj(stream):
vertices = []
faces = []
uvs = []
normals = []
for line in stream:
# make sure there's no new line or trailing spaces
l = line.strip().split(' ')
lineType = l[0].strip()
data = l[1:]
if lineType == 'v':
# vertex
v = tuple(map(float, data))
vertices.append(v)
elif lineType == 'vt':
# uv
uvs.append( tuple(map(float, data)) )
elif lineType == 'vn':
# normal
normals.append( tuple(map(float, data)) )
elif lineType == 'f':
# face (assume all verts/uvs/normals have been processed)
faceVerts = []
faceUvs = []
faceNormals = []
for v in data:
result = v.split('/')
print(result)
# recall that everything is 1 indexed...
faceVerts.append(vertices[int(result[0]) - 1])
if len(result) == 1:
continue # there is only a vertex index
if result[1] != '':
# uvs may not be present, ex: 'f vert//normal ...'
faceUvs.append(uvs[int(result[1]) - 1])
if len(result) <= 2:
# don't continue if only vert and uv are present
continue
faceNormals.append(normals[int(result[2]) - 1])
faces.append( GeoFace(faceVerts, faceUvs, faceNormals) )
else:
# there could be material specs, smoothing, or comments... ignore!
pass
return faces
def exportObj(stream, aaQuads):
# gather some of the needed information
faces = aaQuads
# copy the normals from AAQuad (99% of cases will use all directions)
normals = list(AAQuad.normals)
uvs = set()
for f in faces:
if f.uv is not None:
uvs.add(f.uv)
# convert this to a list because we need to get their index later
uvs = list(uvs)
# we will build a list of vertices as we go and then write everything
# in bulk, disadvantage that MANY verts will be duplicated in the OBJ file
fLines = []
vertices = []
indexOffset = 0
for f in faces:
# recall that OBJ files are 1 indexed
n = 1 + normals.index(f.normal) if f.normal is not None else ''
uv = 1 + uvs.index(f.uv) if f.uv is not None else ''
# this used to be a one liner ;)
fLine = ['f']
for i, vert in enumerate(f.vertices):
# for each vertex of this face
v = 1 + indexOffset + f.vertices.index(vert)
fLine.append(str(v) + '/' + str(uv) + '/' + str(n))
vertices.extend(f.vertices)
indexOffset += len(f.vertices)
fLines.append(' '.join(fLine) + '\n')
# write to the file
stream.write('# shivshank\'s .obj optimizer\n')
stream.write('\n')
if len(normals) > 0:
stream.write('# normals\n')
for n in normals:
stream.write('vn ' + ' '.join(list(map(str, n))) + '\n')
stream.write('\n')
if len(uvs) > 0:
stream.write('# texcoords\n')
for i in uvs:
stream.write('vt ' + ' '.join(list(map(str, i))) + '\n')
stream.write('\n')
# output the vertices and faces
stream.write('# verts\n')
for v in vertices:
stream.write('v ' + ' '.join(list(map(str, v))) + '\n')
stream.write('\n')
stream.write('# faces\n')
for i in fLines:
stream.write(i)
stream.write('\n')
stream.write('\n')
return len(vertices), len(fLines)
def importVox(file):
""" --> a VoxelStruct from this .vox file stream """
# in theory this could elegantly be many functions and classes
# but this is such a simple file format...
# refactor: ? should probably find a better exception type than value error
vox = VoxelStruct()
magic = file.read(4)
if magic != b'VOX ':
print('magic number is', magic)
if userAborts('This does not appear to be a VOX file. Abort?'):
raise ValueError("Invalid magic number")
# the file appears to use little endian consistent with RIFF
version = int.from_bytes(file.read(4), byteorder='little')
if version != 150:
if userAborts('Only version 150 is supported; this file: '
+ str(version) + '. Abort?'):
raise ValueError("Invalid file version")
mainHeader = _readChunkHeader(file)
if mainHeader['id'] != b'MAIN':
print('chunk id:', mainId)
if userAborts('Did not find the main chunk. Abort?'):
raise ValueError("Did not find main VOX chunk. ")
#assert mainHeader['size'] == 0, "main chunk should have size 0"
# we don't need anything from the size or palette header!
# : we can figure out (minimum) bounds later from the voxel data
# : we only need UVs from voxel data; user can export palette elsewhere
nextHeader = _readChunkHeader(file)
while nextHeader['id'] != b'XYZI':
# skip the contents of this header and its children, read the next one
file.read(nextHeader['size'] + nextHeader['childrenSize'])
nextHeader = _readChunkHeader(file)
voxelHeader = nextHeader
assert voxelHeader['id'] == b'XYZI', 'this should be literally impossible'
assert voxelHeader['childrenSize'] == 0, 'why voxel chunk have children?'
seekPos = file.tell()
totalVoxels = int.from_bytes(file.read(4), byteorder='little')
### READ THE VOXELS ###
for i in range(totalVoxels):
# n.b., byte order should be irrelevant since these are all 1 byte
x = int.from_bytes(file.read(1), byteorder='little')
y = int.from_bytes(file.read(1), byteorder='little')
z = int.from_bytes(file.read(1), byteorder='little')
color = int.from_bytes(file.read(1), byteorder='little')
vox.setVoxel(Voxel(x, y, z, color))
# assert that we've read the entire voxel chunk
assert file.tell() - seekPos == voxelHeader['size']
# (there may be more chunks after this but we don't need them!)
#print('\tdone reading voxel data;', totalVoxels , 'voxels read ;D')
return vox
def _readChunkHeader(buffer):
id = buffer.read(4)
if id == b'':
raise ValueError("Unexpected EOF, expected chunk header")
size = int.from_bytes(buffer.read(4), byteorder='little')
childrenSize = int.from_bytes(buffer.read(4), byteorder='little')
return {
'id': id, 'size': size, 'childrenSize': childrenSize
}
def userAborts(msg):
print(msg + ' (y/n)')
u = input()
if u.startswith('n'):
return False
return True
def exportAll():
""" Uses a file to automatically export a bunch of files!
See this function for details on the what the file looks like.
"""
import os, os.path
with open('exporter.txt', mode='r') as file:
# use this as a file "spec"
fromSource = os.path.abspath(file.readline().strip())
toExportDir = os.path.abspath(file.readline().strip())
optimizing = file.readline()
if optimizing.lower() == 'true':
optimizing = True
else:
optimizing = False
print('exporting vox files under', fromSource)
print('\tto directory', toExportDir)
print('\toptimizing?', optimizing)
print()
# export EVERYTHING (.vox) walking the directory structure
for p, dirList, fileList in os.walk(fromSource):
pathDiff = os.path.relpath(p, start=fromSource)
outDir = os.path.join(toExportDir, pathDiff)
# REFACTOR: the loop should be moved to a function
for fileName in fileList:
# only take vox files
if os.path.splitext(fileName)[1] != '.vox':
print('ignored', fileName)
continue
print('exporting', fileName)
# read/import the voxel file
with open(os.path.join(p, fileName), mode='rb') as file:
try:
vox = importVox(file)
except ValueError as exc:
print('aborted', fileName, str(exc))
continue
# mirror the directory structure in the export folder
if not os.path.exists(outDir):
os.makedirs(outDir)
print('\tcreated directory', outDir)
# export a non-optimized version
objName = os.path.splitext(fileName)[0]
rawQuads = vox.toQuads()
with open(os.path.join(outDir, objName + '.obj'), mode='w') as file:
vCount, qCount = exportObj(file, rawQuads)
print('\texported', vCount, 'vertices,', qCount, 'quads')
if optimizing:
# TODO
continue
optiFaces = optimizedGreedyMesh(rawQuads)
bucketHash(optiFaces, *vox.getBounds())
with open(os.path.join(outDir, objName + '.greedy.obj'),
mode='w') as file:
exportObj(file, optiFaces)
def byPrompt():
import os, os.path, sys
from glob import glob
#### set output directory to script file location
# ------------------------------------------------
####
u = os.path.abspath(sys.argv[0]).strip(os.path.basename(sys.argv[0]))
print(u)
#### drag & dropped files
# ---------------------
for i in sys.argv:
if i != sys.argv[0]:
print(i)
#### fully manual prompt ####
# -------------------
# print('Enter an output path:')
# u = input('> ').strip()
while not os.path.exists(u):
print('That path does not exist.')
print('Enter an output path:')
u = input('> ').strip()
outRoot = os.path.abspath(u)
try:
#while True:
#### grab files from prompt (uncomment lines below if needed)
# ----------------------
#print('Enter glob of export files (\'exit\' or blank to quit):')
#u = input('> ').strip()
#if u == 'exit' or u == '':
# break
#u = glob(u)
#### grab drag & dropped files
u = sys.argv
for f in u:
if f != sys.argv[0]:
print('reading VOX file', f)
with open(f, mode='rb') as file:
try:
vox = importVox(file)
except ValueError:
print('\tfile reading aborted')
continue
outFile = os.path.splitext(os.path.basename(f))[0]
outPath = os.path.join(outRoot, outFile+'.obj')
print('exporting VOX to OBJ at path', outPath)
with open(outPath, mode='w') as file:
exportObj(file, vox.toQuads())
except KeyboardInterrupt:
pass
if __name__ == "__main__":
profiling = False
try:
import cProfile
if profiling:
cProfile.run('exportAll()', sort='tottime')
else:
exportAll()
except OSError:
print('No instruction file found, falling back to prompt.')
byPrompt()
Raw
vox_to_obj_exporter.py
"""
This script is designed to export a mass amount of MagicaVoxel .vox files
to .obj. Unlike Magica's internal exporter, this exporter preserves the
voxel vertices for easy manipulating in a 3d modeling program like Blender.
Various meshing algorithms are included (or to be included). MagicaVoxel
uses monotone triangulation (I think). The algorithms that will (or do)
appear in this script will use methods to potentially reduce rendering
artifacts that could be introduced by triangulation of this nature.
I may also include some features like light map generation for easy
importing into Unreal Engine, etc.
Notes:
* There may be a few floating point equality comparisons. They seem to
work but it scares me a little.
* TODO: use constants instead of magic numbers (as defined in AAQuad),
(i.e., ..., 2 -> AAQuad.TOP, ...)
* A lot of assertions should probably be exceptions since they are
error checking user input (this sounds really bad now that I've put
it on paper...). So don't run in optimized mode (who does that
anyways?).
* I am considering adding FBX support.
"""
import math
class AAQuad:
""" A solid colored axis aligned quad. """
normals = [
(-1, 0, 0), # left = 0
(1, 0, 0), # right = 1
(0, 0, 1), # top = 2
(0, 0, -1), # bottom = 3
(0, -1, 0), # front = 4
(0, 1, 0) # back = 5
]
LEFT = 0
RIGHT = 1
TOP = 2
BOTTOM = 3
FRONT = 4
BACK = 5
def __init__(self, verts, uv=None, normal=None):
assert len(verts) == 4, "face must be a quad"
self.vertices = verts
self.uv = uv
self.normal = normal
def __str__(self):
s = []
for i in self.vertices:
s.append( str(i) + '/' + str(self.uv) + '/' + str(self.normal))
return 'f ' + ' '.join(s)
def center(self):
return (
sum(i[0] for i in self.vertices)/4,
sum(i[1] for i in self.vertices)/4,
sum(i[2] for i in self.vertices)/4
)
def bucketHash(faces, origin, maximum, bucket=16):
extents = (
math.ceil((maximum[0] - origin[0])/bucket),
math.ceil((maximum[1] - origin[1])/bucket),
math.ceil((maximum[2] - origin[2])/bucket)
)
buckets = {}
for f in faces:
c = f.center()
# TODO
def optimizedGreedyMesh(faces):
# TODO
edges = adjacencyGraphEdges(faces)
groups = contiguousFaces(faces, edges)
return faces
def adjacencyGraphEdges(faces):
""" Get the list of edges representing adjacent faces. """
# a list of edges, where edges are tuple(face_a, face_b)
edges = []
# build the list of edges in the graph
for root in faces:
for face in faces:
if face is root:
continue
if facesAreAdjacent(root, face):
# the other edge will happen somewhere else in the iteration
# (i.e., the relation isAdjacent is symmetric)
edges.append((root, face))
return edges
def contiguousFaces(faces, adjacencyGraphEdges):
""" Get the list of connected components from a list of graph edges.
The list will contain lists containing the edges within the components.
"""
groups = []
visited = dict((f, False) for f in faces)
for face in faces:
# if the face hasn't been visited, it is not in any found components
if not visited[face]:
g = []
_visitGraphNodes(face, adjacencyGraphEdges, visited, g)
# there is only a new component if face has not been visited yet
groups.append(g)
return groups
def _visitGraphNodes(node, edges, visited, component):
""" Recursive routine used in findGraphComponents """
# visit every component connected to this one
for edge in edges:
# for all x in nodes, (node, x) and (x, node) should be in edges!
# therefore we don't have to check for "edge[1] is node"
if edge[0] is node and not visited[edge[1]]:
assert edge[1] is not node, "(node, node) should not be in edges"
# mark the other node as visited
visited[edge[1]] = True
component.append(edge[1])
# visit all of that nodes connected nodes
_visitGraphNodes(edge[1], edges, visited, component)
def facesAreAdjacent(a, b):
""" Adjacent is defined as same normal, uv, and a shared edge.
This isn't entirely intuitive (i.e., corner faces are not adjacent)
but this definition fits the problem domain.
Only works on AAQuads.
"""
# note: None is == None, this shouldn't matter
if a.uv != b.uv:
return False
if a.normal != b.normal:
return False
# to be adjacent, two faces must share an edge
# use == and not identity in case edge split was used
shared = 0
for vert_a in a.vertices:
for vert_b in b.vertices:
if vert_a == vert_b:
shared += 1
# hooray we have found a shared edge (or a degenerate case...)
if shared == 2:
return True
return False
class GeoFace:
""" An arbitrary geometry face
This should only be used for arbitrary models, not ones we can
reasonably assume are axis aligned.
"""
def __init__(self, verts, uvs=None, normals=None):
self.vertices = verts
assert len(verts) in (3, 4), "only quads and tris are supported"
self.normals = normals
self.uvs = uvs
def toAAQuad(self, skipAssert=False):
q = AAQuad(self.vertices)
if self.normals is not None and len(self.normals) > 0:
if not skipAssert:
for i in self.normals:
assert self.normals[0] == i, \
"face must be axis aligned (orthogonal normals)"
q.normal = self.normals[0]
if self.uvs is not None and len(self.uvs) > 0:
if not skipAssert:
for i in self.uvs:
assert self.uvs[0] == i, \
"face must be axis aligned (orthogonal)"
q.uv = self.uvs[0]
return q
class VoxelStruct:
""" Describes a voxel object
"""
def __init__(self):
# a dict is probably the best way to go about this
# (as a trade off between performance and code complexity)
# see _index for the indexing method
self.voxels = {}
self.colorIndices = set()
def fromList(self, voxels):
self.voxels = {}
for voxel in voxels:
self.setVoxel(voxel)
self.colorIndices.add(voxel.colorIndex)
def setVoxel(self, voxel):
self.voxels[voxel.z*(256**2) + voxel.y * 256 + voxel.x] = voxel
def getVoxel(self, x, y, z):
return self.voxels.get(z*(256**2) + y * 256 + x, None)
def _index(self, x, y, z):
return z*(256**2) + y * 256 + x
def getBounds(self):
origin = (float("inf"), float("inf"), float("inf"))
maximum = (float("-inf"), float("-inf"), float("-inf"))
for key, voxel in self.voxels.items():
origin = (
min(origin[0], voxel.x),
min(origin[1], voxel.y),
min(origin[2], voxel.z)
)
maximum = (
max(maximum[0], voxel.x),
max(maximum[1], voxel.y),
max(maximum[2], voxel.z)
)
return origin, maximum
def zeroOrigin(self):
""" Translate the model so that it's origin is at 0, 0, 0 """
origin, maximum = self.getBounds()
result = {}
xOff, yOff, zOff = origin
for key, voxel in self.voxels.iteritems():
result[self._index(voxel.x-xOff, voxel.y-yOff, voxel.z-zOff)] = \
Voxel(voxel.x-xOff, voxel.y-yOff, voxel.z-zOff,
voxel.colorIndex)
self.voxels = result
return (0, 0, 0), (maximum[0] - xOff,
maximum[1] - yOff,
maximum[2] - zOff)
def toQuads(self):
""" --> a list of AAQuads """
faces = []
for key, voxel in self.voxels.items():
self._getObjFaces(voxel, faces)
return faces
def _getObjFaces(self, voxel, outFaces):
if voxel.colorIndex == 0:
# do nothing if this is an empty voxel
# n.b., I do not know if this ever can happen.
return []
sides = self._objExposed(voxel)
if sides[0]:
f = self._getLeftSide(voxel)
self._getObjFacesSupport(0, voxel.colorIndex, f, outFaces)
if sides[1]:
f = self._getRightSide(voxel)
self._getObjFacesSupport(1, voxel.colorIndex, f, outFaces)
if sides[2]:
f = self._getTopSide(voxel)
self._getObjFacesSupport(2, voxel.colorIndex, f, outFaces)
if sides[3]:
f = self._getBottomSide(voxel)
self._getObjFacesSupport(3, voxel.colorIndex, f, outFaces)
if sides[4]:
f = self._getFrontSide(voxel)
self._getObjFacesSupport(4, voxel.colorIndex, f, outFaces)
if sides[5]:
f = self._getBackSide(voxel)
self._getObjFacesSupport(5, voxel.colorIndex, f, outFaces)
return
n = AAQuad.normals[i]
# note: texcoords are based on MagicaVoxel's texturing scheme!
# meaning a color index of 0 translates to pixel[255]
# and color index [1:256] -> pixel[0:255]
u = ((voxel.colorIndex - 1)/256 + 1/512, 0.5)
outFaces.append(
# this is most definitely not "fun"
AAQuad(f, u, n)
)
def _getObjFacesSupport(self, side, color, faces, outFaces):
n = AAQuad.normals[side]
# note: texcoords are based on MagicaVoxel's texturing scheme!
# meaning a color index of 0 translates to pixel[255]
# and color index [1:256] -> pixel[0:255]
u = ((color - 1)/256 + 1/512, 0.5)
outFaces.append(
# fact: the parameters were coincidentally "f, u, n" at one point!
AAQuad(faces, u, n)
)
# MagicaVoxel does -.5 to +.5 for each cube, we'll do 0.0 to 1.0 ;)
def _getLeftSide(self, voxel):
return [
(voxel.x, voxel.y + 1, voxel.z + 1),
(voxel.x, voxel.y + 1, voxel.z),
(voxel.x, voxel.y, voxel.z),
(voxel.x, voxel.y, voxel.z + 1)
]
def _getRightSide(self, voxel):
return (
(voxel.x + 1, voxel.y, voxel.z + 1),
(voxel.x + 1, voxel.y, voxel.z),
(voxel.x + 1, voxel.y + 1, voxel.z),
(voxel.x + 1, voxel.y + 1, voxel.z + 1)
)
def _getTopSide(self, voxel):
return (
(voxel.x, voxel.y + 1, voxel.z + 1),
(voxel.x, voxel.y, voxel.z + 1),
(voxel.x + 1, voxel.y, voxel.z + 1),
(voxel.x + 1, voxel.y + 1, voxel.z + 1)
)
def _getBottomSide(self, voxel):
return (
(voxel.x, voxel.y, voxel.z),
(voxel.x, voxel.y + 1, voxel.z),
(voxel.x + 1, voxel.y + 1, voxel.z),
(voxel.x + 1, voxel.y, voxel.z)
)
def _getFrontSide(self, voxel):
return (
(voxel.x, voxel.y, voxel.z + 1),
(voxel.x, voxel.y, voxel.z),
(voxel.x + 1, voxel.y, voxel.z),
(voxel.x + 1, voxel.y, voxel.z + 1)
)
def _getBackSide(self, voxel):
return (
(voxel.x + 1, voxel.y + 1, voxel.z + 1),
(voxel.x + 1, voxel.y + 1, voxel.z),
(voxel.x, voxel.y + 1, voxel.z),
(voxel.x, voxel.y + 1, voxel.z + 1)
)
def _objExposed(self, voxel):
""" --> a set of [0, 6) representing which voxel faces are shown
for the meaning of 0-5, see AAQuad.normals
get the sick truth about these voxels' dirty secrets...
"""
# check left 0
side = self.getVoxel(voxel.x - 1, voxel.y, voxel.z)
s0 = side is None or side.colorIndex == 0
# check right 1
side = self.getVoxel(voxel.x + 1, voxel.y, voxel.z)
s1 = side is None or side.colorIndex == 0
# check top 2
side = self.getVoxel(voxel.x, voxel.y, voxel.z + 1)
s2 = side is None or side.colorIndex == 0
# check bottom 3
side = self.getVoxel(voxel.x, voxel.y, voxel.z - 1)
s3 = side is None or side.colorIndex == 0
# check front 4
side = self.getVoxel(voxel.x, voxel.y - 1, voxel.z)
s4 = side is None or side.colorIndex == 0
# check back 5
side = self.getVoxel(voxel.x, voxel.y + 1, voxel.z)
s5 = side is None or side.colorIndex == 0
return s0, s1, s2, s3, s4, s5
class Voxel:
def __init__(self, x, y, z, colorIndex):
self.x = x
self.y = y
self.z = z
self.colorIndex = colorIndex
def genNormals(self, aaQuads, overwrite=False):
# compute CCW normal if it doesn't exist
for face in aaQuads:
if overwrite or face.normal is None:
side_a = (face.vertices[1][0] - face.vertices[0][0],
face.vertices[1][1] - face.vertices[0][1],
face.vertices[1][2] - face.vertices[0][2])
side_b = (face.vertices[-1][0] - face.vertices[0][0],
face.vertices[-1][1] - face.vertices[0][1],
face.vertices[-1][2] - face.vertices[0][2])
# compute the cross product
face.normal = (side_a[1]*side_b[2] - side_a[2]*side_b[1],
side_a[2]*side_b[0] - side_a[0]*side_b[2],
side_a[0]*side_b[1] - side_a[1]*side_b[0])
def importObj(stream):
vertices = []
faces = []
uvs = []
normals = []
for line in stream:
# make sure there's no new line or trailing spaces
l = line.strip().split(' ')
lineType = l[0].strip()
data = l[1:]
if lineType == 'v':
# vertex
v = tuple(map(float, data))
vertices.append(v)
elif lineType == 'vt':
# uv
uvs.append( tuple(map(float, data)) )
elif lineType == 'vn':
# normal
normals.append( tuple(map(float, data)) )
elif lineType == 'f':
# face (assume all verts/uvs/normals have been processed)
faceVerts = []
faceUvs = []
faceNormals = []
for v in data:
result = v.split('/')
print(result)
# recall that everything is 1 indexed...
faceVerts.append(vertices[int(result[0]) - 1])
if len(result) == 1:
continue # there is only a vertex index
if result[1] != '':
# uvs may not be present, ex: 'f vert//normal ...'
faceUvs.append(uvs[int(result[1]) - 1])
if len(result) <= 2:
# don't continue if only vert and uv are present
continue
faceNormals.append(normals[int(result[2]) - 1])
faces.append( GeoFace(faceVerts, faceUvs, faceNormals) )
else:
# there could be material specs, smoothing, or comments... ignore!
pass
return faces
def exportObj(stream, aaQuads):
# gather some of the needed information
faces = aaQuads
# copy the normals from AAQuad (99% of cases will use all directions)
normals = list(AAQuad.normals)
uvs = set()
for f in faces:
if f.uv is not None:
uvs.add(f.uv)
# convert this to a list because we need to get their index later
uvs = list(uvs)
# we will build a list of vertices as we go and then write everything
# in bulk, disadvantage that MANY verts will be duplicated in the OBJ file
fLines = []
vertices = []
indexOffset = 0
for f in faces:
# recall that OBJ files are 1 indexed
n = 1 + normals.index(f.normal) if f.normal is not None else ''
uv = 1 + uvs.index(f.uv) if f.uv is not None else ''
# this used to be a one liner ;)
fLine = ['f']
for i, vert in enumerate(f.vertices):
# for each vertex of this face
v = 1 + indexOffset + f.vertices.index(vert)
fLine.append(str(v) + '/' + str(uv) + '/' + str(n))
vertices.extend(f.vertices)
indexOffset += len(f.vertices)
fLines.append(' '.join(fLine) + '\n')
# write to the file
stream.write('# shivshank\'s .obj optimizer\n')
stream.write('\n')
if len(normals) > 0:
stream.write('# normals\n')
for n in normals:
stream.write('vn ' + ' '.join(list(map(str, n))) + '\n')
stream.write('\n')
if len(uvs) > 0:
stream.write('# texcoords\n')
for i in uvs:
stream.write('vt ' + ' '.join(list(map(str, i))) + '\n')
stream.write('\n')
# output the vertices and faces
stream.write('# verts\n')
for v in vertices:
stream.write('v ' + ' '.join(list(map(str, v))) + '\n')
stream.write('\n')
stream.write('# faces\n')
for i in fLines:
stream.write(i)
stream.write('\n')
stream.write('\n')
return len(vertices), len(fLines)
def importVox(file):
""" --> a VoxelStruct from this .vox file stream """
# in theory this could elegantly be many functions and classes
# but this is such a simple file format...
# refactor: ? should probably find a better exception type than value error
vox = VoxelStruct()
magic = file.read(4)
if magic != b'VOX ':
print('magic number is', magic)
if userAborts('This does not appear to be a VOX file. Abort?'):
raise ValueError("Invalid magic number")
# the file appears to use little endian consistent with RIFF
version = int.from_bytes(file.read(4), byteorder='little')
if version != 150:
if userAborts('Only version 150 is supported; this file: '
+ str(version) + '. Abort?'):
raise ValueError("Invalid file version")
mainHeader = _readChunkHeader(file)
if mainHeader['id'] != b'MAIN':
print('chunk id:', mainId)
if userAborts('Did not find the main chunk. Abort?'):
raise ValueError("Did not find main VOX chunk. ")
#assert mainHeader['size'] == 0, "main chunk should have size 0"
# we don't need anything from the size or palette header!
# : we can figure out (minimum) bounds later from the voxel data
# : we only need UVs from voxel data; user can export palette elsewhere
nextHeader = _readChunkHeader(file)
while nextHeader['id'] != b'XYZI':
# skip the contents of this header and its children, read the next one
file.read(nextHeader['size'] + nextHeader['childrenSize'])
nextHeader = _readChunkHeader(file)
voxelHeader = nextHeader
assert voxelHeader['id'] == b'XYZI', 'this should be literally impossible'
assert voxelHeader['childrenSize'] == 0, 'why voxel chunk have children?'
seekPos = file.tell()
totalVoxels = int.from_bytes(file.read(4), byteorder='little')
### READ THE VOXELS ###
for i in range(totalVoxels):
# n.b., byte order should be irrelevant since these are all 1 byte
x = int.from_bytes(file.read(1), byteorder='little')
y = int.from_bytes(file.read(1), byteorder='little')
z = int.from_bytes(file.read(1), byteorder='little')
color = int.from_bytes(file.read(1), byteorder='little')
vox.setVoxel(Voxel(x, y, z, color))
# assert that we've read the entire voxel chunk
assert file.tell() - seekPos == voxelHeader['size']
# (there may be more chunks after this but we don't need them!)
#print('\tdone reading voxel data;', totalVoxels , 'voxels read ;D')
return vox
def _readChunkHeader(buffer):
id = buffer.read(4)
if id == b'':
raise ValueError("Unexpected EOF, expected chunk header")
size = int.from_bytes(buffer.read(4), byteorder='little')
childrenSize = int.from_bytes(buffer.read(4), byteorder='little')
return {
'id': id, 'size': size, 'childrenSize': childrenSize
}
def userAborts(msg):
print(msg + ' (y/n)')
u = input()
if u.startswith('n'):
return False
return True
def exportAll():
""" Uses a file to automatically export a bunch of files!
See this function for details on the what the file looks like.
"""
import os, os.path
with open('exporter.txt', mode='r') as file:
# use this as a file "spec"
fromSource = os.path.abspath(file.readline().strip())
toExportDir = os.path.abspath(file.readline().strip())
optimizing = file.readline()
if optimizing.lower() == 'true':
optimizing = True
else:
optimizing = False
print('exporting vox files under', fromSource)
print('\tto directory', toExportDir)
print('\toptimizing?', optimizing)
print()
# export EVERYTHING (.vox) walking the directory structure
for p, dirList, fileList in os.walk(fromSource):
pathDiff = os.path.relpath(p, start=fromSource)
outDir = os.path.join(toExportDir, pathDiff)
# REFACTOR: the loop should be moved to a function
for fileName in fileList:
# only take vox files
if os.path.splitext(fileName)[1] != '.vox':
print('ignored', fileName)
continue
print('exporting', fileName)
# read/import the voxel file
with open(os.path.join(p, fileName), mode='rb') as file:
try:
vox = importVox(file)
except ValueError as exc:
print('aborted', fileName, str(exc))
continue
# mirror the directory structure in the export folder
if not os.path.exists(outDir):
os.makedirs(outDir)
print('\tcreated directory', outDir)
# export a non-optimized version
objName = os.path.splitext(fileName)[0]
rawQuads = vox.toQuads()
with open(os.path.join(outDir, objName + '.obj'), mode='w') as file:
vCount, qCount = exportObj(file, rawQuads)
print('\texported', vCount, 'vertices,', qCount, 'quads')
if optimizing:
# TODO
continue
optiFaces = optimizedGreedyMesh(rawQuads)
bucketHash(optiFaces, *vox.getBounds())
with open(os.path.join(outDir, objName + '.greedy.obj'),
mode='w') as file:
exportObj(file, optiFaces)
def byPrompt():
import os, os.path
from glob import glob
print('Enter an output path:')
u = input('> ').strip()
while not os.path.exists(u):
print('That path does not exist.')
print('Enter an output path:')
u = input('> ').strip()
outRoot = os.path.abspath(u)
print('Are we optimizing? (y/n)')
u = input('> ').strip()
# this could be a one liner but I think it's easier to read this way
if u.startswith('y'):
optimizing = True
else:
optimizing = False
try:
while True:
print('Enter glob of export files (\'exit\' or blank to quit):')
u = input('> ').strip()
if u == 'exit' or u == '':
break
u = glob(u)
for f in u:
print('reading VOX file', f)
with open(f, mode='rb') as file:
try:
vox = importVox(file)
except ValueError:
print('\tfile reading aborted')
continue
outFile = os.path.splitext(os.path.basename(f))[0]
outPath = os.path.join(outRoot, outFile)
print('exporting VOX to OBJ at path', outPath)
with open(outPath, mode='w') as file:
exportObj(file, vox.toQuads())
if optimizing:
# TODO
pass
except KeyboardInterrupt:
pass
if __name__ == "__main__":
profiling = False
try:
import cProfile
if profiling:
cProfile.run('exportAll()', sort='tottime')
else:
exportAll()
except OSError:
print('No instruction file found, falling back to prompt.')
byPrompt()
Raw
vox_to_obj_PROMPT.py
"""
SEMI-AUTOMATIC drag and drop support for windows
1. Copy script to directory you want your files copied to.
2. Select the files you want to convert.
3. Drag & drop onto this script.
4. Prompt will appear -- Press "enter" to convert .vox to .obj! (or abort with "y")
Files will be exported to directory of this script.
semi-automatic mod by awesomedata
This script is designed to export a mass amount of MagicaVoxel .vox files
to .obj. Unlike Magica's internal exporter, this exporter preserves the
voxel vertices for easy manipulating in a 3d modeling program like Blender.
Various meshing algorithms are included (or to be included). MagicaVoxel
uses monotone triangulation (I think). The algorithms that will (or do)
appear in this script will use methods to potentially reduce rendering
artifacts that could be introduced by triangulation of this nature.
I may also include some features like light map generation for easy
importing into Unreal Engine, etc.
Notes:
* There may be a few floating point equality comparisons. They seem to
work but it scares me a little.
* TODO: use constants instead of magic numbers (as defined in AAQuad),
(i.e., ..., 2 -> AAQuad.TOP, ...)
* A lot of assertions should probably be exceptions since they are
error checking user input (this sounds really bad now that I've put
it on paper...). So don't run in optimized mode (who does that
anyways?).
* I am considering adding FBX support.
"""
import math
class AAQuad:
""" A solid colored axis aligned quad. """
normals = [
(-1, 0, 0), # left = 0
(1, 0, 0), # right = 1
(0, 0, 1), # top = 2
(0, 0, -1), # bottom = 3
(0, -1, 0), # front = 4
(0, 1, 0) # back = 5
]
LEFT = 0
RIGHT = 1
TOP = 2
BOTTOM = 3
FRONT = 4
BACK = 5
def __init__(self, verts, uv=None, normal=None):
assert len(verts) == 4, "face must be a quad"
self.vertices = verts
self.uv = uv
self.normal = normal
def __str__(self):
s = []
for i in self.vertices:
s.append( str(i) + '/' + str(self.uv) + '/' + str(self.normal))
return 'f ' + ' '.join(s)
def center(self):
return (
sum(i[0] for i in self.vertices)/4,
sum(i[1] for i in self.vertices)/4,
sum(i[2] for i in self.vertices)/4
)
def bucketHash(faces, origin, maximum, bucket=16):
extents = (
math.ceil((maximum[0] - origin[0])/bucket),
math.ceil((maximum[1] - origin[1])/bucket),
math.ceil((maximum[2] - origin[2])/bucket)
)
buckets = {}
for f in faces:
c = f.center()
# TODO
def optimizedGreedyMesh(faces):
# TODO
edges = adjacencyGraphEdges(faces)
groups = contiguousFaces(faces, edges)
return faces
def adjacencyGraphEdges(faces):
""" Get the list of edges representing adjacent faces. """
# a list of edges, where edges are tuple(face_a, face_b)
edges = []
# build the list of edges in the graph
for root in faces:
for face in faces:
if face is root:
continue
if facesAreAdjacent(root, face):
# the other edge will happen somewhere else in the iteration
# (i.e., the relation isAdjacent is symmetric)
edges.append((root, face))
return edges
def contiguousFaces(faces, adjacencyGraphEdges):
""" Get the list of connected components from a list of graph edges.
The list will contain lists containing the edges within the components.
"""
groups = []
visited = dict((f, False) for f in faces)
for face in faces:
# if the face hasn't been visited, it is not in any found components
if not visited[face]:
g = []
_visitGraphNodes(face, adjacencyGraphEdges, visited, g)
# there is only a new component if face has not been visited yet
groups.append(g)
return groups
def _visitGraphNodes(node, edges, visited, component):
""" Recursive routine used in findGraphComponents """
# visit every component connected to this one
for edge in edges:
# for all x in nodes, (node, x) and (x, node) should be in edges!
# therefore we don't have to check for "edge[1] is node"
if edge[0] is node and not visited[edge[1]]:
assert edge[1] is not node, "(node, node) should not be in edges"
# mark the other node as visited
visited[edge[1]] = True
component.append(edge[1])
# visit all of that nodes connected nodes
_visitGraphNodes(edge[1], edges, visited, component)
def facesAreAdjacent(a, b):
""" Adjacent is defined as same normal, uv, and a shared edge.
This isn't entirely intuitive (i.e., corner faces are not adjacent)
but this definition fits the problem domain.
Only works on AAQuads.
"""
# note: None is == None, this shouldn't matter
if a.uv != b.uv:
return False
if a.normal != b.normal:
return False
# to be adjacent, two faces must share an edge
# use == and not identity in case edge split was used
shared = 0
for vert_a in a.vertices:
for vert_b in b.vertices:
if vert_a == vert_b:
shared += 1
# hooray we have found a shared edge (or a degenerate case...)
if shared == 2:
return True
return False
class GeoFace:
""" An arbitrary geometry face
This should only be used for arbitrary models, not ones we can
reasonably assume are axis aligned.
"""
def __init__(self, verts, uvs=None, normals=None):
self.vertices = verts
assert len(verts) in (3, 4), "only quads and tris are supported"
self.normals = normals
self.uvs = uvs
def toAAQuad(self, skipAssert=False):
q = AAQuad(self.vertices)
if self.normals is not None and len(self.normals) > 0:
if not skipAssert:
for i in self.normals:
assert self.normals[0] == i, \
"face must be axis aligned (orthogonal normals)"
q.normal = self.normals[0]
if self.uvs is not None and len(self.uvs) > 0:
if not skipAssert:
for i in self.uvs:
assert self.uvs[0] == i, \
"face must be axis aligned (orthogonal)"
q.uv = self.uvs[0]
return q
class VoxelStruct:
""" Describes a voxel object
"""
def __init__(self):
# a dict is probably the best way to go about this
# (as a trade off between performance and code complexity)
# see _index for the indexing method
self.voxels = {}
self.colorIndices = set()
def fromList(self, voxels):
self.voxels = {}
for voxel in voxels:
self.setVoxel(voxel)
self.colorIndices.add(voxel.colorIndex)
def setVoxel(self, voxel):
self.voxels[voxel.z*(256**2) + voxel.y * 256 + voxel.x] = voxel
def getVoxel(self, x, y, z):
return self.voxels.get(z*(256**2) + y * 256 + x, None)
def _index(self, x, y, z):
return z*(256**2) + y * 256 + x
def getBounds(self):
origin = (float("inf"), float("inf"), float("inf"))
maximum = (float("-inf"), float("-inf"), float("-inf"))
for key, voxel in self.voxels.items():
origin = (
min(origin[0], voxel.x),
min(origin[1], voxel.y),
min(origin[2], voxel.z)
)
maximum = (
max(maximum[0], voxel.x),
max(maximum[1], voxel.y),
max(maximum[2], voxel.z)
)
return origin, maximum
def zeroOrigin(self):
""" Translate the model so that it's origin is at 0, 0, 0 """
origin, maximum = self.getBounds()
result = {}
xOff, yOff, zOff = origin
for key, voxel in self.voxels.iteritems():
result[self._index(voxel.x-xOff, voxel.y-yOff, voxel.z-zOff)] = \
Voxel(voxel.x-xOff, voxel.y-yOff, voxel.z-zOff,
voxel.colorIndex)
self.voxels = result
return (0, 0, 0), (maximum[0] - xOff,
maximum[1] - yOff,
maximum[2] - zOff)
def toQuads(self):
""" --> a list of AAQuads """
faces = []
for key, voxel in self.voxels.items():
self._getObjFaces(voxel, faces)
return faces
def _getObjFaces(self, voxel, outFaces):
if voxel.colorIndex == 0:
# do nothing if this is an empty voxel
# n.b., I do not know if this ever can happen.
return []
sides = self._objExposed(voxel)
if sides[0]:
f = self._getLeftSide(voxel)
self._getObjFacesSupport(0, voxel.colorIndex, f, outFaces)
if sides[1]:
f = self._getRightSide(voxel)
self._getObjFacesSupport(1, voxel.colorIndex, f, outFaces)
if sides[2]:
f = self._getTopSide(voxel)
self._getObjFacesSupport(2, voxel.colorIndex, f, outFaces)
if sides[3]:
f = self._getBottomSide(voxel)
self._getObjFacesSupport(3, voxel.colorIndex, f, outFaces)
if sides[4]:
f = self._getFrontSide(voxel)
self._getObjFacesSupport(4, voxel.colorIndex, f, outFaces)
if sides[5]:
f = self._getBackSide(voxel)
self._getObjFacesSupport(5, voxel.colorIndex, f, outFaces)
return
n = AAQuad.normals[i]
# note: texcoords are based on MagicaVoxel's texturing scheme!
# meaning a color index of 0 translates to pixel[255]
# and color index [1:256] -> pixel[0:255]
u = ((voxel.colorIndex - 1)/256 + 1/512, 0.5)
outFaces.append(
# this is most definitely not "fun"
AAQuad(f, u, n)
)
def _getObjFacesSupport(self, side, color, faces, outFaces):
n = AAQuad.normals[side]
# note: texcoords are based on MagicaVoxel's texturing scheme!
# meaning a color index of 0 translates to pixel[255]
# and color index [1:256] -> pixel[0:255]
u = ((color - 1)/256 + 1/512, 0.5)
outFaces.append(
# fact: the parameters were coincidentally "f, u, n" at one point!
AAQuad(faces, u, n)
)
# MagicaVoxel does -.5 to +.5 for each cube, we'll do 0.0 to 1.0 ;)
def _getLeftSide(self, voxel):
return [
(voxel.x, voxel.y + 1, voxel.z + 1),
(voxel.x, voxel.y + 1, voxel.z),
(voxel.x, voxel.y, voxel.z),
(voxel.x, voxel.y, voxel.z + 1)
]
def _getRightSide(self, voxel):
return (
(voxel.x + 1, voxel.y, voxel.z + 1),
(voxel.x + 1, voxel.y, voxel.z),
(voxel.x + 1, voxel.y + 1, voxel.z),
(voxel.x + 1, voxel.y + 1, voxel.z + 1)
)
def _getTopSide(self, voxel):
return (
(voxel.x, voxel.y + 1, voxel.z + 1),
(voxel.x, voxel.y, voxel.z + 1),
(voxel.x + 1, voxel.y, voxel.z + 1),
(voxel.x + 1, voxel.y + 1, voxel.z + 1)
)
def _getBottomSide(self, voxel):
return (
(voxel.x, voxel.y, voxel.z),
(voxel.x, voxel.y + 1, voxel.z),
(voxel.x + 1, voxel.y + 1, voxel.z),
(voxel.x + 1, voxel.y, voxel.z)
)
def _getFrontSide(self, voxel):
return (
(voxel.x, voxel.y, voxel.z + 1),
(voxel.x, voxel.y, voxel.z),
(voxel.x + 1, voxel.y, voxel.z),
(voxel.x + 1, voxel.y, voxel.z + 1)
)
def _getBackSide(self, voxel):
return (
(voxel.x + 1, voxel.y + 1, voxel.z + 1),
(voxel.x + 1, voxel.y + 1, voxel.z),
(voxel.x, voxel.y + 1, voxel.z),
(voxel.x, voxel.y + 1, voxel.z + 1)
)
def _objExposed(self, voxel):
""" --> a set of [0, 6) representing which voxel faces are shown
for the meaning of 0-5, see AAQuad.normals
get the sick truth about these voxels' dirty secrets...
"""
# check left 0
side = self.getVoxel(voxel.x - 1, voxel.y, voxel.z)
s0 = side is None or side.colorIndex == 0
# check right 1
side = self.getVoxel(voxel.x + 1, voxel.y, voxel.z)
s1 = side is None or side.colorIndex == 0
# check top 2
side = self.getVoxel(voxel.x, voxel.y, voxel.z + 1)
s2 = side is None or side.colorIndex == 0
# check bottom 3
side = self.getVoxel(voxel.x, voxel.y, voxel.z - 1)
s3 = side is None or side.colorIndex == 0
# check front 4
side = self.getVoxel(voxel.x, voxel.y - 1, voxel.z)
s4 = side is None or side.colorIndex == 0
# check back 5
side = self.getVoxel(voxel.x, voxel.y + 1, voxel.z)
s5 = side is None or side.colorIndex == 0
return s0, s1, s2, s3, s4, s5
class Voxel:
def __init__(self, x, y, z, colorIndex):
self.x = x
self.y = y
self.z = z
self.colorIndex = colorIndex
def genNormals(self, aaQuads, overwrite=False):
# compute CCW normal if it doesn't exist
for face in aaQuads:
if overwrite or face.normal is None:
side_a = (face.vertices[1][0] - face.vertices[0][0],
face.vertices[1][1] - face.vertices[0][1],
face.vertices[1][2] - face.vertices[0][2])
side_b = (face.vertices[-1][0] - face.vertices[0][0],
face.vertices[-1][1] - face.vertices[0][1],
face.vertices[-1][2] - face.vertices[0][2])
# compute the cross product
face.normal = (side_a[1]*side_b[2] - side_a[2]*side_b[1],
side_a[2]*side_b[0] - side_a[0]*side_b[2],
side_a[0]*side_b[1] - side_a[1]*side_b[0])
def importObj(stream):
vertices = []
faces = []
uvs = []
normals = []
for line in stream:
# make sure there's no new line or trailing spaces
l = line.strip().split(' ')
lineType = l[0].strip()
data = l[1:]
if lineType == 'v':
# vertex
v = tuple(map(float, data))
vertices.append(v)
elif lineType == 'vt':
# uv
uvs.append( tuple(map(float, data)) )
elif lineType == 'vn':
# normal
normals.append( tuple(map(float, data)) )
elif lineType == 'f':
# face (assume all verts/uvs/normals have been processed)
faceVerts = []
faceUvs = []
faceNormals = []
for v in data:
result = v.split('/')
print(result)
# recall that everything is 1 indexed...
faceVerts.append(vertices[int(result[0]) - 1])
if len(result) == 1:
continue # there is only a vertex index
if result[1] != '':
# uvs may not be present, ex: 'f vert//normal ...'
faceUvs.append(uvs[int(result[1]) - 1])
if len(result) <= 2:
# don't continue if only vert and uv are present
continue
faceNormals.append(normals[int(result[2]) - 1])
faces.append( GeoFace(faceVerts, faceUvs, faceNormals) )
else:
# there could be material specs, smoothing, or comments... ignore!
pass
return faces
def exportObj(stream, aaQuads):
# gather some of the needed information
faces = aaQuads
# copy the normals from AAQuad (99% of cases will use all directions)
normals = list(AAQuad.normals)
uvs = set()
for f in faces:
if f.uv is not None:
uvs.add(f.uv)
# convert this to a list because we need to get their index later
uvs = list(uvs)
# we will build a list of vertices as we go and then write everything
# in bulk, disadvantage that MANY verts will be duplicated in the OBJ file
fLines = []
vertices = []
indexOffset = 0
for f in faces:
# recall that OBJ files are 1 indexed
n = 1 + normals.index(f.normal) if f.normal is not None else ''
uv = 1 + uvs.index(f.uv) if f.uv is not None else ''
# this used to be a one liner ;)
fLine = ['f']
for i, vert in enumerate(f.vertices):
# for each vertex of this face
v = 1 + indexOffset + f.vertices.index(vert)
fLine.append(str(v) + '/' + str(uv) + '/' + str(n))
vertices.extend(f.vertices)
indexOffset += len(f.vertices)
fLines.append(' '.join(fLine) + '\n')
# write to the file
stream.write('# shivshank\'s .obj optimizer\n')
stream.write('\n')
if len(normals) > 0:
stream.write('# normals\n')
for n in normals:
stream.write('vn ' + ' '.join(list(map(str, n))) + '\n')
stream.write('\n')
if len(uvs) > 0:
stream.write('# texcoords\n')
for i in uvs:
stream.write('vt ' + ' '.join(list(map(str, i))) + '\n')
stream.write('\n')
# output the vertices and faces
stream.write('# verts\n')
for v in vertices:
stream.write('v ' + ' '.join(list(map(str, v))) + '\n')
stream.write('\n')
stream.write('# faces\n')
for i in fLines:
stream.write(i)
stream.write('\n')
stream.write('\n')
return len(vertices), len(fLines)
def importVox(file):
""" --> a VoxelStruct from this .vox file stream """
# in theory this could elegantly be many functions and classes
# but this is such a simple file format...
# refactor: ? should probably find a better exception type than value error
vox = VoxelStruct()
magic = file.read(4)
if magic != b'VOX ':
print('magic number is', magic)
if userAborts('This does not appear to be a VOX file. Abort?'):
raise ValueError("Invalid magic number")
# the file appears to use little endian consistent with RIFF
version = int.from_bytes(file.read(4), byteorder='little')
if version != 150:
if userAborts('Only version 150 is supported; this file: '
+ str(version) + '. Abort?'):
raise ValueError("Invalid file version")
mainHeader = _readChunkHeader(file)
if mainHeader['id'] != b'MAIN':
print('chunk id:', mainId)
if userAborts('Did not find the main chunk. Abort?'):
raise ValueError("Did not find main VOX chunk. ")
#assert mainHeader['size'] == 0, "main chunk should have size 0"
# we don't need anything from the size or palette header!
# : we can figure out (minimum) bounds later from the voxel data
# : we only need UVs from voxel data; user can export palette elsewhere
nextHeader = _readChunkHeader(file)
while nextHeader['id'] != b'XYZI':
# skip the contents of this header and its children, read the next one
file.read(nextHeader['size'] + nextHeader['childrenSize'])
nextHeader = _readChunkHeader(file)
voxelHeader = nextHeader
assert voxelHeader['id'] == b'XYZI', 'this should be literally impossible'
assert voxelHeader['childrenSize'] == 0, 'why voxel chunk have children?'
seekPos = file.tell()
totalVoxels = int.from_bytes(file.read(4), byteorder='little')
### READ THE VOXELS ###
for i in range(totalVoxels):
# n.b., byte order should be irrelevant since these are all 1 byte
x = int.from_bytes(file.read(1), byteorder='little')
y = int.from_bytes(file.read(1), byteorder='little')
z = int.from_bytes(file.read(1), byteorder='little')
color = int.from_bytes(file.read(1), byteorder='little')
vox.setVoxel(Voxel(x, y, z, color))
# assert that we've read the entire voxel chunk
assert file.tell() - seekPos == voxelHeader['size']
# (there may be more chunks after this but we don't need them!)
#print('\tdone reading voxel data;', totalVoxels , 'voxels read ;D')
return vox
def _readChunkHeader(buffer):
id = buffer.read(4)
if id == b'':
raise ValueError("Unexpected EOF, expected chunk header")
size = int.from_bytes(buffer.read(4), byteorder='little')
childrenSize = int.from_bytes(buffer.read(4), byteorder='little')
return {
'id': id, 'size': size, 'childrenSize': childrenSize
}
def userAborts(msg):
print(msg + ' (y/n)')
u = input()
if u.startswith('n'):
return False
return True
def exportAll():
""" Uses a file to automatically export a bunch of files!
See this function for details on the what the file looks like.
"""
import os, os.path
with open('exporter.txt', mode='r') as file:
# use this as a file "spec"
fromSource = os.path.abspath(file.readline().strip())
toExportDir = os.path.abspath(file.readline().strip())
optimizing = file.readline()
if optimizing.lower() == 'true':
optimizing = True
else:
optimizing = False
print('exporting vox files under', fromSource)
print('\tto directory', toExportDir)
print('\toptimizing?', optimizing)
print()
# export EVERYTHING (.vox) walking the directory structure
for p, dirList, fileList in os.walk(fromSource):
pathDiff = os.path.relpath(p, start=fromSource)
outDir = os.path.join(toExportDir, pathDiff)
# REFACTOR: the loop should be moved to a function
for fileName in fileList:
# only take vox files
if os.path.splitext(fileName)[1] != '.vox':
print('ignored', fileName)
continue
print('exporting', fileName)
# read/import the voxel file
with open(os.path.join(p, fileName), mode='rb') as file:
try:
vox = importVox(file)
except ValueError as exc:
print('aborted', fileName, str(exc))
continue
# mirror the directory structure in the export folder
if not os.path.exists(outDir):
os.makedirs(outDir)
print('\tcreated directory', outDir)
# export a non-optimized version
objName = os.path.splitext(fileName)[0]
rawQuads = vox.toQuads()
with open(os.path.join(outDir, objName + '.obj'), mode='w') as file:
vCount, qCount = exportObj(file, rawQuads)
print('\texported', vCount, 'vertices,', qCount, 'quads')
if optimizing:
# TODO
continue
optiFaces = optimizedGreedyMesh(rawQuads)
bucketHash(optiFaces, *vox.getBounds())
with open(os.path.join(outDir, objName + '.greedy.obj'),
mode='w') as file:
exportObj(file, optiFaces)
def byPrompt():
import os, os.path, sys
from glob import glob
#### set output directory to first .vox file location
# ------------------------------------------------
####
u = os.path.abspath(sys.argv[0]).strip(os.path.basename(sys.argv[0]))
print(u)
#### drag & dropped files
# ---------------------
for i in sys.argv:
if i != sys.argv[0]:
print(i)
#### fully manual prompt ####
# -------------------
# print('Enter an output path:')
# u = input('> ').strip()
while not os.path.exists(u):
print('That path does not exist.')
print('Enter an output path:')
u = input('> ').strip()
outRoot = os.path.abspath(u)
print('\nStop file conversion to .obj? (type "y" or press "enter" to convert.)')
u = input('> ').strip()
if u.startswith('y'):
exit;
else:
try:
#while True:
#### grab files from prompt (uncomment lines below if needed)
# ----------------------
#print('Enter glob of export files (\'exit\' or blank to quit):')
#u = input('> ').strip()
#if u == 'exit' or u == '':
# break
#u = glob(u)
#### grab drag & dropped files
u = sys.argv
for f in u:
if f != sys.argv[0]:
print('reading VOX file', f)
with open(f, mode='rb') as file:
try:
vox = importVox(file)
except ValueError:
print('\tfile reading aborted')
continue
outFile = os.path.splitext(os.path.basename(f))[0]
outPath = os.path.join(outRoot, outFile+'.obj')
print('exporting VOX to OBJ at path', outPath)
with open(outPath, mode='w') as file:
exportObj(file, vox.toQuads())
except KeyboardInterrupt:
pass
if __name__ == "__main__":
profiling = False
try:
import cProfile
if profiling:
cProfile.run('exportAll()', sort='tottime')
else:
exportAll()
except OSError:
print('No instruction file found, falling back to prompt.')
byPrompt()
@awesomez
Copy link
Author

Not a problem! -- I'm just glad I could help somehow! ^___^

@AleskeyMijaylob
Copy link

AleskeyMijaylob commented May 5, 2020 •
edited
Loading

No instruction file found, falling back to prompt.
Enter an output path:

_
._. im not programmer

@awesomez
Copy link
Author

awesomez commented May 6, 2020 •
edited
Loading

No instruction file found, falling back to prompt.
Enter an output path:

The "output path" is the path to where you want the .obj file to be exported to -- i.e. C:\Windows\Users\yourname\Documents
NOTE: Don't use the file ending in "_PROMPT.py" file if you want to output to the current directory.

Next -- Did you drag/drop the .vox file onto one of the .py scripts?
If you did, and you've pressed Enter, and this message still appears, there are three possibilities:

1.) Old Python version prior to version 3.5 (I think?)
2.) Magica Voxel version too new
3.) Use / instead of \ for path to output directory to see if that works

Been a while since I've played with this script, but you don't need an "output directory" when you have an "instruction file" (which is described a little in the main branch of this repository).

FINAL NOTE:

You'd want to use "vox_to_obj_AUTO.py" (the first script above) most of the time (rather than "vox_to_obj_PROMPT.py", which gives you a prompt for the output directory unless an "instruction file" exists in the same folder as the script)

@raavek
Copy link

raavek commented Jan 25, 2021

This script is awesome thank you so much!

I've been really stumped and this script is really helping. Im not a programmer but I got this script to output just fine.
I do have a question/huge favor/guidance if I could ask. Is there any way to export an obj file that preserves internal voxel structure as well, not just the external surface? So the export would have an object containing internal voxel structure with color palette applied?

@awesomez
Copy link
Author

This script is awesome thank you so much!

I just adapted it to make it more user-friendly -- thanks though! :)

Is there any way to export an obj file that preserves internal voxel structure as well, not just the external surface? So the export would have an object containing internal voxel structure with color palette applied?

The "importVox(file)" method does exactly what you want -- it returns a VoxelStruct class / set of data, which is what you would want to output to a file (such as JSON), which you'd have to do yourself in Python atm. Finally, you'd need to import that JSON into whatever app you want that data for.

I know you said you're not a programmer, but since I don't have time to make it myself, you might want to give it a go. On the plus side - Python is very straightforward, and instead of curly brackets, etc to define "groups" of code -- it uses TAB spacing. That said, if you end up making a version of this script that outputs JSON (which, IMO, is the best raw-text data exchange format available), please let me know. It would be very nice to have a JSON version of this script in this repository for anyone interested in using the raw voxel data structure (instead of just the .obj) to use later in another application.
Like I said, it should be VERY straightforward to go this route (you're really just exporting the "VoxelStruct" class's individual properties / data as variables, one at a time, for each individual property like x, y, z, color -- and writing that to a simple text file using the JSON format -- that's it). To do my part (and give back to your efforts) -- if you do decide to take it on, and you have any issues implementing it (or questions about how or what to do or where to look), I don't mind walking you through getting a JSON export up and running (a little bit at a time), but I don't think there are too many things that could go wrong if you glance at the basics of Python and study that function/method I pointed out (importVox) a tiny bit. Most of the stuff you'd need to do exists in the script already (including writing to a file -- except this is binary, not plaintext, so you'd have to look into writing to a plain text file). Other than that, it should be pretty simple. Please let me know if you plan to do this (and if you run into any issues, assuming you do!) 👍

@raavek
Copy link

raavek commented Jan 25, 2021

This script is awesome thank you so much!

I just adapted it to make it more user-friendly -- thanks though! :)

Is there any way to export an obj file that preserves internal voxel structure as well, not just the external surface? So the export would have an object containing internal voxel structure with color palette applied?

The "importVox(file)" method does exactly what you want -- it returns a VoxelStruct class / set of data, which is what you would want to output to a file (such as JSON), which you'd have to do yourself in Python atm. Finally, you'd need to import that JSON into whatever app you want that data for.

I know you said you're not a programmer, but since I don't have time to make it myself, you might want to give it a go. On the plus side - Python is very straightforward, and instead of curly brackets, etc to define "groups" of code -- it uses TAB spacing. That said, if you end up making a version of this script that outputs JSON (which, IMO, is the best raw-text data exchange format available), please let me know. It would be very nice to have a JSON version of this script in this repository for anyone interested in using the raw voxel data structure (instead of just the .obj) to use later in another application.
Like I said, it should be VERY straightforward to go this route (you're really just exporting the "VoxelStruct" class's individual properties / data as variables, one at a time, for each individual property like x, y, z, color -- and writing that to a simple text file using the JSON format -- that's it). To do my part (and give back to your efforts) -- if you do decide to take it on, and you have any issues implementing it (or questions about how or what to do or where to look), I don't mind walking you through getting a JSON export up and running (a little bit at a time), but I don't think there are too many things that could go wrong if you glance at the basics of Python and study that function/method I pointed out (importVox) a tiny bit. Most of the stuff you'd need to do exists in the script already (including writing to a file -- except this is binary, not plaintext, so you'd have to look into writing to a plain text file). Other than that, it should be pretty simple. Please let me know if you plan to do this (and if you run into any issues, assuming you do!) 👍

Phew. Ok, Ive always wanted to start learning python. I will give this a go. Ive needed to learn about JSON as well so this should be a good challenge. I've never coded anything outside of an arduino or a basic webpage in my life. If I do get this coded when I put the code up here I will do something called a fork off your code? Im really new and basic at this sorry lol. You've lit a fire in me and now I really want to do this lol

@awesomez
Copy link
Author

Phew. Ok, Ive always wanted to start learning python. I will give this a go. Ive needed to learn about JSON as well so this should be a good challenge. I've never coded anything outside of an arduino or a basic webpage in my life.

You should be set then -- Python is extremely useful/versatile, but it is easier to learn than programming webpages and Arduino stuff!!
And JSON is just a great (highly-useful!) data-interchange format that is a lot like XML, but smaller and less verbose -- and makes A LOT more sense imo. It's not much different than CSS syntax for webpages -- so extremely basic.

If I do get this coded when I put the code up here I will do something called a fork off your code? Im really new and basic at this sorry lol. You've lit a fire in me and now I really want to do this lol

That's great to hear! -- It sounds like you'd benefit from this experience greatly!

If you want to make a "https://pastebin.com/" link when you're done (i.e. create a link on that site and paste it here), I can add your new python script(s) to this (current) repository for you if you'd like. That way everything will stay in one place, and you can always find it with the other .obj scripts in the future. This means you wouldn't have to mess with github (unless you want to -- and in that case, a fork would let you add onto/develop your script(s) further in the future if you wish). If future development isn't what you're after, pastbin and simply posting the link to your script here to be added to this repository when you're done would still allow you to fork it later, assuming you wanted to make changes. However, I offered to keep your changes here to keep all the new scripts in one place (by keeping them in the same area). I can do this in the future for you too, assuming you decide to make future changes (if you wish). I am subscribed here, and active, so I don't mind the github bit.

But yeah -- I'm glad you're considering learning this stuff! -- It's extremely useful. After all, Voxels are really cool, and Magica Voxel is a pretty nice tool to create them with. Having a straightforward JSON interchange format from .vox files is definitely handy for working with basic voxel data, and currently there are only paid options available for such a simple script -- so you'd be doing many others a favor too. :)

Good luck -- and let me know if you run into any issues! -- I'll be glad to help! :)

@raavek
Copy link

raavek commented Jan 25, 2021

Do you have an email address or way to contact you? Ive been staring at this code for hours and I have some questions but I dont know who I should ask them too. I drew a picture to try to make sense of what I was looking at. I really do want to understand what is going on here.

https://i.imgur.com/decy4QC.png

@awesomez
Copy link
Author

Do you have an email address or way to contact you? Ive been staring at this code for hours and I have some questions but I dont know who I should ask them too. I drew a picture to try to make sense of what I was looking at. I really do want to understand what is going on here.

https://i.imgur.com/decy4QC.png

You can contact me here -- it goes straight to my email. No worries.

I was looking at your questions and to answer your most basic ones -- The part you've boxed-out in red is building a string.
Something like "self.uv = uv" is actually "building" the 'object' by defining the relationship between "uv" and "self" (initialized in the init part) as an 'alias' (that is, when referencing "self.uv", you are actually referencing the sole "uv" that was initialized in the init list of arguments -- not "self" which was also defined there). You are essentially "building" up an "object" at this step, rather than simply assigning values (as you would in other languages). You, instead, are assigning references to the original data stores -- i.e. whatever goes in "self.uv" ends up inside the "uv" container, but whatever object you're dealing with gets stored in the "self" data container. That is, you're not actually referencing "uv" when you go "self.uv = uv" -- you're assigning the data container (initialized in init of course) to the phrase "self.uv" so that when you reference that "uv" data container, you're doing it with "self.uv" as the label for that specific object's data.

Since you're defining a class here (and not really using the class yet), this is all "setup" work for your data. You're telling the computer where you're going to put the data -- and how to do that -- but not necessarily when it needs to be done. At least not until you call one of the methods defined by the "def" keyword.

I know that was a bit long-winded, but maybe that makes a little more sense?

Either way -- You might be getting ahead of yourself.

I suggest starting with the VoxelStruct class, as it is more straightforward, and ultimately where you want to be.
You only need to grab the properties out of that VoxelStruct you've imported (in "importVox", the struct is named "vox" at the very beginning of the function/method call, but since it returns "vox" as a VoxelStruct, you can call your VoxelStruct variable whatever you want. After you call "voxelstruct = importVox(...)" for example, you can now access the properties inside your VoxelStruct called "voxelstruct" and write them to a file if you wish. You would only need to create a new function/method to call this -- OR erase and use the "exportObj" function/method to automatically output the JSON file. There is some useful info in that function/method too, so back it up before you delete it in case you want to see how that file is written.

Please let me know if this helps give you a bit more direction. Definitely watch some python videos if you can though, as it's not too hard -- just a little different because you have to "build" your data objects (for more complex behaviors/referencing). In your case though -- remember, all you're doing is grabbing the output from an already "imported" VoxelStruct, and writing those values somewhere else in the "exportObj" routine (which you might call "exportVox" and change any references from "exportObj" to "exportVox" so that when you drag and drop your .vox file on there, a JSON file is output instead of a new .obj file. :)

@raavek
Copy link

raavek commented Jan 26, 2021 •
edited
Loading

I haven't started writing code yet but I have been studying python a lot and looking at this code. Still getting my brains around this. An update to where my thought process is:

https://i.imgur.com/aFt0GQ5.png

edit: Still tearing through the code. I found some pieces further down the list that I think I need to focus on. I will update when I can. This is kinda fun.

@awesomez
Copy link
Author

I noticed you're focusing on faces.
If you're trying to do something based on face geometry (i.e. put something on a particular 'face' of a voxel), then yes, the face thing is important. However, if you're just wanting the center position of the voxel itself (that is, the local 0,0,0 position of the individual voxel), you should look elsewhere (i.e. the 3d voxel index).
The reasoning here is because the actual index can be used to calculate the location of the face in physical space (from the center of each voxel) -- and even checking of the voxel should have a face in a particular position (say, behind it). The script itself uses 0-1 to calculate front/back (for example), but .vox files use -0.5-0.5 to calculate which side of a box a face is on (for front/back , left/right , top/bottom respectively). The 0-1 method avoids dealing with signage issues that might arise with multiplication with numbers like -0.5 for example, assuming you want to reference a voxel (and its faces) directly through math calculations. You can simply subtract 0.5 from 0 to get the -0.5 spatial position (for the back), or subtract 0.5 from 1 to get 0.5 (for the front of a cube), for example.

Voxels themselves aren't terribly complex -- you mainly just need a clear forward axis (z), a flat master list of voxels (i.e. count all voxels across each axis, store them to said master list while skipping any you've already found on a previous axis in order to avoid repeating them).
If you know your maximum grid size (assumed to be 256x256x256 in the script), you can go all the way to 255 on every other axis, then get that last 'plane' (possibly the ground level of voxels at the base) and scoop them up in either the first (or the final) loop in order to prevent the repetition.
From this point on, you just use the 'center' of each voxel offset to figure out where the faces are physically in the world in whatever direction you like (and store/get data from there if you wish). If the origin of your model is at the center of the base, this makes it easier (as you don't have to deal with negative values on the grid). This means the grid itself should always be a power of 2 though, in order to make those line up. Your smallest grid will be 2x2x2 at minimum. As weird as this sounds -- it is quite common believe it or not.

Hopefully that gives you some "behind the scenes" insight into this script's thought process. :)

Good luck! 👍

@raavek
Copy link

raavek commented Jan 26, 2021

I see what you are saying! I dont really want to change any specific quad/face on a voxel I would like each voxel to be monotone in color, but have all voxels retain the palette. But I do want all the internal voxels faces to be present within the render. Te reason I was looking at the faces was to try to figure out how the script determines which faces get rendered or not. I see what you are saying about a master 'position' list for voxel placement. I think I saw somewhere in the code that 1 voxel becomes 0.1m³ square in an obj file.

I think if I can find that master position index like you said and then figure out how the code determines whether to render a particular voxel or not I think then I can start figuring out how to edit the code to fit the desired purpose.

Since it is helping me to visualize this, heres another pic in cause you want a "behind the scenes" into a complete novice's first time digging into code lol.

https://i.imgur.com/0RTnF8q.png

@awesomez
Copy link
Author

The "outfaces.append" bit appends the particular quad geometry to the particular "normal" direction of the voxel on the particular sides of the voxel that need it.

What it sounds like you want is the voxel.colorIndex bit, which points to the color in the palette (0-255) used with that particular (whole) voxel, meaning that voxels don't typically let you color a single side (at least in MagicaVoxel) because their 'color' data is associated with the actual voxel 'index' you're looking at (and not any of the particular sides -- only UVs are done there). Remember, this is just exporting an .obj file, so it needs actual geometry that is also UV-mapped to specific colors on a texture regardless of palette size (and doesn't care how it renders otherwise). This means, unless you particularly care about the sides themselves, the geometry shouldn't matter.

In practice, most voxel rendering apps have one of two things -- 1) a mesh optimization step (for rendering optimized geometry based on a voxel underskinning), or 2) simply use cubes and render all sides of them (because 256x256x256 voxel cubes aren't terribly performance-heavy). You can run a culling pass of course, but it requires CPU more processing power than to simply leave them be (or do this only once, when things aren't being edited, and reverting to the cube-based rendering until that process completes).

Ideally, in JSON, storing the sides that are "appended" as a flat list of quads to draw (which is what this is doing -- except the quads are just exported instead to .obj files) to be rendered like this later is what the "optimization" step I mentioned above is usually responsible for. The only reason it is done here is because an .obj file expects (renderable) geometry definitions. Though, if you want to piggyback off of this, you probably can (assuming you don't want to reverse-engineer the quad-appending process prior to export and only want the static geo definitions of the quads for rendering instead).

Remember, voxels are grid-based and essentially arbitrary (and numerically-defined by an index) -- they don't include geometry (or even colors) naturally. In order to give them particular properties (like color), these are typically stored alongside their numerical index (in a separate and flat array, just like the voxels themselves). The only thing you keep in mind is that they shouldn't repeat themselves when counting them (despite their seemingly 3d nature), because they could just as easily work in 2d. All data in voxels is relative to the index itself -- and usually just a flat list that corresponds to the particular index of the particular placement of a voxel (in a predefined grid) -- and nothing else.
So to talk about "rendering" a voxel is to talk about the "else" -- and not the voxel itself.
Everything "else" can be figured out based on placement in the grid.

Just remember, that grid has an origin too -- and its center is usually based on the size of the 'empty' grid (pre-voxels). So in our case, 256x256x256 is the grid size (and its origin would be x=256/2 , y=256, and z=256/2). The y is special because the origin is the bottom-center, and x/z are special because they must always be a power of 2 (i.e. no less than 2 voxels wide/long -- but they can still be 1 voxel tall thanks to the special Y origin of the voxel grid being at the bottom rather than the middle).

Hopefully this helps! -- And it's interesting to see your thought process too btw! :)

@raavek
Copy link

raavek commented Jan 28, 2021

This helped a TON. I understand what im looking at way more now.
So I think I understand the (F,U,N,) referred to in the commented code.

Here is my latest brain dump pic. Trust me I'm still digging on this. It might be learning python the hard way but for me its almost easier since I can visualize the code working on the simplistic voxels. Its like my brain can kind of see the code working since its easy to mentally think of 6 quads getting assembled into 1 voxel and then those voxels getting indexed. You've been a huge help so far. And I will be actually coding soon.

https://i.imgur.com/eJ3smdN.png

@raavek
Copy link

raavek commented Jan 28, 2021

A big thing that I think is throwing me off is sometimes I will see something in code and it doesn't feel like its been referenced or defined before.... Maybe Ive been looking at the code wrong in a way.

Whenever I programmed something on an arduino, I couldn't do any sort of reference calls to anything without first defining it, and it kind of feels like the code I'm looking at does. But maybe I'm reading it wrong, I think I need to really fundamentally figure out the def statement.
Def getObject(self, x, ,y ,z):

I'm starting to see that I need to look at the stuff in the parentheses way more. I was trying to look through the arguments under the def statements without first really understanding what they are doing.

I'm going to tear through some python courses over the weekend. Ive needed to learn this for a long time now, and im really glad you are helping me so much.

@awesomez
Copy link
Author

Here is my latest brain dump pic. Trust me I'm still digging on this. It might be learning python the hard way but for me its almost easier since I can visualize the code working on the simplistic voxels. Its like my brain can kind of see the code working since its easy to mentally think of 6 quads getting assembled into 1 voxel and then those voxels getting indexed. You've been a huge help so far. And I will be actually coding soon.

https://i.imgur.com/eJ3smdN.png

These are actually extremely helpful to me believe it or not. My brain works way different than yours, but it's great to understand how you are seeing (and learning) the code. I am a very visual person, but my 'visual' is more external (i.e. I have to see it in front of me in order to understand it), while your 'visual' seems to be internal (i.e. you can imagine it and roll it around in your head and see the actual processes). This is a difficult and slow process for me, however -- at least without some kind of visual aid. And seeing the visual way you were breaking the code down in your images makes me realize I might be more like you than I thought, as your images are a helpful visual aid I never considered before. I typically learn by studying the API (which is incredibly time-consuming btw), but I now see how having actual code examples could be highly useful visually.

This is especially interesting to me because, in my spare time, I am working on a more ergonomic (visual) approach to writing / understanding code. I didn't realize how different Python really was until looking at it through that lens. There are still a few things I don't fully like about Python's specific methodology, but the gist of it is pretty clear -- code is shorter, easier to write (if you know its shorthand), and reference-heavy (and also a lot slower sometimes too, thanks to being shorthand- and reference-heavy). Almost every other language is more verbose, while Python's shorthand is a bit of an enigma in how to interpret it coming from those other languages. For example, how do you know where a reference comes from in memory if you don't actually need to define it yourself? -- In reality, Python really just "imports" data when it is necessary (that is, when the function is actually called then the memory will be allocated based around the arguments in the "def" statement and the variables defined in the function itself). Python is pretty linear -- but following the line can be hard for the uninitiated in the 'jargon' that is Python's shorthand.
They call code a "language" these days. And for good reason. I go into some of the parallels elsewhere in a single (monolithic) thread. It may be long, but those who survive say it's a pretty interesting to read. If you're interested, feel free to check it out here:

https://forum.unity.com/threads/data-oriented-visual-scripting-the-structure-of-a-language.819939/

A big thing that I think is throwing me off is sometimes I will see something in code and it doesn't feel like its been referenced or defined before.... Maybe Ive been looking at the code wrong in a way.

Whenever I programmed something on an arduino, I couldn't do any sort of reference calls to anything without first defining it, and it kind of feels like the code I'm looking at does. But maybe I'm reading it wrong, I think I need to really fundamentally figure out the def statement.
Def getObject(self, x, ,y ,z):

That's the problem with references in code languages -- they are intended to 'abstract' away "verbose" things. However, this desire to make code more "readable" is a double-edged sword. If you're not careful, you actually lose readability in other situations. You might want to get it out of your way and focus on the core meaning of your code (which is what Python aims to achieve, and actually does a decent job at it too -- if you understand Python's shorthand -- and you know all of the forms and permutations it might take -- and this really isn't something one can immediately do coming from another language -- at least without an intense -- and confusing -- introduction). I'm sorry I forgot to warn you about this part. I don't use shorthand often, so I kind of forgot it existed.

And about the def "statement" -- keep in mind, it is more of a function declaration (than a statement). You "def-ine" the functions and the arguments (so they can be called elsewhere). Therefore the "values" defined in them are not active until the function is actually called elsewhere (i.e. inside another function -- or via command-line, as Python is designed to do).
Therefore, please remember that your code is typically not "run" simply by executing the script (as it tends to be with other languages) -- Instead, with Python -- a function (i.e. the "def" declaration) must be called somewhere else to execute your code's functionality. In this case, the code gets called in the command-line. However, since you are using the AUTOrun script, the part that asks you to "wait" to execute the code (so you can get some input) was removed. This means that the function to actually output the .obj file when the .vox file is dropped on the script in the OS is executed immediately (rather than waiting for input from the command-line).

So, to clarify that last part:
The first line of code (import math) runs, then all the class (and def) declarations are run, then finally, the first actual logic of the script (if name == "main": ) is executed as the first line of code describing the actual program (outside of the initial imports/definitions). Following this, since no "instruction" file (exporter.txt) usually exists, the first actual statement that gets run is the "print()" and "byPrompt()" functions. See the ### code comments below to understand the final export "spaghetti-loop" bit better:

if __name__ == "__main__":   ###This is the program module name -- https://www.freecodecamp.org/news/if-name-main-python-example/
    profiling = False    ###This is the first line run on the main program. -- I don't need to test how long it took my script to run
    try:
        import cProfile   ###Since I don't want to test how long my script took to run, I go immediately to the "else" statement
        if profiling:
            cProfile.run('exportAll()', sort='tottime')   ### "runs" the code using the profiler (to test the speed of the program)
        else:
            exportAll()   ###This is the first code that 'does' anything -- it usually throws an OSError when "exporter.txt" file isn't found
    except OSError:
        print('No instruction file found, falling back to prompt.')    ###Not finding "exporter.txt" booted us out of exportAll() function
        byPrompt()   ###Now we execute THIS function instead.

Talk about a tangled mess just to export, right? -- Hopefully that helps explain some Python though. :)

@raavek
Copy link

raavek commented Jan 28, 2021

Where to begin?? First of all, thank you. I know its taking you time to talk to me about this and I really truly do appreciate it! Looking at the code, and then making little images really helps, but then talking to you about my thoughts not only fixes any mistakes i might have had, but it also reinforces what ive already learned. Its like a visual journal. That's also pretty crazy cool that you are working on a more ergonomic programming approach. That task has to be gargantuan. Like drag and drop modules?

You said (sorry I dont know how to quote in markdown...): So, to clarify that last part:
The first line of code (import math) runs, then all the class (and def) declarations are run, then finally, the first actual logic of the script (if name == "main": ) is executed as the first line of code describing the actual program (outside of the initial imports/definitions).

Dude this blew my mind wide open and all of this makes so much more sense now. I was trying to read the code like I would another language from top down. Knowing the order of operations is an AMAZING help.

https://forum.unity.com/threads/data-oriented-visual-scripting-the-structure-of-a-language.819939/

I'll check this out! I'm actually doing this entire thing for a unity project myself! I would love to tell you about it if you like. In fact...I know this is scope creep, but I was thinking if I could nail this python script, I could maybe somehow figure out a way to get this script functioning inside of Unity and maybe release it as a free asset? All over the internet are people who use vox models and have to import them into Unity, I was thinking if there was a direct asset that could handle that....but right now thats way beyond the scope of what I'm currently intending to do. If you are curious why i want internal voxel structure preserved, my idea was to make a game that uses emergent gameplay. The vision is to use the color palette to apply in game properties to a voxel. (IE brownWood color has properties that would let it ignite) When i build the models in Magicavoxel, I build full models with internal voxels, as I dont want just the surface layer in the game but the internal voxels/data as well. Its a huge scope I know, but I think I have pieced together a development strategy for achieving at least a surface level goal. Here's a screenshot of some of the art so far. My short/medium term goal: Get a tree in a scene inside of unity and burn it down where the fire spreads somewhat organically voxel to voxel through the surface as well as the internal voxels.

https://gyazo.com/a9c73e7577552d695dd9fb6487162f57

Anyway back to the task at hand! I think I found something pointing me towards my goal.
https://imgur.com/TIj5HZd

OK so what I've gathered here is
def expObject(stream, AAQuads):
I think this is the crucial step I needed to find in the code (I've stared at the entire code at this point, for a long time lol) I think uvs = list(uvs) might be the bulk list of voxels indexed within a voxelstruct.
I notice in the code that it is writing to \n' using a stream write. Im wondering if this is some form of data writing that allows it to be written/rewritten as more data is gathered. Im wondering if there is a point where the data that is written is the bulk vertices (maybe the full uvs list?) and then some sort of other data is applied to the \n' file that takes away the internal voxels? I know this is kind of a guess on my part. If there is a point at which all the vertices are exported to a file in bulk, I wonder if this should be where I start editing the code...

https://i.imgur.com/VJOmZln.png
Here is me looking at def importVox(file): and I don't think anything in this needs to be changes for my purposes, but I'm trying to take this slow and look through ALL the code and try to get a good understanding before I start doing surgery on it. I don't quite understand what is so important about XYZI referenced in the code...I think it has something to do with vox files specifically (ephtracy/voxel-model#19) I've actually been googling trying to find a detailed document on how to read a vox file's raw data. Its gibberish if i open it in a notepad. I don't understand what the code means when it talks about chunks. I guess I don't really need to read a vox files raw data...I just want to know how its structured...and I dont quite know what to google to find it out. A question I am not quite sure of is the term chunk, and whether that is something specific to the .vox format or python. I suspect vox.

https://i.imgur.com/PClknIq.png
This area im the least sure of. I suspect a possible issue is when 2 voxels share the same quad/face, which color is selected for that quad? In my pic I have a blue voxel and a green voxel sharing the same quad. My question is, what color is that quad going to be? I think this section of the code solves this problem, but it might present another one, as I want all the voxels inside to retain their color palette, and if they are all rendered into squares sharing the same vertices this could be a problem...

Tonight I will be diving into this more.

@raavek
Copy link

raavek commented Jan 29, 2021 •
edited
Loading

I was relooking at the def importVox(file): section of the code and I think I found something that is important.

https://i.imgur.com/EoDpqPX.png

I'm thinking vox.setVoxel(Voxel(x, y, z, color)) is like the glue that basically builds everything together. I think it is here that the object that is defined at a voxel (which is an object made up of quads) is associated with its positional and color data within the voxel structure.

So your explanation of how the code runs was a huge huge benefit. I want to go back over the code once again keeping that in mind. Now I can kind of see how this is all working together. Another thing that was throwing me for a loop was I kept seeing voxel and voxels referenced and I wasn't sure how to separate the 2 terms. It was like I know voxel and voxels are 2 separate refences within code, pluralizing a word doesn't do anything to a term codewise but it still was kind of messing with me. But after looking at the importVox section and how it relates to the voxstruct section, it is starting to become clearer I think.

@awesomez
Copy link
Author

Where to begin?? First of all, thank you. I know its taking you time to talk to me about this and I really truly do appreciate it! Looking at the code, and then making little images really helps, but then talking to you about my thoughts not only fixes any mistakes i might have had, but it also reinforces what ive already learned. Its like a visual journal. That's also pretty crazy cool that you are working on a more ergonomic programming approach. That task has to be gargantuan. Like drag and drop modules?

No problem! -- I'm enjoying this too. This is giving me a bit of a break from my standard-fare activities (since I'm not usually messing with Python), and is helping me re-establish my familiarity with this code style. I'm at a point where I need to examine this level of intricacy because programming complex geometric relationships is definitely a chore with more "verbose" languages -- so it's great to remind myself how things like this look in a (visually) more-concise language.

Speaking of visual scripting -- The answer to your question about the drag-and-drop module bit is "Not necessarily." -- Or to put it another way -- node modules were cool and unheard of for scripting back in 1995, but nowadays, many node-based languages exist -- and they all suck for one reason or another for general-purpose scripting (even while still having tons of merit and value despite their shortcomings!) So, with what I'm doing right now, I'm evaluating many different programming structures (such as Python and scripting complex things like geo, uvs, normals, faces, for example) and working out how to make those kinds of complex programs as easy (and straightforward) to write as a sentence is easy and straightforward to say -- without over-complicating the logic behind it with syntax (i.e. the way you tell the program what you want it to do -- i.e. with multiple ** asterisks, lol). This has been really fun so far -- but I'm getting to the end of the road with my research. I think this python script was actually quite a huge step to figuring out the last bits of what I need to consider before I am going to need to start looking for an avenue to deploy this knowledge in the near future. So thanks for that! :)

You said (sorry I dont know how to quote in markdown...): So, to clarify that last part:
The first line of code (import math) runs, then all the class (and def) declarations are run, then finally, the first actual logic of the script (if name == "main": ) is executed as the first line of code describing the actual program (outside of the initial imports/definitions).

Dude this blew my mind wide open and all of this makes so much more sense now. I was trying to read the code like I would another language from top down. Knowing the order of operations is an AMAZING help.

Hehe -- I'm glad I could help. This is always the first step of learning any language, but people rarely learn it first. They go straight into syntax. This tripped me up too with Python, so I wanted to make sure I explained it to you in an understandable way. I couldn't find anything on the internet about Python's order of operations or how it executed (explained in a reasonable way), so I had to excavate code just like you in order to finally understand it even remotely. Just figured I'd save you a few steps. :)

https://forum.unity.com/threads/data-oriented-visual-scripting-the-structure-of-a-language.819939/

I'll check this out! I'm actually doing this entire thing for a unity project myself! I would love to tell you about it if you like. In fact...I know this is scope creep, but I was thinking if I could nail this python script, I could maybe somehow figure out a way to get this script functioning inside of Unity and maybe release it as a free asset? All over the internet are people who use vox models and have to import them into Unity, I was thinking if there was a direct asset that could handle that....but right now thats way beyond the scope of what I'm currently intending to do. If you are curious why i want internal voxel structure preserved, my idea was to make a game that uses emergent gameplay. The vision is to use the color palette to apply in game properties to a voxel. (IE brownWood color has properties that would let it ignite) When i build the models in Magicavoxel, I build full models with internal voxels, as I dont want just the surface layer in the game but the internal voxels/data as well. Its a huge scope I know, but I think I have pieced together a development strategy for achieving at least a surface level goal. Here's a screenshot of some of the art so far. My short/medium term goal: Get a tree in a scene inside of unity and burn it down where the fire spreads somewhat organically voxel to voxel through the surface as well as the internal voxels.

https://gyazo.com/a9c73e7577552d695dd9fb6487162f57

That sounds cool! I would love to hear about it!
Feel free to hit me up on Unity's forums if you like. I'm pretty active there. :)

Since there's a lot to get into here though, I'll tackle this first:

Anyway back to the task at hand! I think I found something pointing me towards my goal.
https://imgur.com/TIj5HZd

OK so what I've gathered here is
def expObject(stream, AAQuads):
I think this is the crucial step I needed to find in the code (I've stared at the entire code at this point, for a long time lol) I think uvs = list(uvs) might be the bulk list of voxels indexed within a voxelstruct.
I notice in the code that it is writing to \n' using a stream write. Im wondering if this is some form of data writing that allows it to be written/rewritten as more data is gathered. Im wondering if there is a point where the data that is written is the bulk vertices (maybe the full uvs list?) and then some sort of other data is applied to the \n' file that takes away the internal voxels? I know this is kind of a guess on my part. If there is a point at which all the vertices are exported to a file in bulk, I wonder if this should be where I start editing the code...

Sorry, the data has already been collected by this point. It writes it all in one go. No need for a for loop or anything. The command is enough. By "internal voxels" I think you mean the "internal faces" in this case, since there is no need to write the voxel indexes (and colors) to an .obj file (with actual, renderable, geometry that can be loaded into any 3d application). Again, a 'voxel' is nothing more than a position in an array that is either on or off -- it has no concept of 'sides' -- and in this particular case, the only reason we care about 'sides' is because we need the geometry to render in Blender, for example, so we can modify the mesh (on a mesh level -- not a 'voxel' one). Keep in mind, a 'voxel' has no need for UV information at its core. It simply says "I am here" -- or not. Whatever you do with that information is up to you though -- such as working out what sides are on/off or are a certain color or whatever. The mesh itself (which includes uvs and normals) is a completely separate concept. And since we're exporting an .obj file here, we're exporting the kind of data an .obj file expects -- faces, uvs, and normals.

Sadly, this "expObject()" function is after the program is done translating voxel data to geometry (faces/uvs/normals) and is now translating that geometry into file data and storing it in an actual .obj file -- (to be read in other 3d programs, like Blender). That means this data is simply geometry/uv-mapping data strictly meant for external applications that read .obj file format and is being used to write this data. Thus the "stream" argument as its input -- "stream" as in "file-stream" -- of plain old binary bits and bytes written to a file -- with zero concern about what this data means -- is literally the data currently being written to the hard-drive on your machine right now (when this function is executed of course).

If you want the full voxel data -- that data is literally located in the voxel struct, and requires no further processing (just grab the positions and color after "importing" the vox file and save the position and color data to simple a file). You don't have to worry about what sides are being drawn or anything to get the voxel data.
That said -- one thing that you might have to do (in Unity) is determine how you want to build your meshes there. The problem with using the geometry (instead of the voxels) is that geometry is harder to edit explicitly like this. You'd need something called a meshing algorithm to handle adding/removing voxels. That mesher knows how to build the faces/uvs/normals (and remove them) -- but this would be necessary to have in Unity itself. Reading .vox data directly is a little tougher than converting that to a simpler data format (for use in Unity). This is why I suggested using JSON. If you have a mesher (that is, a tool/script with a 'meshing' algorithm that turns voxel data into an actual mesh in the game while it is running), you can make it take your JSON data for voxels as input, and then turn it into an actual mesh (with the proper sides being shown/hidden). I described how something like this might work in my last message, but sometimes a special shader is required if there's lots of voxels (and performance is a must).
Model-swapping is a low-tech workaround though (i.e. gradually swap between more 'broken' versions of an object to 'animate' them away, using particle effects of voxels flying away in particular places and hitting the ground as the object 'falls apart' to keep the object 'breakdown' appearing 'smooth' -- with the 'choppy' charm of old-school pixel art as it animates away (in fire, for example).

Just something to think about.

https://i.imgur.com/VJOmZln.png
Here is me looking at def importVox(file): and I don't think anything in this needs to be changes for my purposes, but I'm trying to take this slow and look through ALL the code and try to get a good understanding before I start doing surgery on it. I don't quite understand what is so important about XYZI referenced in the code...I think it has something to do with vox files specifically (ephtracy/voxel-model#19) I've actually been googling trying to find a detailed document on how to read a vox file's raw data. Its gibberish if i open it in a notepad. I don't understand what the code means when it talks about chunks. I guess I don't really need to read a vox files raw data...I just want to know how its structured...and I dont quite know what to google to find it out. A question I am not quite sure of is the term chunk, and whether that is something specific to the .vox format or python. I suspect vox.

So the XYZI is probably not implemented in MagicaVoxel yet (my guess is it was an inverted worldspace -- i.e. 'flipped' voxels -- but I could be wrong). The "Chunk" bit can be thought of as kind of a "class" -- That is, just a group of specific data 'shaped' a certain way because it is intended for a particular purpose. In other words, the "RGBA" chunk is just an array of 256 color indexes -- the palette. No other data exists in that particular "chunk" though -- but it could -- and if it did, the IMPORT application (say this python script) would need to account for its size. That size is based on its binary composition -- or in other words -- how many bytes the data consists of.

If you look at that link you provided in the github to MagicaVoxel, you'll notice stuff like "int(32)" and whatnot down toward the bottom of the thread. When you open a .vox file in something like a hex editor, you see the number of bytes being offset. So, since a single byte is 8 bits (i.e. a single letter in hexadecimal), 16 bits are 2 bytes (usually two letters in hexadecimal), and 32 bits are 4 bytes. This means that "int(32)" is represented as 4 bytes (32 bits). This is the key part of how that guy broke down the file format of the .vox file near the bottom of that thread. The .vox file looks like gibberish in a text editor (because we're talking in binary), but in a HEX editor, you can see the specific byte values from top to bottom in the .vox file.

For some extra fun:
In hexadecimal, two bytes go from 00-FF (or for one digit, 0-F, which is 0-16 bits, each with their specific on/off value combination, representing numbers from 0-255). This 256 individual numbers can become 65535 individual numbers with 2 bytes (i.e. 256x256 = 65535). In modern programming languages though, this int(32) -- or 65535 -- is called a "short" because, in general, a standard "integer" goes up to 4 bytes (256x256x256x256 = 4,294,967,296 possible values). However, this requires more space on the file, so a short is usually just fine for most book-keeping purposes (such as counting 3 dimensions of voxels).

https://i.imgur.com/PClknIq.png
This area im the least sure of. I suspect a possible issue is when 2 voxels share the same quad/face, which color is selected for that quad? In my pic I have a blue voxel and a green voxel sharing the same quad. My question is, what color is that quad going to be? I think this section of the code solves this problem, but it might present another one, as I want all the voxels inside to retain their color palette, and if they are all rendered into squares sharing the same vertices this could be a problem...

This section here is only setting the normals (using the cross-product) of the side -- not the colors. The normals are just used for lighting and only matter to the .obj export (to determine whether the back of the face can be seen or not -- and how to 'shade' that face when rendering).

To answer your question about coloring faces though -- I think you might be overthinking it.
Remember, you're dealing with a flat (1 dimensional) array for the sides. Each index is evaluated only once. That is, no side should exist when a voxel is against another voxel (because each individual voxel checks its own surroundings 1 voxel extra in each direction and sets ALL of its 'side' flags accordingly during that same pass. The part you're discussing talking about a green side quad overlapping with another voxel's side quad (on the opposite side) will never happen for the reasons mentioned above. An index is just one index, and all sides are handled during that index's turn to manage its own data (and in this case -- the data is the visibility of its different sides). The reason the "color=0" is used is that empty voxels have a "color" that is set to 0 (that is, they're transparent and invisible -- like 'empty' space), so these voxels don't need any 'side' data. The extra check here is likely redundant, but it's possible it's necessary when no "quads" actually exist on the current voxel (such as when it's 'empty' and therefore no geometry exists to find the 'normal' of in this section -- but I don't remember if it's ever used this way or not).

I was relooking at the def importVox(file): section of the code and I think I found something that is important.

https://i.imgur.com/EoDpqPX.png

I'm thinking vox.setVoxel(Voxel(x, y, z, color)) is like the glue that basically builds everything together. I think it is here that the object that is defined at a voxel (which is an object made up of quads) is associated with its positional and color data within the voxel structure.

So your explanation of how the code runs was a huge huge benefit. I want to go back over the code once again keeping that in mind. Now I can kind of see how this is all working together. Another thing that was throwing me for a loop was I kept seeing voxel and voxels referenced and I wasn't sure how to separate the 2 terms. It was like I know voxel and voxels are 2 separate refences within code, pluralizing a word doesn't do anything to a term codewise but it still was kind of messing with me. But after looking at the importVox section and how it relates to the voxstruct section, it is starting to become clearer I think.

Last but definitely not least -- You are absolutely right on this. And because I got to this message last, I spent a lot of time explaining stuff you probably already know, as it's clear in this image that you got to this without the help of the rest of this post, However, I think I'll leave it all in there anyway, just in case anyone else is trying to follow along and hasn't done as much digging as you have. :D
And yeah, "voxels" typically refers to the index that is actually contained in all voxels, and "voxel" is usually a reference to a specific one in that voxel list. The issue with Python is that an array of data is not very easy to distinguish (in other words, an 'array' in Python is not "array[]" but it is often an "array" (and you assign to it with * or ** typically, which makes it something else -- more like a 'dictionary' -- and not really an 'array' anymore). However, to the uninitiated, passing it in as an argument (without the * or ** before it) makes it seem like you're working with a single value -- but you're not. You're still working with the past references assigned previously (with the * and **) to another function. Now you are working with the result of that operation (and therefore have no need to specify the * or ** before the "voxels" argument, for example.

Finally, the "vox.setVoxel(Voxel(x,y,z,color))" bit is setting the voxel on an INSTANCE of the VoxelStruct class (named "vox" in this example). The "setVoxel()" bit is a function that is part of the "VoxelStruct" class, and any VoxelStruct INSTANCE can call the "setVoxel()" function on itself, since every instance has access to its own class's "def" functions (but not others'). So by creating the "vox" by calling "vox = VoxelStruct(...)", you have created a new INSTANCE of the VoxelStruct class/data definition and are storing that instance in "vox", and now are setting data in that new instance only and not ALL other instances of VoxelStruct that happen to exist (i.e. the "self" argument is referring to the specific INSTANCE of that class/data structure you are working with, and ensures you don't accidentally set the properties/values of other VoxelStruct instances). Before this point, you have only "defined" the VoxelStruct() class with "def" -- but now you have a real instance of that data in the program's memory (and you can do all sorts of stuff with it now). You can get/set data on this new instance, create or destroy other instances of the (VoxelStruct) class, and even write the data you've stored in the instance(s) to a file (such as JSON!) and whatever else you can think of. The "instance" is the core of everything else you're working with. After importing the .vox file, all of your data is stored in this new instance ("vox") of the VoxelStruct class. So use this power wisely. :)

You're doing pretty awesome already, though I hope this helps fill in some of the gaps! 👍

@raavek
Copy link

raavek commented Jan 31, 2021

I haven't given up. I just needed a bit of a breather. Wrote some music (the thing im actually good at) to kind of restructure and regroup. I'll be tackling this soon. I plan on going over the entire script from top to bottom and drawing 1 massive diagram for how I think it is all working.

Ive been getting yt video suggestions for a game called SHENZHEN IO and I've been watching those. I think my brain almost wants to take that visual idea and have one giant image that explains all the pieces that I can then look over. Anyways, random post I know. I've had to read and reread your posts many times to try to come to grips with the abstract nature of this. But I just wanted to say im not done with this by a longshot.

@awesomez
Copy link
Author

awesomez commented Feb 1, 2021

No worries -- I totally get it. When you get into mesh creation, it can very quickly get overwhelming with all the ins-and-outs. However, if what you are wanting is to gather information on each side, what you can do (rather than inspecting the quads themselves), is make your own list of sides and note what voxel index (and therefore what color) these sides belong to.
You can, at least for now, hijack the function and comment-out the stuff you don't need.

And as far as SHENZHEN IO goes -- I've never seen that game before, but it seems cool, despite being a bit more complex than what I use. I use something called "Pencil" for things like mapping out data. It is really good mind-mapping software. Once you learn how to use it, it becomes second-nature for stuff like this. Using Shift to drag a copy of a node and dragging the points on the sides of a rectangle to hook them up with one another is really all it takes. Using this in combination with the Snip tool could really help with code-architecture stuff.

https://pencil.evolus.vn/

Good stuff indeed. :)

@raavek
Copy link

raavek commented Feb 3, 2021

Here is an update of where I am at looking at this code! I will type more when I can but this is my current "living" document I have in GIMP all layered out so I can move things around as they make more sense.

https://i.imgur.com/IAPVxmL.png

@raavek
Copy link

raavek commented Feb 4, 2021 •
edited
Loading

I have so much typed on this image.

https://i.imgur.com/unHUc6Z.png

@awesomez
Copy link
Author

awesomez commented Feb 5, 2021

I have so much typed on this image.

https://i.imgur.com/unHUc6Z.png

You know that stereotypical conspiracy theorist's apartment with red string strung all across the room "connecting" different news articles and photos and whatnot? -- This image is totally that apartment, but digital. 👍

Kidding aside -- Let's start with the * and ** for now.

Think of these as "wildcard characters", rather than "multiply" operators. These "wildcard" chars get 'substituted' for variable 'arguments' instead of strings (i.e. if Windows were Python and you searched folders for ' C:* ', this would return the contents of each and every file on your C: drive and load them into memory, rather than just the location of the files themselves).
The "for x,y in z" statement is essentially a foreach loop in C# -- except a little more 'robust' because it allows you to specify a x (key) and y (value) pair to look for inside of "z" (the data you're looking through). This is different in Python than simply a numerical index of "z" (since it allows it to be used as a dict in Python, whereas in C#, that key/value lookup would have been a more complicated set of statements).

To clarify the * and ** itself though -- the ** simply says there are two parts to "z" (x and y) -- or "key" and "value" pair -- rather than simply a single numerical index (the single * wildcard) that needs to be evaluated for each individual element you're looking for. You pass in the key/value into the function from the function that is running the function that needs the key/value data.
The single wildcard acts as a foreach does in C# (i.e. "foreach 'voxel' in (all) 'voxels', bring in the data being passed in by * through the function / scope that is calling the current function / scope who needs that data, and now apply that data to the particular 'voxel' I am specifying here").

Remember, Python is not my favorite language (due to its convoluted way of abstracting things), so I am not that familiar with how Python handles the memory behind the scenes (as Python is an interpreted language, and the order of operations is not always clear), but the heavy use of the * wildcard characters implies to me that a string reference is replaced behind the scenes with the variable key/value or index you're referring to when it is in use, and only after that, the program interprets it by replacing the * with the actual value. This, however, is a bit of an educated guess. Python, at its core, is still string-based (and therefore what the user means to say has to be interpreted, rather than compiled -- which takes longer at runtime as a trade-off to avoid the lengthy time it would take to compile and test otherwise). Therefore, it is a bit of a Wild-West scenario in that anything can pretty much go as far as syntax -- if the creators of Python deem it useful. I tend to avoid languages like this though, as they can change often (and get rather involved if you get too deep into them!)

So that leads me to my second bit:

I am having just a little trouble following your diagram (so I imagine you might be having a little bit of trouble keeping things in order too). I suggest you use a mind-mapping software like evolus's Pencil (it's free) in order to string together high-level script, class, and function definitions, and contain them in boxes (alongside what arguments link to where). If you're going to code something this complex -- you really do need this kind of software -- badly.

Third:

With what it sounds like you're wanting to do in Unity, it seems like you could just iterate over the voxels that have Quad faces, check which 'side' they're on, and output this data to some script in Unity. This would allow each 'side' or 'surface' to be counted and 'damaged' as the voxel burns away, as the voxel (again) is nothing more than a single numerical index -- and the HP of the individual surfaces are nothing more than the 6 sides of that index that need to be kept track of.
So if you want the entire voxel to disappear (i.e. when all 'sides' are HP=0), you will have to have a voxel display algorithm to handle remeshing the voxels when that voxel is removed. There are free solutions on github for meshing voxels btw (marching cubes algorithm comes to mind) -- but if you want to keep digging at the internals though -- just remember that the OBJ file is only handling quads, and none of this stuff is dynamic at all. You likely won't get much out of studying the quad export process, aside from how (and whether) it optimizes the sides of the quads or not. This script's sole purpose is mainly to export 1x1x1 sides for any voxels that aren't pressing against other voxels. The data you'd need is in the VoxelStruct instancing bit.

I have more to add, but I've got very little time today. However, I wanted to at least mention to keep in mind that we're talking about a script and its class instances (and not particularly the voxels themselves -- as an 'instance' is just the container/structure where the data is stored to be retrieved or modified later). The 'voxels' in general are essentially considered "empty" until we actually 'fill' them with the proper quads to export later. Though, again -- this is not dynamic -- and it seems like you want dynamic (i.e. something to use in Unity, versus something to simply export voxels / faces -- though to get a more simple version of the 'vox' file to work with in Unity is never a bad idea, but even doing this, you only really need the index of the particular voxel to determine the properties of the sides -- and not much else). I think the only bit a .vox has that is 'extra' is the 'normal' which just basically points to which face/direction a voxel has faces. I don't think this is calculated in this script -- MagicaVoxel probably does this for us for each voxel in the grid.

Hopefully that helps for now!

@awesomez
Copy link
Author

I have so much typed on this image.

https://i.imgur.com/unHUc6Z.png

Sorry it has been so long since I got back with you. Life tends to catch up sometimes.
How have things progressed? -- Any luck?

I would respond to the images above, but I'm not entirely sure how much has changed since we last spoke. Therefore I figured I'd ask first (just in case I missed something in my hasty reply previously that you later figured out).

@raavek
Copy link

raavek commented Feb 13, 2021

I messaged you on unity :)

@Chu-4hun
Copy link

Chu-4hun commented May 15, 2021 •
edited
Loading

i have python 3.9, but i cant drop .vox file on .py file

(MagicaVoxel ver 0.99.6.3 win64)

image
image

when i drop the file nothing happens its just not working, can you help please?

P.S. sorry for the localized explorer

@astraldata
Copy link

i have python 3.9, but i cant drop .vox file on .py file

(MagicaVoxel ver 0.99.6.3 win64)

image
image

when i drop the file nothing happens its just not working, can you help please?

P.S. sorry for the localized explorer

Have you tried the one ending in _AUTO as well? (Take a peek at whether the directory your .vox file is in is writeable, as I think you have to specify the directory with the original script, as it appears you're using.)

If that vox_to_obj_AUTO.py script doesn't work, I've also seen this happen when someone has another python version / interpreter / handler installed on their machine as well. Check your "Run as Administrator" and UAC settings too, as it could be a windows issue.
And as silly as it sounds, ensure the .vox file is actually a Magica Voxel .vox file (and not a .vox generated from some other application).

That's about all I can offer at the moment. Hopefully some of that works.

@renjithrv33
Copy link

Is it possible to get the MTL file on conversion. The imported OBJ don't have the MTL file and it loses the colors

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment