Part 2: Creating an OpenGL friendly mesh exporter for Maya

Part 2: Creating an exporter

This is part 2 of a series and it is about getting started with visualizing triangle meshes with Python 2.7 using the libraries PyOpenGL and PyQt4.

Part 1
Part 2
Part 3

I will assume you know python, you will not need a lot of Qt or OpenGL experience, though I will also not go into the deeper details of how OpenGL works. For that I refer you to official documentation and the excellent (C++) tutorials at https://open.gl/. Although they are C++, there is a lot of explanation about OpenGL and why to do certain calls in a certain order.

On a final note: I will make generalizations and simplifications when explaining things. If you think something works different then I say it probably does, this is to try and convey ideas to beginners, not to explain low level openGL implementations.

2.1 File layout

Now that we can draw a model, it is time to define the data that we need to give OpenGL and decide upon a file format that can contain all this data.

Starting off with the elements, we have a way to draw them (GL_TRIANGLES in our case) and we have a data type (GL_UNSIGNED_INT in our case). Given this data type and the number of elements we can actually determine the buffer size regardless of the data type, allowing our file to support not only a number of element values, but allowing those values to be of all the supported types.

Similarly we can look at the vertex layout. We probably want a vertex count and a size of per-vertex-data. This size is a little more complicated because the attribute layout can be very flexible. I suppose it’s easier if we also write the vertex element size instead of trying to figure out what it should be based on the layout.

Then we can look at the attribute layout. We can assume all our data is tightly packed, so we can infer the offset (last argument of glVertexAttribPointer). That leaves us with a layout location, a number of values per vertex, and a data type. Of course first we need to write how many attributes we have.

After that all we need to do is fill in the buffer data. So for vertexCount * vertexElementSize bytes we specify binary data for the vertex buffer and for elementCount * elementDataSize we specify binary data for the elements buffer.

Our file format now looks like this:

Version nr byte So we can change the format later and not break things.
Vertex count unsigned int Because the elements_array can use at most an unsigned int we can never point to vertices beyond the maximum of this data, so no need to store more bytes.
Vertex element size byte Size in bytes of a vertex, based on all attribute sizes combined.
Element count unsigned int
Element data size byte To infer whether indices are unsigned char, unsigned short or unsigned int.
Render type GLenum OpenGL defines variables as GL_TRIANGLES as a GLenum type which is in turn just an unsigned int.
Number of attributes byte
[For each attribute]
Attribute location byte
Attribute dimensions byte Is it a single value, vec2, 3 or 4?
Attribute type GLenum Are the values float, or int, more types listed in the OpenGL documenation for glVertexAttribuPointer.
[End for]
Vertex buffer vertexCount * vertexElementSize bytes
Elements buffer elementCount * elementDataSize bytes

That brings us to the next step, gather this information in Maya.

2.2 Maya mesh exporter

To export I’ll use the maya API (OpenMaya). It provides a way to quickly iterate over a mesh’ data without allocating too much memory using the MItMeshPolygons. This will iterate over all the faces and allow us to extract the individual triangles and face vertices.

There are a few steps to do. First let’s make a script to generate a test scene:

from maya import cmds
from maya.OpenMaya import *
cmds.file(new=True, force=True)
cmds.polyCube()
meshShapeName = 'pCubeShape1'
outputFilePath = 'C:/Test.bgm'

Now with these variables in mind we have to convert the shape name to actual maya API objects that we can read data from.

# get an MDagPath from the given mesh path
p = MDagPath()
l = MSelectionList()
MGlobal.getSelectionListByName(mayaShapeName, l)
l.getDagPath(0, p)

# get the iterator
poly = MItMeshPolygon(p)

This sets us up to actually start saving data. Because openGL requires us to provide all the data of a vertex at 1 vertex index we have to remap some of Maya’s data. In Maya a vertex (actually face-vertex in Maya terms) is a list of indices that points to e.g. what vertex to use, what normal to use, etc. All with separate indices. In OpenGL all these indices must match. The way I’ll go about this is to simply take the triangulation and generate 3 unique vertices for each triangle. This means that to find the vertex count can be determined by counting the triangles in the mesh. Maya meshes don’t expose functionality to query this , so instead I’ll iterate over all the faces and count the triangles in them.

# open the file as binary
with open(outputFilePath, 'wb') as fh:
    # fixing the vertex data size to just X, Y, Z floats for the vertex position
    vertexElementSize = 3 * ctypes.sizeof(ctypes.c_float)
  
    # using unsigned integers as elements
    indexElementSize = ctypes.sizeof(ctypes.c_uint)
  
    # gather the number of vertices
    vertexCount = 0
    while not poly.isDone():
        vertices = MPointArray()
        vertexList = MIntArray()
        poly.getTriangles(vertices, vertexList, space)
        vertexCount += vertexList.length()
        poly.next()
    poly.reset()
    # start writing
    fh.write(struct.pack('B', FILE_VERSION))
    fh.write(struct.pack('I', vertexCount))
    fh.write(struct.pack('B', vertexElementSize))
    # currently I'm duplicating all vertices per triangle, so total indices matches    total vertices
    fh.write(struct.pack('I', vertexCount))
    fh.write(struct.pack('B', indexElementSize))
    fh.write(struct.pack('I', GL_TRIANGLES))  # render type

As you can see we had to make some assumptions about vertex data size and we had to gather some intel on our final vertex count, but this is a good setup. Next step is to write the attribute layout. I’ve made the assumption here to write only X Y Z position floats at location 0. We can expand the exporter later with more features, as our file format supports variable attribute layouts. We can write our position attribute next:

# attribute layout
# 1 attribute
fh.write(struct.pack('B', 1))
# at location 0
fh.write(struct.pack('B', 0))
# of 3 floats
fh.write(struct.pack('B', 3))
fh.write(struct.pack('I', GL_FLOAT))

Note that I am using a constant GL_FLOAT here, if you do not wish to install PyOpenGL for your maya, you can quite simply include this at the top of the file instead:

import ctypes
GL_TRIANGLES = 0x0004
GL_UNSIGNED_INT = 0x1405
GL_FLOAT = 0x1406

After that comes streaming the vertex buffer. For this I use the same iterator I used to count the vertex count. The code is pretty much the same only now I write the vertices instead of counting the vertex list.

# iter all faces
while not poly.isDone():
    # get triangulation of this face
    vertices = MPointArray()
    vertexList = MIntArray()
    poly.getTriangles(vertices, vertexList, space)
  
    # write the positions
    for i in xrange(vertexList.length()):
        fh.write(struct.pack('3f', vertices[i][0], vertices[i][1], vertices[i][2]))
  
    poly.next()

Last is the element buffer.

# write the elements buffer
for i in xrange(vertexCount):
   fh.write(struct.pack('I', i))

2.3 All the data

The next step naturally is to export more than just the position. Here is a more elaborate way to extract all the attributes. First we need to get some global data from the mesh. This goes right after where we create the MItMeshPolygons.

fn = MFnMesh(p)
tangents = MFloatVectorArray()
fn.getTangents(tangents, space)
colorSetNames = []
fn.getColorSetNames(colorSetNames)
uvSetNames = []
fn.getUVSetNames(uvSetNames)

Next we have to change our vertexElementSize code to the following:

# compute the vertex data size, write 4 floats for the position for more convenient transformation in shaders
# position, tangent, normal, color sets, uv sets
vertexElementSize = (4 + 3 + 3 + 4 * len(colorSetNames) + 2 * len(uvSetNames)) * ctypes.sizeof(ctypes.c_float)

The attribute layout is significantly changed. I’m also changing the point data from a vec3 to a vec4. I’m filling in the w component as 1.0, this to indicate a point instead of a vector. It will make transforming vertices in shaders a step simpler.

# attribute layout

# Since NVidia is the only driver to implement a default attribute layout I am following this as much as possible
# on other drivers using a custom shader is mandatory and modern buffers will never work with the fixed function pipeline.
# http://developer.download.nvidia.com/opengl/glsl/glsl_release_notes.pdf
# https://stackoverflow.com/questions/20573235/what-are-the-attribute-locations-for-fixed-function-pipeline-in-opengl-4-0-cor

# num attributes
fh.write(struct.pack('B', 3 + len(colorSetNames) + len(uvSetNames)))
# vec4 position at location 0
fh.write(struct.pack('B', 0))
fh.write(struct.pack('B', 4))
fh.write(struct.pack('I', GL_FLOAT))
# vec3 tangent at location 1
fh.write(struct.pack('B', 1))
fh.write(struct.pack('B', 3))
fh.write(struct.pack('I', GL_FLOAT))
# vec3 normal at location 2
fh.write(struct.pack('B', 2))
fh.write(struct.pack('B', 3))
fh.write(struct.pack('I', GL_FLOAT))
# vec4 color at locations (3,7) and 16+
used = {}
for i in xrange(len(colorSetNames)):
    idx = 3 + i
    if idx > 7:
        idx = 11 + i
        used.add(idx)
    fh.write(struct.pack('B', idx))
    fh.write(struct.pack('B', 4))
    fh.write(struct.pack('I', GL_FLOAT))
# vec2 uvs at locations 8-15 and 16+, but avoiding overlap with colors
idx = 8
for i in xrange(len(uvSetNames)):
    while idx in used:
        idx += 1
    fh.write(struct.pack('B', idx))
    fh.write(struct.pack('B', 2))
    fh.write(struct.pack('I', GL_FLOAT))
    idx += 1

Most of the MItMeshPolygon iterator functions, like getNormals(), gives us a list of the normals for all vertices in this face. The problem is that this data is not triangulated.

To extract the triangulation we used getTriangles(), which gives us a list of vertices used in the face. These vertex numbers are object-wide, so they keep getting bigger the further we get.

That means they’re useless if we want to use them to look up the normal returned by getNormals(), because that array is always very short, containing just the normals for this face.

So we have to do some mapping from the triangulated vertex indices into indices that match the data we’ve got. Either that or get all the normals from the mesh in 1 big array but that is not memory efficient. So at the top of the while loop (just inside) I’ve added the following dictionary:

# map object indices to local indices - because for some reason we can not query the triangulation as local indices
# but all getters do want us to provide local indices
objectToFaceVertexId = {}
count = poly.polygonVertexCount()
for i in xrange(count):
    objectToFaceVertexId[poly.vertexIndex(i)] = i

That allows us to extract all the data we want for these triangles like so:

# get per-vertex data
normals = MVectorArray()
poly.getNormals(normals, space)
colorSet = []
for i, colorSetName in enumerate(colorSetNames):
    colorSet.append(MColorArray())
    poly.getColors(colorSet[i], colorSetName)
uvSetU = []
uvSetV = []
for i, uvSetName in enumerate(uvSetNames):
    uvSetU.append(MFloatArray())
    uvSetV.append(MFloatArray())
    poly.getUVs(uvSetU[i], uvSetV[i], uvSetName)

Handling fairly small sets of data at a time. Last we have to write the data, replacing the loop writing 3 floats per vertex we had before with this longer loop:

# write the data
for i in xrange(vertexList.length()):
    localVertexId = objectToFaceVertexId[vertexList[i]]
    tangentId = poly.tangentIndex(localVertexId)
  
    fh.write(struct.pack('4f', vertices[i][0], vertices[i][1], vertices[i][2], 1.0))
    fh.write(struct.pack('3f', tangents[tangentId][0], tangents[tangentId][1], tangents[tangentId][2]))
    fh.write(struct.pack('3f', normals[localVertexId][0], normals[localVertexId][1], normals[localVertexId][2]))
    for j in xrange(len(colorSetNames)):
        fh.write(struct.pack('4f', colorSet[j][localVertexId][0], colorSet[j][localVertexId][1], colorSet[j][localVertexId][2], colorSet[j][localVertexId][3]))
    for j in xrange(len(uvSetNames)):
        fh.write(struct.pack('2f', uvSetU[j][localVertexId], uvSetV[j][localVertexId]))

And that completes the exporter with full functionality, extracting all possible data from a maya mesh we want. Unless you want blind data and skin clusters, but that’s a whole different story!

2.4 Code

Here is the final code as a function, with an additional function to export multiple selected meshes to multiple files, using Qt for UI. Note that if you wish to use PySide or PyQt5 instead the QFileDialog.getExistingDirectory and QSettings.value return types are different and require some work.

import os
import struct
from maya import cmds
from maya.OpenMaya import *
import ctypes

GL_TRIANGLES = 0x0004
GL_UNSIGNED_INT = 0x1405
GL_FLOAT = 0x1406
FILE_EXT = '.bm'  # binary mesh
FILE_VERSION = 0
EXPORT_SPACE = MSpace.kWorld  # export meshes in world space for now


def exportMesh(mayaShapeName, outputFilePath, space):
    # get an MDagPath from the given mesh path
    p = MDagPath()
    l = MSelectionList()
    MGlobal.getSelectionListByName(mayaShapeName, l)
    l.getDagPath(0, p)
  
    # get the mesh and iterator
    fn = MFnMesh(p)
    poly = MItMeshPolygon(p)
  
    tangents = MFloatVectorArray()
    fn.getTangents(tangents, space)
    colorSetNames = []
    fn.getColorSetNames(colorSetNames)
    uvSetNames = []
    fn.getUVSetNames(uvSetNames)
  
    # open the file as binary
    with open(outputFilePath, 'wb') as fh:
        # compute the vertex data size, write 4 floats for the position for more convenient transformation in shaders
        # position, tangent, normal, color sets, uv sets
        vertexElementSize = (4 + 3 + 3 + 4 * len(colorSetNames) + 2 * len(uvSetNames)) * ctypes.sizeof(ctypes.c_float)
  
        # using unsigned integers as elements
        indexElementSize = ctypes.sizeof(ctypes.c_uint)
  
        # gather the number of vertices
        vertexCount = 0
        while not poly.isDone():
            vertices = MPointArray()
            vertexList = MIntArray()
            poly.getTriangles(vertices, vertexList, space)
            vertexCount += vertexList.length()
            poly.next()
        poly.reset()
  
        # start writing
        fh.write(struct.pack('B', FILE_VERSION))
        fh.write(struct.pack('I', vertexCount))
        fh.write(struct.pack('B', vertexElementSize))
        # currently I'm duplicating all vertices per triangle, so total indices matches total vertices
        fh.write(struct.pack('I', vertexCount))
        fh.write(struct.pack('B', indexElementSize))
        fh.write(struct.pack('I', GL_TRIANGLES))  # render type
  
        # attribute layout
  
        # Since NVidia is the only driver to implement a default attribute layout I am following this as much as possible
        # on other drivers using a custom shader is mandatory and modern buffers will never work with the fixed function pipeline.
        # http://developer.download.nvidia.com/opengl/glsl/glsl_release_notes.pdf
        # https://stackoverflow.com/questions/20573235/what-are-the-attribute-locations-for-fixed-function-pipeline-in-opengl-4-0-cor
  
        # num attributes
        fh.write(struct.pack('B', 3 + len(colorSetNames) + len(uvSetNames)))
        # vec4 position at location 0
        fh.write(struct.pack('B', 0))
        fh.write(struct.pack('B', 4))
        fh.write(struct.pack('I', GL_FLOAT))
        # vec3 tangent at location 1
        fh.write(struct.pack('B', 1))
        fh.write(struct.pack('B', 3))
        fh.write(struct.pack('I', GL_FLOAT))
        # vec3 normal at location 2
        fh.write(struct.pack('B', 2))
        fh.write(struct.pack('B', 3))
        fh.write(struct.pack('I', GL_FLOAT))
        # vec4 color at locations (3,7) and 16+
        used = {}
        for i in xrange(len(colorSetNames)):
            idx = 3 + i
            if idx > 7:
                idx = 11 + i
                used.add(idx)
            fh.write(struct.pack('B', idx))
            fh.write(struct.pack('B', 4))
            fh.write(struct.pack('I', GL_FLOAT))
        # vec2 uvs at locations 8-15 and 16+, but avoiding overlap with colors
        idx = 8
        for i in xrange(len(uvSetNames)):
            while idx in used:
                idx += 1
            fh.write(struct.pack('B', idx))
            fh.write(struct.pack('B', 2))
            fh.write(struct.pack('I', GL_FLOAT))
            idx += 1
  
        # iter all faces
        while not poly.isDone():
            # map object indices to local indices - because for some reason we can not query the triangulation as local indices
            # but all getters do want us to provide local indices
            objectToFaceVertexId = {}
            count = poly.polygonVertexCount()
            for i in xrange(count):
                objectToFaceVertexId[poly.vertexIndex(i)] = i
  
            # get triangulation of this face
            vertices = MPointArray()
            vertexList = MIntArray()
            poly.getTriangles(vertices, vertexList, space)
  
            # get per-vertex data
            normals = MVectorArray()
            poly.getNormals(normals, space)
            colorSet = []
            for i, colorSetName in enumerate(colorSetNames):
                colorSet.append(MColorArray())
                poly.getColors(colorSet[i], colorSetName)
            uvSetU = []
            uvSetV = []
            for i, uvSetName in enumerate(uvSetNames):
                uvSetU.append(MFloatArray())
                uvSetV.append(MFloatArray())
                poly.getUVs(uvSetU[i], uvSetV[i], uvSetName)
  
            # write the data
            for i in xrange(vertexList.length()):
                localVertexId = objectToFaceVertexId[vertexList[i]]
                tangentId = poly.tangentIndex(localVertexId)
  
                fh.write(struct.pack('4f', vertices[i][0], vertices[i][1], vertices[i][2], 1.0))
                fh.write(struct.pack('3f', tangents[tangentId][0], tangents[tangentId][1], tangents[tangentId][2]))
                fh.write(struct.pack('3f', normals[localVertexId][0], normals[localVertexId][1], normals[localVertexId][2]))
                for j in xrange(len(colorSetNames)):
                    fh.write(struct.pack('4f', colorSet[j][localVertexId][0], colorSet[j][localVertexId][1], colorSet[j][localVertexId][2], colorSet[j][localVertexId][3]))
                for j in xrange(len(uvSetNames)):
                    fh.write(struct.pack('2f', uvSetU[j][localVertexId], uvSetV[j][localVertexId]))
  
            poly.next()
  
        # write the elements buffer
        for i in xrange(vertexCount):
            fh.write(struct.pack('I', i))


def exportSelected():
    selectedMeshShapes = cmds.select(ls=True, type='mesh', l=True) or []
    selectedMeshShapes += cmds.listRelatives(cmds.select(ls=True, type='transform', l=True) or [], c=True, type='mesh', f=True) or []
    from PyQt4.QtCore import QSettings
    from PyQt4.QtGui import QFileDialog
    settings = QSettings('GLMeshExport')
    mostRecentDir = str(settings.value('mostRecentDir').toPyObject())
    targetDir = QFileDialog.getExistingDirectory(None, 'Save selected meshes in directory', mostRecentDir)
    if targetDir and os.path.exists(targetDir):
        settings.setValue('mostRecentDir', targetDir)
        for i, shortName in enumerate(cmds.ls(selectedMeshShapes)):
            exportMesh(selectedMeshShapes[i],
                       os.path.join(targetDir, shortName.replace('|', '_'), FILE_EXT),
                       EXPORT_SPACE)

Leave a Reply

Your email address will not be published. Required fields are marked *