Detecting wire color in Maya

When creating a shape node with Maya’s API in the draw event you simply get the state of the object. Sadly, this can never be retrieved anywhere else (unless we’d override all Maya nodes to have them store the value somewhere). After a long search I found no way of replicating what Maya does before drawing a node, so I had to come up with a different method.

When determining the color of an object’s wireframe there’s all kind of inÎfluences. Is it:
>selected
>a lead selection
>templated
>referenced
>in a layer which is templated or referenced
>does it have an override color set
>does it have a layer with a color
>does it have a parent with an override color
and most hated of all:
>is it purple? (affected by a selected object)

Now luckily layers drive the overridesEnabled, overrideColor and overrideDisplayType attributes, so we don’t really have to worry about those.

An important part is determining in which order of importance these colors are determined. Essentially templating is most important

>objects turn orange when selected and templated simultanously
>green when selected as last (lead)
>white when selected
>purple when influenced by other selected objects (affected)
>gray when templated
>black when referenced
>overrideColor when enableOverrides is True
>blue otherwise

All these properties are inherited from parents as well. So when a referenced object has a drawingOverride, it is still black, when an object is affected by another selected object but also selected it will still be green (or white). When an object’s parent is templated, the object itself appears templated, etcetera.

Do realize this only applies to shapes, as they are the only objects actually being drawn!

The next problem once we know all this information, is determining what colour links to that info. There’s the displayColor and displayRGBColor commands for that, and lucky for use, they have a list feature. So printing each entry and then reading for quite a while we find out the names of the attributes (which mostly match those in the Window -> Settings/preferences -> Color Settings but not always).

So some colors can be set freely, such as the template color. Other colors can only be set to certain indices, displayColor returns a number and we’ll have to use the colorIndex command to get to the actual color. We could hardcore the colors, but then the result is not matching the display if the user changes his settings.

So let’s start with the most basic scenario, a given node’s child shape’s dormant color. Here we run immediately into the next issue, every shape type can have it’s own deselected and selected color and most of the names do not match the nodeType name. For example a measure node is of type distanceDimShape and it’s color needs to be retrieved as ‘dimension’. So here’s a partial list of name conversion:

def displayColorType(inObj):
    objtype = cmds.nodeType(inObj)
    if objtype == 'nurbsSurface':
        trims = cmds.listConnections(shape, s=True, d=False, type='planarTrimSurface')
        if trims:
            obtype = 'trimmedSurface'
        else:
            objtype = 'surface'
    if objtype == 'nurbsCurve':
        projectCurves = cmds.listConnections(shape, s=True, d=False. type='projectCurve')
        if projectCurves:
            objtype = 'curveOnSurface'
        else:
            objtype = 'curve'
    if objtype == 'mesh':
        objtype = 'polymesh'
    if objtype == 'joint' and cmds.listRelatives(shape, ad=True, type='effector'):
        objtype = 'segment'
    if objtype == 'cluster':
        objtype = 'locator'
    if objtype == 'distanceDimShape':
        objtype = 'dimension'
    return objtype

This function is not limited to shapes, but it will result in errors when you attempt to use

cmds.displayColor(displayColorType(),q=True)

on a transform node.

So I assume this to be all I need but the list may get longer when it turns out certain objects try to get their color by the wrong name. Now let’s check whether the object is selected or not and return either the dormant or the active color related to its type:

def drawColor(inObj):
    shapes = cmds.listRelatives(inObj, ad=True, type='shape', f=True)
    if not shapes:
        if cmds.nodeType(inObj) != 'transform':
            shape = inObj
        else: #transform node without shapes has no color
            return None
    else:
        shape = shapes[0]

    nodetype = displayColorType( shape )
    if shape in cmds.ls(sl=True,l=True):
        return cmds.colorIndex( cmds.displayColor(nodetype, q=True, active=True), q=True )
    return cmds.colorIndex( cmds.displayColor(nodetype, q=True, dormant=True), q=True )

print( drawColor( cmds.polyCube()[0] ) )

Now this will immediately print the wrong color, as the parent is selected and not the shape.

#result: [1.0, 1.0, 1.0]

So let’s solve that bit with the following function:

def isParentSelected(inObj):
    selection = cmds.ls(sl=True, l=True)
    target = cmds.ls(inObj, l=True)[0] #ensure full path
    while target:
        if target in selection:
            return True
        target = cmds.listRelatives(target, p=True, f=True)[0]
    return False

Now in drawColor on line 11 instead of using

if shape in cmds.ls(sl=True, l=True)

I will use

if isParentSelected(shape):

Now it returns the the selected color, which is white.

#result: [1.0, 1.0, 1.0]

But our object is in the lead, so it is green. So with some modifications this isn’t too difficult. Let the isParentSelected return the parent (or None) instead of True. I also fixed the part where I forgot to check if we had a selection, ls and listRelatives return None instead of an empty list if there is no result, so we get problems at ‘target in selection’ if there is no selection.

def isParentSelected(inObj, ignoreSelf=False):
    selection = cmds.ls(sl=True, l=True)
    if not selection: #no selection, no result
        return
    if not ignoreSelf:
        if inObj in selection:
            return inObj
    targets = cmds.listRelatives(inObj, ap=True, f=True)
    if not targets:
        return
    for target in targets:
        if target in selection:
            return target
    return

Then the last bit of drawColor becomes this:

    nodetype = displayColorType( shape )
    selected = isParentSelected( shape )
    if selected:
        if selected == cmds.ls(os=True, l=True)[-1]:
            return cmds.colorIndex( cmds.displayColor('lead', q=True, active=True), q=True )
        return cmds.colorIndex( cmds.displayColor(nodetype, q=True, active=True), q=True )
    return cmds.colorIndex( cmds.displayColor(nodetype, q=True, dormant=True), q=True )

The ls function returns the ordered selection, ensuring that the last entry is indeed the lead entry and we use the active lead displayColor isntead of the active nodetype displayColor. The lead color can also be customised, but not per object type.

#result: [0.2630000114440918, 1.0, 0.63899999856948853]

Now this post is getting rather long, so more on this later, where I’ll have a look into overrides.

Leave a Reply

Your email address will not be published. Required fields are marked *