Use property collector to retrieve names quickly

gistfile1.txt

#!/usr/bin/env python
"""
Written by Nathan Prziborowski
Github: https://github.com/prziborowski
This code is released under the terms of the Apache 2
http://www.apache.org/licenses/LICENSE-2.0.html
The property collector can be used to fetch a subset of properties
for a large amount of objects with fewer round trips that iterating.
This sample shows how to use the TraversalSpec to get properties
of another object without multiple calls.
"""
import sys
from pyVmomi import vim, vmodl
from pyVim.connect import SmartConnectNoSSL, Disconnect
from pyVim.task import WaitForTask
from tools import cli

__author__ = 'prziborowski'


def setup_args():
    parser = cli.build_arg_parser()
    return cli.prompt_for_password(parser.parse_args())


def get_obj(si, root, vim_type):
    container = si.content.viewManager.CreateContainerView(root, vim_type,
                                                           True)
    view = container.view
    container.Destroy()
    return view


def get_filter_spec(containerView, objType, path):
    traverse_spec = vmodl.query.PropertyCollector.TraversalSpec()
    traverse_spec.name = 'traverse'
    traverse_spec.path = 'view'
    traverse_spec.skip = False
    traverse_spec.type = vim.view.ContainerView

    obj_spec = vmodl.query.PropertyCollector.ObjectSpec()
    obj_spec.obj = containerView
    obj_spec.skip = True
    obj_spec.selectSet.append(traverse_spec)

    prop_spec = vmodl.query.PropertyCollector.PropertySpec()
    prop_spec.type = objType
    prop_spec.pathSet = path

    return vmodl.query.PropertyCollector.FilterSpec(propSet=[prop_spec],
                                                    objectSet=[obj_spec])

def process_result(result, objects):
    for o in result.objects:
        if o.obj not in objects:
            objects[o.obj] = {}
        for p in o.propSet:
            objects[o.obj][p.name] = p.val

def collect_properties(si, root, vim_type, props):
    objects = {}
    # Start with all the VMs from container, which is easier to write than
    # PropertyCollector to retrieve them.
    view_mgr = si.content.viewManager
    container = view_mgr.CreateContainerView(root,
                                             [vim_type], True)
    try:
        filter_spec = get_filter_spec(container, vim_type, props)
        options = vmodl.query.PropertyCollector.RetrieveOptions()
        pc = si.content.propertyCollector
        result = pc.RetrievePropertiesEx([filter_spec], options)
        process_result(result, objects)
        while result.token is not None:
            result = pc.ContinueRetrievePropertiesEx(result.token)
            process_result(result, objects)
    finally:
        container.Destroy()
    return objects


def main():
    args = setup_args()
    si = SmartConnectNoSSL(host=args.host,
                           user=args.user,
                           pwd=args.password,
                           port=args.port)
    vms = collect_properties(si, si.content.rootFolder, vim.VirtualMachine,
                             ['config', 'name', 'guest', 'parent', 'runtime'])
    hosts = collect_properties(si, si.content.rootFolder, vim.HostSystem,
                               ['name'])
    print("VMs: %d" % len(vms))
    print("Hosts: %d" % len(hosts))

# Start program
if __name__ == "__main__":
    main()

When I have more than 1000 virtual machines in single vCenter and need to get data from multiple attributes(such as: vim.VirtualMachine, ['config', 'name', 'guest', 'parent', 'runtime'] and vim.HostSystem, ['name']), I find this method is still very slow. Maybe 15minutes. Can you tell me how to make it bettter? Thanks!

Can you share the example of how you modified this to collect both objects and I'd presume the 'runtime.host.name' for vim.HostSystem?
Retrieving those properties from only 1000 VMs should still be doable in seconds at the worst.

I see your post at vmware/pyvmomi#851. I'll take a look at that later today and see if there is anything problematic there.

I wasn't able to run your script as it was missing parts. But from what I can gather of it you are getting a container view of all objects on VC? That could be slower than just getting 2 container views of vim.VirtualMachine and vim.HostSystem and doing separate property collector calls on each.
I updated the gist with that example and for 950 VMs and 36 hosts it runs in 13 seconds.

def parse_propspec(propspec):
    """Parses property specifications

    :param propspec: the property specifications need to be parses,
        '{'VirtualMachine': ['name']}' for example
    :return: a sequence of 2-tuples. each containing a managed object type
    and a list of properties applicable to that type

    useage:
        propspec = {
            'VirtualMachine': ['name'],
            'Datastore': ['name']
        }
        properties = parse_propspec(propspec)
    """
    props = []
    for objtype, objprops in propspec.items():
        motype = getattr(vim, objtype, None)
        if motype is None:
            raise vCenterPropertyNotExist(motype)
        props.append((motype, objprops,))
    return props

Yes! I use the function to deal with many container views, I later updated my method with your gist example and the effect is remarkable.
At the same time, I change my method again, Use a single container view to get the corresponding data.
Thank you very much for solving one of my problems.
What's the principle of your gist with that example? expecially fiter_spec.

Thanks again!

The filter_spec part is mainly showing how to traverse from one object (ContainerView) to another (VirtualMachine/HostSytem) through looking at the 'view' property of ContainerView.
I still need to learn a bit more about how to do that with multiple types of objects at once 😄

You can use retrievePropertiesEx to fetch config.name for all vms and this is pretty fast.