Skip to content

Instantly share code, notes, and snippets.

@jriguera

jriguera/00_openstack_vmware.md

Last active February 5, 2019 13:52
Show Gist options
  • Save jriguera/272518633da0ba9edff1 to your computer and use it in GitHub Desktop.
Save jriguera/272518633da0ba9edff1 to your computer and use it in GitHub Desktop.
Download ZIP
OpenStack and VMware integration
Raw
00_openstack_vmware.md

This guide reflects all the steps needed to get Vmware integrated within OpenStack (VMware vCenter version: 5.1 and OpenStack Liberty)

Official documentation available:

http://docs.openstack.org/liberty/config-reference/content/vmware.html http://docs.openstack.org/liberty/config-reference/content/list-of-compute-config-options.html#config_table_nova_vmware

Just follow the steps available in the links above.

Code: https://github.com/openstack/nova/tree/stable/liberty/nova/virt/vmwareapi

Notes:

  • VMware hypervisor is implemented as a different availability zone (AZ). IMO, that is the best way to do it, because vmware cluster normally has DRS enabled and stretched over AZ (nearby DCs). Nova-compute VMware hypervisor sees all the resources on the cluster, from OpenStack point of view, a VMware cluster is just another hypervisor with a lot of resources (CPUs, memory, disk ...). With KVM hypervisor with 2 AZ, an user (from a specific project) could select where to deploy a VM by choosing 3 over availability zones. It is possible to filter which AZ a project can see by defining metadata on the AZ.
  • This implementation is using Neutron with external (provider) networks only, so it is not needed to use other neutron ML2 mechanism plugins for OpenStack. Neutron is not aware of VMware and it does not use drivers to manage VMware networks, it just reserve IPs (DHCP) and provide access to metadata. For a full OpenStack integration you could use OVSvApp (https://github.com/hp-networking/ovsvapp/blob/master/OVSvApp_Solution.pdf) which is free (or VMware-NSX6 which cost money but adds a lot of flexibility). Nova-compute is in charge of doing the basic networking stuff in VMware (VIF integration) by linking a VM with pre-defined portgroups in specific (dv)switches.
  • In the same way as Neutron, Cinder (persistent block storage) and Glance (image service) are not integrated with VMware, that means they do not know anything about it. If somebody tries to attach a volume to a VM running on VMware AZ, it will fail. Glance does not need to be integrated with VMware, only is needed if you want to use a vwmare datastore to save the images.
  • Given the fact that the VMware cluster has HA and DRS enabled, the nova compute hypervisor server can be running inside the VMware cluster as a VM. With this setup, it is not needed to create a pacemaker setup to maintain the HA (only one compute node). Also the server needs to run the ML2 neutron agent mechanism (it is a neutron requirement, even in this setup -with external networks only- is not needed). To sum up, the idea is running a VM within the VM cluster acting as nova-compute and the neutron agent (linuxbridge ML2 agent in this case). (This solution is becoming equivalent like the VMware OpenStack appliance!).
  • It is needed an VMware user to perform all the operations with the privileges defined in the official documentation. Nova-compute VMware needs to get the definition of the user with the format user@domain otherwise it cannot get permissions for all the objects in VMware but it is able to log in in VCenter (weird!).

To enable VNC ports to see the console of vmware VMs:

  • Show current rules of firewall: esxcli network firewall ruleset list
  • Update the exsi firewall rules for VNC:
    • https://github.com/openstack-vmwareapi-team/Tools
    • Copy zip update to vmfs datastore.
    • Change vib acceptance level partent: esxcli software acceptance set --level=CommunitySupported
    • Install it: esxcli software vib install -d "/vmfs/volumes/datastore1/openstackvncfirewall.zip"
Raw
01_nova.conf
[DEFAULT]
# Logs / State
debug = True
verbose = True
fatal_deprecations = False
log_dir = /var/log/nova
state_path = /var/lib/nova
rootwrap_config = /etc/nova/rootwrap.conf
api_paste_config = /etc/nova/api-paste.ini
service_down_time = 120
# Quota (optional)
quota_cores = 20
quota_fixed_ips = -1
quota_floating_ips = 10
quota_injected_file_content_bytes = 10240
quota_injected_file_path_length = 255
quota_injected_files = 5
quota_instances = 10
quota_key_pairs = 100
quota_metadata_items = 128
quota_networks = 3
quota_ram = 51200
quota_security_group_rules = 20
quota_security_groups = 10
quota_server_group_members = 10
quota_server_groups = 10
cpu_allocation_ratio = 2.0
disk_allocation_ratio = 1.0
max_instances_per_host = 50
max_io_ops_per_host = 10
ram_allocation_ratio = 1.0
ram_weight_multiplier = 5.0
reserved_host_disk_mb = 0
reserved_host_memory_mb = 0
# Driver
compute_driver = vmwareapi.VMwareVCDriver
instances_path = /var/lib/nova/instances
allow_resize_to_same_host = True
image_cache_manager_interval = 0
resume_guests_state_on_host_boot = False
# Api's
enabled_apis = osapi_compute,metadata
osapi_compute_workers = 20
# Rpc all
rpc_backend = nova.openstack.common.rpc.impl_kombu
executor_thread_pool_size = 64
rpc_conn_pool_size = 30
rpc_response_timeout = 60
instance_build_timeout = 0
timeout_nbd = 120
# Metadata
metadata_host = x.x.x.x
metadata_port = 8775
metadata_workers = 20
# Network
dhcp_domain = openstacklocal
force_dhcp_release = True
dhcpbridge_flagfile = /etc/nova/nova.conf
firewall_driver = nova.virt.firewall.NoopFirewallDriver
my_ip = x.x.x.x
security_group_api = neutron
network_api_class = nova.network.neutronv2.api.API
# Authentication
auth_strategy = keystone
## Vif
linuxnet_interface_driver = nova.network.linux_net.NeutronLinuxBridgeInterfaceDriver
libvirt_vif_type = ethernet
vif_plugging_timeout = 10
vif_plugging_is_fatal = False
# Configdrive
force_config_drive = True
# Policy
max_age = 0
# Common
memcached_servers = x.x.x.x:11211
[cinder]
catalog_info = volumev2:cinderv2:internalURL
cross_az_attach = True
[vmware]
host_ip = x.x.x.x
host_username = "[email protected]"
host_password = "XXXXXXXX"
cluster_name = "OpenStack"
datastore_regex = "openstack"
integration_bridge = "brq383fa876-8d"
insecure=True
[spice]
agent_enabled = True
enabled = True
keymap = en-us
# Console Url and binds
html5proxy_base_url = http://x.x.x.x:6082/spice_auto.html
server_listen = x.x.x.x
server_proxyclient_address = x.x.x.x
[vnc]
enabled = False
[glance]
host = x.x.x.x
port = 9292
[neutron]
url = http://x.x.x.x:9696
region_name = RegionOne
auth_plugin = password
# Keystone client plugin password option
password = adfff3f0fcc1f339sdfa416dsfc
# Keystone client plugin username option
username = neutron
project_name = service
user_domain_name = Default
project_domain_name = Default
# Keystone client plugin authentication URL option
auth_url = http://x.x.x.x:35357/v3
metadata_proxy_shared_secret = 433dfsdad49dfg8ydu1hjgj4
service_metadata_proxy = True
[conductor]
use_local = false
topic = conductor
manager = nova.conductor.manager.ConductorManager
workers = 20
Raw
02_import_oslo_vmware.md

Python does not know the modules, it cannot import the nova driver:

2015-11-18 09:54:52.959 11853 INFO nova.virt.driver [-] Loading compute driver 'vmwareapi.VMwareVCDriver'
2015-11-18 09:54:52.963 11853 ERROR nova.virt.driver [-] Unable to load the virtualization driver
2015-11-18 09:54:52.963 11853 ERROR nova.virt.driver Traceback (most recent call last):
2015-11-18 09:54:52.963 11853 ERROR nova.virt.driver   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/virt/driver.py", line 1454, in load_compute_driver
2015-11-18 09:54:52.963 11853 ERROR nova.virt.driver     virtapi)
2015-11-18 09:54:52.963 11853 ERROR nova.virt.driver   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_utils/importutils.py", line 52, in import_object_ns
2015-11-18 09:54:52.963 11853 ERROR nova.virt.driver     return import_class(import_str)(*args, **kwargs)
2015-11-18 09:54:52.963 11853 ERROR nova.virt.driver   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_utils/importutils.py", line 27, in import_class
2015-11-18 09:54:52.963 11853 ERROR nova.virt.driver     __import__(mod_str)
2015-11-18 09:54:52.963 11853 ERROR nova.virt.driver ImportError: No module named vmwareapi
2015-11-18 09:54:52.963 11853 ERROR nova.virt.driver

Fix

Just type pip install oslo.vmware (in this case within the python virtual environment: /openstack/venvs/nova-master/). If pip has problems to install it, you could use easy_install . Otherwise use the force.

Raw
03_fw_etimedout.md

Now the driver is loaded, but it cannot create a session on the VCenter host:

2015-11-18 11:10:49.193 19357 CRITICAL nova [-] ConnectionError: ('Connection aborted.', error(110, 'ETIMEDOUT'))
2015-11-18 11:10:49.193 19357 ERROR nova Traceback (most recent call last):
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/bin/nova-compute", line 11, in <module>
2015-11-18 11:10:49.193 19357 ERROR nova     sys.exit(main())
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/cmd/compute.py", line 72, in main
2015-11-18 11:10:49.193 19357 ERROR nova     db_allowed=CONF.conductor.use_local)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/service.py", line 271, in create
2015-11-18 11:10:49.193 19357 ERROR nova     db_allowed=db_allowed)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/service.py", line 148, in __init__
2015-11-18 11:10:49.193 19357 ERROR nova     self.manager = manager_class(host=self.host, *args, **kwargs)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/compute/manager.py", line 718, in __init__
2015-11-18 11:10:49.193 19357 ERROR nova     self.driver = driver.load_compute_driver(self.virtapi, compute_driver)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/virt/driver.py", line 1454, in load_compute_driver
2015-11-18 11:10:49.193 19357 ERROR nova     virtapi)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_utils/importutils.py", line 50, in import_object_ns
2015-11-18 11:10:49.193 19357 ERROR nova     return import_class(import_value)(*args, **kwargs)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/virt/vmwareapi/driver.py", line 166, in __init__
2015-11-18 11:10:49.193 19357 ERROR nova     self._session = VMwareAPISession(scheme=scheme)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/virt/vmwareapi/driver.py", line 616, in __init__
2015-11-18 11:10:49.193 19357 ERROR nova     insecure=insecure)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 183, in __init__
2015-11-18 11:10:49.193 19357 ERROR nova     self._create_session()
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 122, in func
2015-11-18 11:10:49.193 19357 ERROR nova     return evt.wait()
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2015-11-18 11:10:49.193 19357 ERROR nova     return hubs.get_hub().switch()
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2015-11-18 11:10:49.193 19357 ERROR nova     return self.greenlet.switch()
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/common/loopingcall.py", line 123, in _inner
2015-11-18 11:10:49.193 19357 ERROR nova     idle = self.f(*self.args, **self.kw)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 95, in _func
2015-11-18 11:10:49.193 19357 ERROR nova     result = f(*args, **kwargs)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 254, in inner
2015-11-18 11:10:49.193 19357 ERROR nova     return f(*args, **kwargs)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 230, in _create_session
2015-11-18 11:10:49.193 19357 ERROR nova     session_manager = self.vim.service_content.sessionManager
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 199, in vim
2015-11-18 11:10:49.193 19357 ERROR nova     pool_maxsize=self._pool_size)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/vim.py", line 44, in __init__
2015-11-18 11:10:49.193 19357 ERROR nova     pool_maxsize)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/service.py", line 201, in __init__
2015-11-18 11:10:49.193 19357 ERROR nova     cache=_CACHE)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/suds/client.py", line 115, in __init__
2015-11-18 11:10:49.193 19357 ERROR nova     self.wsdl = reader.open(url)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/suds/reader.py", line 150, in open
2015-11-18 11:10:49.193 19357 ERROR nova     d = self.fn(url, self.options)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/suds/wsdl.py", line 136, in __init__
2015-11-18 11:10:49.193 19357 ERROR nova     d = reader.open(url)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/suds/reader.py", line 74, in open
2015-11-18 11:10:49.193 19357 ERROR nova     d = self.download(url)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/suds/reader.py", line 92, in download
2015-11-18 11:10:49.193 19357 ERROR nova     fp = self.options.transport.open(Request(url))
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/service.py", line 148, in open
2015-11-18 11:10:49.193 19357 ERROR nova     resp = self.session.get(request.url, verify=self.verify)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/requests/sessions.py", line 477, in get
2015-11-18 11:10:49.193 19357 ERROR nova     return self.request('GET', url, **kwargs)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
2015-11-18 11:10:49.193 19357 ERROR nova     resp = self.send(prep, **send_kwargs)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
2015-11-18 11:10:49.193 19357 ERROR nova     r = adapter.send(request, **kwargs)
2015-11-18 11:10:49.193 19357 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
2015-11-18 11:10:49.193 19357 ERROR nova     raise ConnectionError(err, request=request)
2015-11-18 11:10:49.193 19357 ERROR nova ConnectionError: ('Connection aborted.', error(110, 'ETIMEDOUT'))
2015-11-18 11:10:49.193 19357 ERROR nova

Fix

Check FW ports (80 and 443) connectivity from the nova-compute server to the vcenter host.

Raw
04_vcenter_cert.md

SSL certificate verification failed:

2015-11-19 16:17:01.875 10265 INFO nova.virt.driver [-] Loading compute driver 'vmwareapi.VMwareVCDriver'
2015-11-19 16:17:02.022 10265 CRITICAL nova [-] SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2015-11-19 16:17:02.022 10265 ERROR nova Traceback (most recent call last):
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/bin/nova-compute", line 11, in <module>
2015-11-19 16:17:02.022 10265 ERROR nova     sys.exit(main())
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/cmd/compute.py", line 72, in main
2015-11-19 16:17:02.022 10265 ERROR nova     db_allowed=CONF.conductor.use_local)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/service.py", line 271, in create
2015-11-19 16:17:02.022 10265 ERROR nova     db_allowed=db_allowed)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/service.py", line 148, in __init__
2015-11-19 16:17:02.022 10265 ERROR nova     self.manager = manager_class(host=self.host, *args, **kwargs)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/compute/manager.py", line 718, in __init__
2015-11-19 16:17:02.022 10265 ERROR nova     self.driver = driver.load_compute_driver(self.virtapi, compute_driver)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/virt/driver.py", line 1454, in load_compute_driver
2015-11-19 16:17:02.022 10265 ERROR nova     virtapi)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_utils/importutils.py", line 50, in import_object_ns
2015-11-19 16:17:02.022 10265 ERROR nova     return import_class(import_value)(*args, **kwargs)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/virt/vmwareapi/driver.py", line 166, in __init__
2015-11-19 16:17:02.022 10265 ERROR nova     self._session = VMwareAPISession(scheme=scheme)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/virt/vmwareapi/driver.py", line 616, in __init__
2015-11-19 16:17:02.022 10265 ERROR nova     insecure=insecure)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 183, in __init__
2015-11-19 16:17:02.022 10265 ERROR nova     self._create_session()
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 122, in func
2015-11-19 16:17:02.022 10265 ERROR nova     return evt.wait()
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2015-11-19 16:17:02.022 10265 ERROR nova     return hubs.get_hub().switch()
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2015-11-19 16:17:02.022 10265 ERROR nova     return self.greenlet.switch()
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/common/loopingcall.py", line 123, in _inner
2015-11-19 16:17:02.022 10265 ERROR nova     idle = self.f(*self.args, **self.kw)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 95, in _func
2015-11-19 16:17:02.022 10265 ERROR nova     result = f(*args, **kwargs)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 254, in inner
2015-11-19 16:17:02.022 10265 ERROR nova     return f(*args, **kwargs)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 230, in _create_session
2015-11-19 16:17:02.022 10265 ERROR nova     session_manager = self.vim.service_content.sessionManager
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 199, in vim
2015-11-19 16:17:02.022 10265 ERROR nova     pool_maxsize=self._pool_size)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/vim.py", line 44, in __init__
2015-11-19 16:17:02.022 10265 ERROR nova     pool_maxsize)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/service.py", line 201, in __init__
2015-11-19 16:17:02.022 10265 ERROR nova     cache=_CACHE)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/suds/client.py", line 115, in __init__
2015-11-19 16:17:02.022 10265 ERROR nova     self.wsdl = reader.open(url)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/suds/reader.py", line 150, in open
2015-11-19 16:17:02.022 10265 ERROR nova     d = self.fn(url, self.options)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/suds/wsdl.py", line 136, in __init__
2015-11-19 16:17:02.022 10265 ERROR nova     d = reader.open(url)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/suds/reader.py", line 74, in open
2015-11-19 16:17:02.022 10265 ERROR nova     d = self.download(url)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/suds/reader.py", line 92, in download
2015-11-19 16:17:02.022 10265 ERROR nova     fp = self.options.transport.open(Request(url))
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/service.py", line 148, in open
2015-11-19 16:17:02.022 10265 ERROR nova     resp = self.session.get(request.url, verify=self.verify)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/requests/sessions.py", line 477, in get
2015-11-19 16:17:02.022 10265 ERROR nova     return self.request('GET', url, **kwargs)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
2015-11-19 16:17:02.022 10265 ERROR nova     resp = self.send(prep, **send_kwargs)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
2015-11-19 16:17:02.022 10265 ERROR nova     r = adapter.send(request, **kwargs)
2015-11-19 16:17:02.022 10265 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/requests/adapters.py", line 431, in send
2015-11-19 16:17:02.022 10265 ERROR nova     raise SSLError(e, request=request)
2015-11-19 16:17:02.022 10265 ERROR nova SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2015-11-19 16:17:02.022 10265 ERROR nova

Fix

Quick: put insecure = True in nova.conf. Long, specify the CA certificate.

Raw
05_vcenter_permissions.md

The driver is properly loaded, it is able to connect with the VCenter and it gets the version and set up a session:

2015-11-18 13:24:32.243 20680 INFO nova.virt.driver [-] Loading compute driver 'vmwareapi.VMwareVCDriver'
2015-11-18 13:24:37.449 20680 INFO oslo_vmware.api [-] Successfully established new session; session ID is c548d.
2015-11-18 13:24:37.450 20680 INFO nova.virt.vmwareapi.driver [-] VMware vCenter version: 5.5.0
2015-11-18 13:24:37.550 20680 CRITICAL nova [-] NotFound: The specified cluster 'XXXXX' was not found in vCenter
2015-11-18 13:24:37.550 20680 ERROR nova Traceback (most recent call last):
2015-11-18 13:24:37.550 20680 ERROR nova   File "/openstack/venvs/nova-master/bin/nova-compute", line 11, in <module>
2015-11-18 13:24:37.550 20680 ERROR nova     sys.exit(main())
2015-11-18 13:24:37.550 20680 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/cmd/compute.py", line 72, in main
2015-11-18 13:24:37.550 20680 ERROR nova     db_allowed=CONF.conductor.use_local)
2015-11-18 13:24:37.550 20680 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/service.py", line 271, in create
2015-11-18 13:24:37.550 20680 ERROR nova     db_allowed=db_allowed)
2015-11-18 13:24:37.550 20680 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/service.py", line 148, in __init__
2015-11-18 13:24:37.550 20680 ERROR nova     self.manager = manager_class(host=self.host, *args, **kwargs)
2015-11-18 13:24:37.550 20680 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/compute/manager.py", line 718, in __init__
2015-11-18 13:24:37.550 20680 ERROR nova     self.driver = driver.load_compute_driver(self.virtapi, compute_driver)
2015-11-18 13:24:37.550 20680 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/virt/driver.py", line 1454, in load_compute_driver
2015-11-18 13:24:37.550 20680 ERROR nova     virtapi)
2015-11-18 13:24:37.550 20680 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_utils/importutils.py", line 50, in import_object_ns
2015-11-18 13:24:37.550 20680 ERROR nova     return import_class(import_value)(*args, **kwargs)
2015-11-18 13:24:37.550 20680 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/virt/vmwareapi/driver.py", line 181, in __init__
2015-11-18 13:24:37.550 20680 ERROR nova     % self._cluster_name)
2015-11-18 13:24:37.550 20680 ERROR nova NotFound: The specified cluster 'XXXXXX' was not found in vCenter
2015-11-18 13:24:37.550 20680 ERROR nova

Fix

Specify the correct cluster name (not datacenter!) in the nova.conf file. Log in manually in the VCenter with the user provided and look if it is able to see all the resources (based on the permissions given and needed).

Raw
06_image_cache.md

After starting the nova-compute, a FileNotFound exception appears. It tries to see if X.X.X.X_base is on the datastore

2015-11-18 16:09:46.742 24747 DEBUG oslo_vmware.api [req-3e1f4a83-c8c3-4089-b03a-024be9d6c7a8 - - - - -] Waiting for the task: (returnval){
   value = "session[afd4e23d-38f0-2357-2621-fd2ff5df2837]52f0dc2a-4a1a-48bc-0560-ca37fec2ded7"
   _type = "Task"
 } to complete. wait_for_task /openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py:385
2015-11-18 16:09:46.742 24747 DEBUG oslo_vmware.api [-] Invoking VIM API to read info of task: (returnval){
   value = "session[afd4e23d-38f0-2357-2621-fd2ff5df2837]52f0dc2a-4a1a-48bc-0560-ca37fec2ded7"
   _type = "Task"
 }. _poll_task /openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py:397
2015-11-18 16:09:46.743 24747 DEBUG oslo_vmware.api [-] Waiting for function _invoke_api to return. func /openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py:121
2015-11-18 16:09:47.242 24747 DEBUG oslo_vmware.api [-] Invoking VIM API to read info of task: (returnval){
   value = "session[afd4e23d-38f0-2357-2621-fd2ff5df2837]52f0dc2a-4a1a-48bc-0560-ca37fec2ded7"
   _type = "Task"
 }. _poll_task /openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py:397
2015-11-18 16:09:47.243 24747 DEBUG oslo_vmware.api [-] Waiting for function _invoke_api to return. func /openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py:121
2015-11-18 16:09:47.258 24747 ERROR oslo_vmware.common.loopingcall [-] in fixed duration looping call
2015-11-18 16:09:47.258 24747 ERROR oslo_vmware.common.loopingcall Traceback (most recent call last):
2015-11-18 16:09:47.258 24747 ERROR oslo_vmware.common.loopingcall   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/common/loopingcall.py", line 76, in _inner
2015-11-18 16:09:47.258 24747 ERROR oslo_vmware.common.loopingcall     self.f(*self.args, **self.kw)
2015-11-18 16:09:47.258 24747 ERROR oslo_vmware.common.loopingcall   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 428, in _poll_task
2015-11-18 16:09:47.258 24747 ERROR oslo_vmware.common.loopingcall     raise task_ex
2015-11-18 16:09:47.258 24747 ERROR oslo_vmware.common.loopingcall FileNotFoundException: File [datastore] X.X.X.X_base was not found
2015-11-18 16:09:47.258 24747 ERROR oslo_vmware.common.loopingcall

Fix

It is not needed a fix, is "normal" behaviour, after analyzing the code, that folder is created the first time that one VM is deployed on the hypervisor. That folder is just for caching purposes and speed up the creation of new VMs. If you do not want to see the error while no servers have been deployed on the hypervisor, just create manually that folder on the datastore (or disable the caching). Nova launches some threads which are in charge of maintain the caches and get the resources available in the cluster from time to time.

Raw
07_vcenter_openstack_extension.md

Not enough permissions to define the OpenStack extension in VCenter:

2015-11-18 13:26:54.064 20988 INFO nova.virt.driver [-] Loading compute driver 'vmwareapi.VMwareVCDriver'
2015-11-18 13:27:00.196 20988 INFO oslo_vmware.api [-] Successfully established new session; session ID is 7d426.
2015-11-18 13:27:00.197 20988 INFO nova.virt.vmwareapi.driver [-] VMware vCenter version: 5.5.0
2015-11-18 13:27:02.008 20988 WARNING suds.client [-] Web service reported a SOAP processing fault using an unexpected HTTP status code 200. Reporting as an internal server error.
2015-11-18 13:27:02.010 20988 ERROR suds.client [-] <?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:ns0="urn:vim25" xmlns:ns1="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
   <ns1:Body>
      <ns0:RegisterExtension>
         <ns0:_this type="ExtensionManager">ExtensionManager</ns0:_this>
         <ns0:extension>
            <ns0:description>
               <ns0:label>OpenStack</ns0:label>
               <ns0:summary>OpenStack services</ns0:summary>
            </ns0:description>
            <ns0:key>org.openstack.compute</ns0:key>
            <ns0:version>1.0</ns0:version>
            <ns0:lastHeartbeatTime>2015-11-18T13:27:01.002345</ns0:lastHeartbeatTime>
            <ns0:managedEntityInfo>
               <ns0:type>instance</ns0:type>
            </ns0:managedEntityInfo>
         </ns0:extension>
      </ns0:RegisterExtension>
   </ns1:Body>
</SOAP-ENV:Envelope>
2015-11-18 13:27:02.012 20988 CRITICAL nova [-] WebFault: Server raised fault: 'Permission to perform this operation was denied.'
2015-11-18 13:27:02.012 20988 ERROR nova Traceback (most recent call last):
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/bin/nova-compute", line 11, in <module>
2015-11-18 13:27:02.012 20988 ERROR nova     sys.exit(main())
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/cmd/compute.py", line 72, in main
2015-11-18 13:27:02.012 20988 ERROR nova     db_allowed=CONF.conductor.use_local)
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/service.py", line 271, in create
2015-11-18 13:27:02.012 20988 ERROR nova     db_allowed=db_allowed)
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/service.py", line 148, in __init__
2015-11-18 13:27:02.012 20988 ERROR nova     self.manager = manager_class(host=self.host, *args, **kwargs)
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/compute/manager.py", line 718, in __init__
2015-11-18 13:27:02.012 20988 ERROR nova     self.driver = driver.load_compute_driver(self.virtapi, compute_driver)
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/virt/driver.py", line 1454, in load_compute_driver
2015-11-18 13:27:02.012 20988 ERROR nova     virtapi)
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_utils/importutils.py", line 50, in import_object_ns
2015-11-18 13:27:02.012 20988 ERROR nova     return import_class(import_value)(*args, **kwargs)
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/virt/vmwareapi/driver.py", line 197, in __init__
2015-11-18 13:27:02.012 20988 ERROR nova     self._register_openstack_extension()
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/virt/vmwareapi/driver.py", line 255, in _register_openstack_extension
2015-11-18 13:27:02.012 20988 ERROR nova     constants.EXTENSION_TYPE_INSTANCE)
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/virt/vmwareapi/driver.py", line 627, in _call_method
2015-11-18 13:27:02.012 20988 ERROR nova     return self.invoke_api(module, method, self.vim, *args, **kwargs)
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 347, in invoke_api
2015-11-18 13:27:02.012 20988 ERROR nova     return _invoke_api(module, method, *args, **kwargs)
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 122, in func
2015-11-18 13:27:02.012 20988 ERROR nova     return evt.wait()
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2015-11-18 13:27:02.012 20988 ERROR nova     return hubs.get_hub().switch()
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2015-11-18 13:27:02.012 20988 ERROR nova     return self.greenlet.switch()
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/common/loopingcall.py", line 123, in _inner
2015-11-18 13:27:02.012 20988 ERROR nova     idle = self.f(*self.args, **self.kw)
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 95, in _func
2015-11-18 13:27:02.012 20988 ERROR nova     result = f(*args, **kwargs)
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 290, in _invoke_api
2015-11-18 13:27:02.012 20988 ERROR nova     return api_method(*args, **kwargs)
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/vim_util.py", line 452, in register_extension
2015-11-18 13:27:02.012 20988 ERROR nova     vim.client.service.RegisterExtension(extension_manager, os_ext)
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/suds/client.py", line 521, in __call__
2015-11-18 13:27:02.012 20988 ERROR nova     return client.invoke(args, kwargs)
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/suds/client.py", line 581, in invoke
2015-11-18 13:27:02.012 20988 ERROR nova     result = self.send(soapenv)
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/suds/client.py", line 621, in send
2015-11-18 13:27:02.012 20988 ERROR nova     original_soapenv=original_soapenv)
2015-11-18 13:27:02.012 20988 ERROR nova   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/suds/client.py", line 670, in process_reply
2015-11-18 13:27:02.012 20988 ERROR nova     raise WebFault(fault, replyroot)
2015-11-18 13:27:02.012 20988 ERROR nova WebFault: Server raised fault: 'Permission to perform this operation was denied.'
2015-11-18 13:27:02.012 20988 ERROR nova

Fix

Check the user permisions. Have a look at the SOAP request and exception trace, it seems it has not enought permissions to register a new extension in VCenter. Also, really important, nova-compute needs to get the definition of the vmware user with the format user@domain (nova.conf file) otherwise it cannot get permissions for all the objects in VMware but it is able to log in in VCenter (weird!).

Raw
08_portbindingfailed_vmware.md

PortBindingFailed when a VM is launched in VMware

2015-11-18 16:16:05.826 24781 DEBUG keystoneclient.session [-] RESP: [204] date: Wed, 18 Nov 2015 16:16:05 GMT content-length: 0 x-openstack-request-id: req-24e6c8e0-40a6-4f4c-8dc5-6cf76
bfedbf2 _http_log_response /openstack/venvs/nova-master/lib/python2.7/site-packages/keystoneclient/session.py:215
2015-11-18 16:16:05.827 24781 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
2015-11-18 16:16:05.827 24781 ERROR nova.compute.manager Traceback (most recent call last):
2015-11-18 16:16:05.827 24781 ERROR nova.compute.manager   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/compute/manager.py", line 1564, in _allocate_network_async
2015-11-18 16:16:05.827 24781 ERROR nova.compute.manager     dhcp_options=dhcp_options)
2015-11-18 16:16:05.827 24781 ERROR nova.compute.manager   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 727, in allocate_for_instance
2015-11-18 16:16:05.827 24781 ERROR nova.compute.manager     self._delete_ports(neutron, instance, created_port_ids)
2015-11-18 16:16:05.827 24781 ERROR nova.compute.manager   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-11-18 16:16:05.827 24781 ERROR nova.compute.manager     six.reraise(self.type_, self.value, self.tb)
2015-11-18 16:16:05.827 24781 ERROR nova.compute.manager   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 719, in allocate_for_instance
2015-11-18 16:16:05.827 24781 ERROR nova.compute.manager     security_group_ids, available_macs, dhcp_opts)
2015-11-18 16:16:05.827 24781 ERROR nova.compute.manager   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 342, in _create_port
2015-11-18 16:16:05.827 24781 ERROR nova.compute.manager     raise exception.PortBindingFailed(port_id=port_id)
2015-11-18 16:16:05.827 24781 ERROR nova.compute.manager PortBindingFailed: Binding failed for port 3e30270f-65a5-4bcc-8fdd-9cfdea77e7ab, please check neutron logs for more information.
2015-11-18 16:16:05.827 24781 ERROR nova.compute.manager
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [req-1a669093-aa55-492c-9954-e0ba8bc1b2f8 41add127c7504313bb6271b3968d26b6 07505495bd234dddb7220a8f5f3e7422 - - -] [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437] Instance failed to spawn
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437] Traceback (most recent call last):
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/compute/manager.py", line 2155, in _build_resources
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     yield resources
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/compute/manager.py", line 2009, in _build_and_run_instance
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     block_device_info=block_device_info)
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/virt/vmwareapi/driver.py", line 406, in spawn
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     admin_password, network_info, block_device_info)
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/virt/vmwareapi/vmops.py", line 689, in spawn
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     metadata)
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/virt/vmwareapi/vmops.py", line 284, in build_virtual_machine
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     network_info)
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/virt/vmwareapi/vif.py", line 181, in get_vif_info
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     for vif in network_info:
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/network/model.py", line 507, in __iter__
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     return self._sync_wrapper(fn, *args, **kwargs)
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/network/model.py", line 498, in _sync_wrapper
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     self.wait()
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/network/model.py", line 530, in wait
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     self[:] = self._gt.wait()
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     return self._exit_event.wait()
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     return hubs.get_hub().switch()
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     return self.greenlet.switch()
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     result = function(*args, **kwargs)
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/compute/manager.py", line 1581, in _allocate_network_async
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     six.reraise(*exc_info)
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/compute/manager.py", line 1564, in _allocate_network_async
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     dhcp_options=dhcp_options)
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 727, in allocate_for_instance
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     self._delete_ports(neutron, instance, created_port_ids)
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     six.reraise(self.type_, self.value, self.tb)
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 719, in allocate_for_instance
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     security_group_ids, available_macs, dhcp_opts)
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 342, in _create_port
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]     raise exception.PortBindingFailed(port_id=port_id)
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437] PortBindingFailed: Binding failed for port 3e30270f-65a5-4bcc-8fdd-9cfdea77e7ab, please check neutron logs for more information.
2015-11-18 16:16:05.830 24781 ERROR nova.compute.manager [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437]
2015-11-18 16:16:05.835 24781 INFO nova.compute.manager [req-1a669093-aa55-492c-9954-e0ba8bc1b2f8 41add127c7504313bb6271b3968d26b6 07505495bd234dddb7220a8f5f3e7422 - - -] [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437] Terminating instance
2015-11-18 16:16:05.838 24781 DEBUG nova.virt.vmwareapi.vmops [req-1a669093-aa55-492c-9954-e0ba8bc1b2f8 41add127c7504313bb6271b3968d26b6 07505495bd234dddb7220a8f5f3e7422 - - -] [instance: 1d42a398-4831-4a17-8234-8c35f8e3e437] Destroying instance destroy /openstack/venvs/nova-master/lib/python2.7/site-packages/nova/virt/vmwareapi/vmops.py:1071

Fix

Nova compute is contacting Neutron to allocate the resources (dhcp, ports ...) and Neutron it does not how to do it. Check the integration_bridge parameter in the nova-compute configuration (nova.conf) and create a portgroup in VMware (in the proper (dvs)switch) with the same name (integration_bridge). The port binding mechanism has to be Ephemeral - no binding and the VLANID has to be the same as you defined within OpenStack. At this point, this setting will depend if you are using OVS or linuxbridge ML2 plugin mechanism in Neutron. If you are using OVS, br-int normally is the default name you have to give to the portgroup in vmware. In the case of linuxbridge implementation, you have to find out the UUID of the network, take the first 11 characters of the UUID and prefix them with brq to build something like 'brq383fa876-8d', that will be the name of the portgroup in VMware. (You can also get the proper name going to another compute node, type brctl show and get the attached one to the correct vlan interface). You have to create one portgroup (with the proper name) per external network available in VMware.

Raw
09_nopermissionexception_vmware.md

NoPermissionException after spawing a vm and them fails!

2015-11-20 17:43:25.103 17869 DEBUG oslo_vmware.api [-] Waiting for function _invoke_api to return. func /openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py:121
2015-11-20 17:43:25.117 17869 DEBUG oslo_vmware.api [-] Fault list: ['NoPermission'] _invoke_api /openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py:326
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api [-] Error occurred while reading info of task: (returnval){
   value = "task-4114159"
   _type = "Task"
 }.
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api Traceback (most recent call last):
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 403, in _poll_task
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api     'info')
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 347, in invoke_api
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api     return _invoke_api(module, method, *args, **kwargs)
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 122, in func
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api     return evt.wait()
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api     return hubs.get_hub().switch()
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api     return self.greenlet.switch()
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/common/loopingcall.py", line 123, in _inner
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api     idle = self.f(*self.args, **self.kw)
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 95, in _func
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api     result = f(*args, **kwargs)
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 331, in _invoke_api
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api     details=excep.details)
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api NoPermissionException: Error occurred while calling RetrievePropertiesEx.
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api Faults: ['NoPermission']
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api Details: {'privilegeId': 'System.Read', 'object': 'group-d1'}
2015-11-20 17:43:25.118 17869 ERROR oslo_vmware.api
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall [-] in fixed duration looping call
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall Traceback (most recent call last):
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/common/loopingcall.py", line 76, in _inner
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall     self.f(*self.args, **self.kw)
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 408, in _poll_task
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall     task)
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall     six.reraise(self.type_, self.value, self.tb)
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 403, in _poll_task
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall     'info')
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 347, in invoke_api
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall     return _invoke_api(module, method, *args, **kwargs)
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 122, in func
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall     return evt.wait()
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall     return hubs.get_hub().switch()
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall     return self.greenlet.switch()
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/common/loopingcall.py", line 123, in _inner
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall     idle = self.f(*self.args, **self.kw)
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 95, in _func
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall     result = f(*args, **kwargs)
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall   File "/openstack/venvs/nova-master/lib/python2.7/site-packages/oslo_vmware/api.py", line 331, in _invoke_api
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall     details=excep.details)
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall NoPermissionException: Error occurred while calling RetrievePropertiesEx.
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall Faults: ['NoPermission']
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall Details: {'privilegeId': 'System.Read', 'object': 'group-d1'}
2015-11-20 17:43:25.119 17869 ERROR oslo_vmware.common.loopingcall

Fix

The permission for the vmware user has to be applied on the vcenter root object and propagate to beneath.

@wdhwang
Copy link

wdhwang commented Jan 3, 2017

Hi:
I am testing the integration of Openstack Mitaka with VMWare. I could connect to vCenter, and upload template image to vCenter from glance successfully. When I create new virtual machine, and I got the following messages:
2017-01-03 15:28:09.858 30784 DEBUG oslo_vmware.api [-] Waiting for function oslo_vmware.api._invoke_api to return. func /usr/lib/python2.7/dist-packages/oslo_vmware/api.py:122
2017-01-03 15:28:09.881 30784 DEBUG oslo_vmware.exceptions [-] Fault PlatformConfigFault not matched. get_fault_class /usr/lib/python2.7/dist-packages/oslo_vmware/exceptions.py:295
2017-01-03 15:28:09.882 30784 ERROR oslo_vmware.common.loopingcall [-] in fixed duration looping call
2017-01-03 15:28:09.882 30784 ERROR oslo_vmware.common.loopingcall Traceback (most recent call last):
2017-01-03 15:28:09.882 30784 ERROR oslo_vmware.common.loopingcall File "/usr/lib/python2.7/dist-packages/oslo_vmware/common/loopingcall.py", line 76, in _inner
2017-01-03 15:28:09.882 30784 ERROR oslo_vmware.common.loopingcall self.f(*self.args, **self.kw)
2017-01-03 15:28:09.882 30784 ERROR oslo_vmware.common.loopingcall File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 429, in _poll_task
2017-01-03 15:28:09.882 30784 ERROR oslo_vmware.common.loopingcall raise task_ex
2017-01-03 15:28:09.882 30784 ERROR oslo_vmware.common.loopingcall VimFaultException: ���������������������������������
2017-01-03 15:28:09.882 30784 ERROR oslo_vmware.common.loopingcall Faults: ['PlatformConfigFault']
2017-01-03 15:28:09.882 30784 ERROR oslo_vmware.common.loopingcall
2017-01-03 15:28:09.885 30784 ERROR nova.compute.manager [req-ee7af3a1-abb2-4087-9eeb-9151c4d4b15b 92c2aea342d74017ae1eda26b8bd7c8d 03a6aacf106a4244b9e13801738a2960 - - -] [instance: 68bf6874-16dd-4dce-bced-88dace54a4eb] Instance failed to spawn

Any help would be much appreciated.

@jriguera
Copy link
Author

jriguera commented Feb 9, 2017

Sorry for being late. The best approach to have Nova + vmwareapi.VMwareVCDriver is first setup a nova node as it was a KVM node. Once it is working, install all the python dependencies for the new driver, and change the nova configuration file according to the settings of the new driver. Do not delete neutron-agent on the node.

About the error, it seems to me that the settings to connect to VCenter are not properly setup, or the user has not enough permissions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment