Skip to content

Instantly share code, notes, and snippets.

@nelsnelson

nelsnelson/gist:2d8593423b6e9478e029

Last active August 29, 2015 14:05
Show Gist options
  • Save nelsnelson/2d8593423b6e9478e029 to your computer and use it in GitHub Desktop.
Save nelsnelson/2d8593423b6e9478e029 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
2014-08-26 03:40:09.520 WARNING nova.virt.disk.mount.nbd [req-a144b56c-094b-4d89-9ba5-937aa1fee111 ServersTestJSON-698736888 ServersTestJSON-1373125453] No free nbd devices
2014-08-26 03:40:09.520 DEBUG nova.openstack.common.lockutils [req-a144b56c-094b-4d89-9ba5-937aa1fee111 ServersTestJSON-698736888 ServersTestJSON-1373125453] Released semaphore "nbd-allocation-lock" fro
m (pid=15425) lock /opt/stack/nova/nova/openstack/common/lockutils.py:291
2014-08-26 03:40:09.521 DEBUG nova.openstack.common.lockutils [req-a144b56c-094b-4d89-9ba5-937aa1fee111 ServersTestJSON-698736888 ServersTestJSON-1373125453] Semaphore / lock released "_inner_get_dev" f
rom (pid=15425) inner /opt/stack/nova/nova/openstack/common/lockutils.py:328
2014-08-26 03:40:09.521 INFO nova.virt.disk.mount.api [req-a144b56c-094b-4d89-9ba5-937aa1fee111 ServersTestJSON-698736888 ServersTestJSON-1373125453] Device allocation failed. Will retry in 2 seconds.
2014-08-26 03:40:10.049 DEBUG nova.openstack.common.periodic_task [-] Running periodic task ComputeManager._reclaim_queued_deletes from (pid=15425) run_periodic_tasks /opt/stack/nova/nova/openstack/comm
on/periodic_task.py:193
2014-08-26 03:40:10.049 DEBUG nova.compute.manager [-] CONF.reclaim_instance_interval <= 0, skipping... from (pid=15425) _reclaim_queued_deletes /opt/stack/nova/nova/compute/manager.py:5670
2014-08-26 03:40:10.049 DEBUG nova.openstack.common.loopingcall [-] Dynamic looping call <bound method Service.periodic_tasks of <nova.service.Service object at 0x7f263545d710>> sleeping for 5.00 second
s from (pid=15425) _inner /opt/stack/nova/nova/openstack/common/loopingcall.py:132
2014-08-26 03:40:11.521 WARNING nova.virt.disk.mount.api [req-a144b56c-094b-4d89-9ba5-937aa1fee111 ServersTestJSON-698736888 ServersTestJSON-1373125453] Device allocation failed after repeated retries.
2014-08-26 03:40:11.522 DEBUG nova.virt.disk.mount.api [req-a144b56c-094b-4d89-9ba5-937aa1fee111 ServersTestJSON-698736888 ServersTestJSON-1373125453] Fail to mount, tearing back down from (pid=15425) d
o_mount /opt/stack/nova/nova/virt/disk/mount/api.py:221
2014-08-26 03:40:11.522 DEBUG nova.virt.disk.api [req-a144b56c-094b-4d89-9ba5-937aa1fee111 ServersTestJSON-698736888 ServersTestJSON-1373125453] No free nbd devices from (pid=15425) mount /opt/stack/nov
a/nova/virt/disk/api.py:310
2014-08-26 03:40:11.522 ERROR nova.virt.disk.api [req-a144b56c-094b-4d89-9ba5-937aa1fee111 ServersTestJSON-698736888 ServersTestJSON-1373125453] Failed to mount container filesystem '<nova.virt.disk.api
._DiskImage object at 0x7f2628068310>' on '/opt/stack/data/nova/instances/2bcb634e-9339-4297-89eb-cc3e58dffd82/rootfs':
--
No free nbd devices
2014-08-26 03:40:11.530 ERROR nova.compute.manager [req-a144b56c-094b-4d89-9ba5-937aa1fee111 ServersTestJSON-698736888 ServersTestJSON-1373125453] [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] Instan
ce failed to spawn
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] Traceback (most recent call last):
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] File "/opt/stack/nova/nova/compute/manager.py", line 2127, in _build_resources
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] yield resources
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] File "/opt/stack/nova/nova/compute/manager.py", line 2013, in _build_and_run_instance
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] block_device_info=block_device_info)
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2491, in spawn
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] block_device_info)
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3953, in _create_domain_and_network
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] power_on=power_on)
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3859, in _create_domain
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] self._create_domain_setup_lxc(instance)
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3821, in _create_domain_setup_lxc
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] use_cow=CONF.use_cow_images)
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] File "/opt/stack/nova/nova/virt/disk/api.py", line 392, in setup_container
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] raise exception.NovaException(img.errors)
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] NovaException:
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] --
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] No free nbd devices
2014-08-26 03:40:11.530 TRACE nova.compute.manager [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82]
2014-08-26 03:40:11.550 AUDIT nova.compute.manager [req-a144b56c-094b-4d89-9ba5-937aa1fee111 ServersTestJSON-698736888 ServersTestJSON-1373125453] [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] Termin
ating instance
2014-08-26 03:40:11.558 WARNING nova.virt.libvirt.driver [-] [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] During wait destroy, instance disappeared.
2014-08-26 03:40:13.108 AUDIT nova.compute.manager [req-7164633a-c81a-4866-b037-69fcfc8cd2ed ServersTestJSON-698736888 ServersTestJSON-1373125453] [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] Termin
ating instance
2014-08-26 03:40:13.113 WARNING nova.virt.libvirt.driver [-] [instance: 2bcb634e-9339-4297-89eb-cc3e58dffd82] During wait destroy, instance disappeared.
2014-08-26 03:43:17.875 WARNING nova.compute.manager [-] Found 1 in the database and 5 on the hypervisor.
$ virsh -c lxc:/// list --all
Id Name State
----------------------------------------------------
36519 instance-00000015 running
- instance-00000069 shut off
- instance-0000006a shut off
- instance-0000006b shut off
- instance-0000006c shut off
- instance-0000006d shut off
@nelsnelson
Copy link
Author 

$ ps ax | grep nbd
35233 ?        Ssl    0:00 /usr/bin/qemu-nbd -c /dev/nbd2 /opt/stack/data/nova/instances/3ead1cf4-e516-4d0d-89a7-2af8de935696/disk
35237 ?        S<     0:00 [nbd2]
36246 ?        Ssl    0:00 /usr/bin/qemu-nbd -c /dev/nbd0 /opt/stack/data/nova/instances/3b79d690-a760-4ae7-aca2-340c86357cc1/disk
36250 ?        S<     0:00 [nbd0]
36255 ?        S      0:00 [jbd2/nbd0-8]
36815 pts/0    S+     0:00 grep nbd

@nelsnelson
Copy link
Author 

$ ps -eaf | grep qemu-nbd
root     35233     1  0 03:43 ?        00:00:00 /usr/bin/qemu-nbd -c /dev/nbd2 /opt/stack/data/nova/instances/3ead1cf4-e516-4d0d-89a7-2af8de935696/disk
root     37174     1  0 03:56 ?        00:00:00 /usr/bin/qemu-nbd -c /dev/nbd11 /opt/stack/data/nova/instances/7441b842-1cf0-48a4-944a-ead9ae3bcbc5/disk
root     39377     1  0 04:25 ?        00:00:00 /usr/bin/qemu-nbd -c /dev/nbd9 /opt/stack/data/nova/instances/d5eacc15-8a25-4c8b-8064-13688f5e2c91/disk
nels.ne+ 40250  2300  0 04:25 pts/3    00:00:00 grep qemu-nbd

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment