Skip to content

Instantly share code, notes, and snippets.

@briantd
Created May 16, 2017 19:20
Show Gist options
  • Save briantd/85b1347c2c3d491842748c7bea8d8a55 to your computer and use it in GitHub Desktop.
Save briantd/85b1347c2c3d491842748c7bea8d8a55 to your computer and use it in GitHub Desktop.
Troubleshooting REX-Ray: server log produced by attempting to mount an azureUD volume using rexray service (failed to get virtual machine)
swarm-manager000000:/home/docker# rexray volume mount myvolume2
INFO[0000] updated log level logLevel=debug
DEBU[0000] os.args time=1494962294201 val=[rexray volume mount myvolume2]
DEBU[0000] activating libStorage cmd=mount time=1494962294201
DEBU[0000] parseSafeHost - no change postParse=unix:///var/run/libstorage/930586076.sock preParse=unix:///var/run/libstorage/930586076.sock time=1494962294206
DEBU[0000] read spec file host=unix:///var/run/libstorage/930586076.sock path=/var/run/rexray/rexray.spec time=1494962294206
DEBU[0000] is local unix server active sock=/var/run/libstorage/930586076.sock time=1494962294207
DEBU[0000] set host in context host=unix:///var/run/libstorage/930586076.sock time=1494962294207
DEBU[0000] set host in config host=unix:///var/run/libstorage/930586076.sock time=1494962294207
DEBU[0000] not starting embedded server; already running host=unix:///var/run/libstorage/930586076.sock time=1494962294208
DEBU[0000] creating libStorage client cmd=mount host=unix:///var/run/libstorage/930586076.sock time=1494962294208
DEBU[0000] created scoped scope å new=libstorage.client parentScopes=rexray,
INFO[0000] configured logging host=unix:///var/run/libstorage/930586076.sock libstorage.logging.httpRequests=true libstorage.logging.httpResponses=true libstorage.logging.level=debug time=1494962294282
DEBU[0000] got configured host address host=unix:///var/run/libstorage/930586076.sock service=azureud storageDriver=libstorage time=1494962294295
DEBU[0000] getHost proto == "unix" getHost=libstorage-server host=unix:///var/run/libstorage/930586076.sock service=azureud storageDriver=libstorage time=1494962294297
INFO[0000] created libStorage client clientType=integration disableKeepAlive=false enableInstanceIDHeaders=true enableLocalDevicesHeaders=true host=unix:///var/run/libstorage/930586076.sock lAddr=libstorage-server logRequests=true logResponses=true lsxMutexPath=/var/run/libstorage/lsx.lock lsxPath=/var/lib/libstorage/lsx-linux service=azureud storageDriver=libstorage time=1494962294310
INFO[0000] registered custom context key externalID=Libstorage-Tx internalID=2 keyBitmask=2
INFO[0000] registered custom context key externalID=Libstorage-Instanceid internalID=3 keyBitmask=2
INFO[0000] registered custom context key externalID=Libstorage-Localdevices internalID=4 keyBitmask=2
INFO[0000] registered custom context key externalID=Authorization internalID=5 keyBitmask=2
INFO[0000]
INFO[0000] -------------------------- HTTP REQUEST (CLIENT) -------------------------
INFO[0000] GET /services HTTP/1.1
INFO[0000] Host: libstorage-server
INFO[0000] Libstorage-Tx: txID=f289aa3a-141c-4cf0-6aee-c565d273b8ee, txCR=1494962294
INFO[0000]
DEBU[0000] successful connection host=unix:///var/run/libstorage/930586076.sock service=azureud storageDriver=libstorage time=1494962294311
INFO[0000]
INFO[0000] -------------------------- HTTP RESPONSE (CLIENT) -------------------------
INFO[0000] HTTP/1.1 200 OK
INFO[0000] Content-Length: 221
INFO[0000] Content-Type: application/json
INFO[0000] Date: Tue, 16 May 2017 19:18:14 GMT
INFO[0000] Libstorage-Servername: daffodil-raccoon-gn
INFO[0000]
INFO[0000] {
INFO[0000] "azureud": {
INFO[0000] "name": "azureud",
INFO[0000] "driver": {
INFO[0000] "name": "azureud",
INFO[0000] "type": "block",
INFO[0000] "nextDevice": {
INFO[0000] "ignore": false,
INFO[0000] "prefix": "sd",
INFO[0000] "pattern": "[c-z]"
INFO[0000] }
INFO[0000] }
INFO[0000] }
INFO[0000] }
INFO[0000] initializing executors cache host=unix:///var/run/libstorage/930586076.sock service=azureud storageDriver=libstorage time=1494962294317
INFO[0000]
INFO[0000] -------------------------- HTTP REQUEST (CLIENT) -------------------------
INFO[0000] GET /executors HTTP/1.1
INFO[0000] Host: libstorage-server
INFO[0000] Libstorage-Tx: txID=9a117cfb-04a9-45f6-574b-e98937def8a3, txCR=1494962294
INFO[0000]
INFO[0000]
INFO[0000] -------------------------- HTTP RESPONSE (CLIENT) -------------------------
INFO[0000] HTTP/1.1 200 OK
INFO[0000] Content-Length: 157
INFO[0000] Content-Type: application/json
INFO[0000] Date: Tue, 16 May 2017 19:18:14 GMT
INFO[0000] Libstorage-Servername: daffodil-raccoon-gn
INFO[0000]
INFO[0000] {
INFO[0000] "lsx-linux": {
INFO[0000] "name": "lsx-linux",
INFO[0000] "md5checksum": "55741970708ecf145f6af884a86407aa",
INFO[0000] "size": 12139296,
INFO[0000] "lastModified": 1493847475
INFO[0000] }
INFO[0000] }
DEBU[0000] updating executor host=unix:///var/run/libstorage/930586076.sock service=azureud storageDriver=libstorage time=1494962294319
DEBU[0000] waiting on executor lock host=unix:///var/run/libstorage/930586076.sock service=azureud storageDriver=libstorage time=1494962294320
DEBU[0000] executor exists, getting local checksum host=unix:///var/run/libstorage/930586076.sock service=azureud storageDriver=libstorage time=1494962294322
DEBU[0000] getting executor checksum host=unix:///var/run/libstorage/930586076.sock service=azureud storageDriver=libstorage time=1494962294322
DEBU[0000] got local executor checksum host=unix:///var/run/libstorage/930586076.sock localChecksum=55741970708ecf145f6af884a86407aa service=azureud storageDriver=libstorage time=1494962294419
DEBU[0000] signalling executor lock host=unix:///var/run/libstorage/930586076.sock service=azureud storageDriver=libstorage time=1494962294419
INFO[0000] initializing supported cache host=unix:///var/run/libstorage/930586076.sock server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962294420
DEBU[0000] waiting on executor lock host=unix:///var/run/libstorage/930586076.sock server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962294420 txCR=1494962294 txID=b2a1ae38-c455-4fe4-4ccb-cff148c706aa
DEBU[0000] invoking executor cli args=[azureud supported] cmd=/var/lib/libstorage/lsx-linux host=unix:///var/run/libstorage/930586076.sock server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962294420 txCR=1494962294 txID=b2a1ae38-c455-4fe4-4ccb-cff148c706aa
DEBU[0000] signalling executor lock host=unix:///var/run/libstorage/930586076.sock server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962294572 txCR=1494962294 txID=b2a1ae38-c455-4fe4-4ccb-cff148c706aa
DEBU[0000] cached supported flag host=unix:///var/run/libstorage/930586076.sock server=daffodil-raccoon-gn service=azureud storageDriver=libstorage supported=15 time=1494962294573 txCR=1494962294 txID=b2a1ae38-c455-4fe4-4ccb-cff148c706aa
INFO[0000] initializing instance ID cache host=unix:///var/run/libstorage/930586076.sock server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962294573
DEBU[0000] waiting on executor lock host=unix:///var/run/libstorage/930586076.sock server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962294573 txCR=1494962294 txID=23dd2e39-535f-4eb2-736f-a167c2af1ec6
DEBU[0000] invoking executor cli args=[azureud instanceID] cmd=/var/lib/libstorage/lsx-linux host=unix:///var/run/libstorage/930586076.sock server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962294573 txCR=1494962294 txID=23dd2e39-535f-4eb2-736f-a167c2af1ec6
DEBU[0000] signalling executor lock host=unix:///var/run/libstorage/930586076.sock server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962294717 txCR=1494962294 txID=23dd2e39-535f-4eb2-736f-a167c2af1ec6
DEBU[0000] cached instanceID host=unix:///var/run/libstorage/930586076.sock instanceID=azureud=swarm-manager000000 server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962294717 txCR=1494962294 txID=23dd2e39-535f-4eb2-736f-a167c2af1ec6
DEBU[0000] xli instanceID success host=unix:///var/run/libstorage/930586076.sock instanceID=azureud=swarm-manager000000 server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962294717 txCR=1494962294 txID=23dd2e39-535f-4eb2-736f-a167c2af1ec6
INFO[0000] successefully dialed libStorage server host=unix:///var/run/libstorage/930586076.sock server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962294717
INFO[0000] storage driver initialized host=unix:///var/run/libstorage/930586076.sock service=azureud storageDriver=libstorage time=1494962294718
INFO[0000] os driver initialized host=unix:///var/run/libstorage/930586076.sock osDriver=linux service=azureud storageDriver=libstorage time=1494962294720
INFO[0000] linux integration driver successfully initialized host=unix:///var/run/libstorage/930586076.sock integrationDriver=linux libstorage.integration.volume.operations.create.default.IOPS=0 libstorage.integration.volume.operations.create.default.availabilityZone= libstorage.integration.volume.operations.create.default.fsType=ext4 libstorage.integration.volume.operations.create.default.size=16 libstorage.integration.volume.operations.create.default.type= libstorage.integration.volume.operations.create.implicit=true libstorage.integration.volume.operations.mount.path=/var/lib/libstorage/volumes libstorage.integration.volume.operations.mount.rootPath=/data osDriver=linux service=azureud storageDriver=libstorage time=1494962294750
INFO[0000] path cache initializion disabled host=unix:///var/run/libstorage/930586076.sock integrationDriver=linux osDriver=linux service=azureud storageDriver=libstorage time=1494962294761
INFO[0000] libStorage integration driver successfully initialized host=unix:///var/run/libstorage/930586076.sock integrationDriver=linux libstorage.integration.volume.operations.create.disable=false libstorage.integration.volume.operations.mount.preempt=true libstorage.integration.volume.operations.path.cache.async=true libstorage.integration.volume.operations.path.cache.enabled=false libstorage.integration.volume.operations.remove.disable=true libstorage.integration.volume.operations.unmount.ignoreusedcount=false osDriver=linux service=azureud storageDriver=libstorage time=1494962294780
INFO[0000] integration driver initialized host=unix:///var/run/libstorage/930586076.sock integrationDriver=linux osDriver=linux service=azureud storageDriver=libstorage time=1494962294780
INFO[0000] created libStorage client host=unix:///var/run/libstorage/930586076.sock integrationDriver=linux osDriver=linux service=azureud storageDriver=libstorage time=1494962294781
DEBU[0000] waiting on executor lock host=unix:///var/run/libstorage/930586076.sock instanceID=azureud=swarm-manager000000 integrationDriver=linux osDriver=linux server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962294796 txCR=1494962294 txID=3a123253-131c-4b73-7ec8-58afef602149
DEBU[0000] invoking executor cli args=[azureud localDevices quick] cmd=/var/lib/libstorage/lsx-linux host=unix:///var/run/libstorage/930586076.sock instanceID=azureud=swarm-manager000000 integrationDriver=linux osDriver=linux server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962294797 txCR=1494962294 txID=3a123253-131c-4b73-7ec8-58afef602149
DEBU[0001] signalling executor lock host=unix:///var/run/libstorage/930586076.sock instanceID=azureud=swarm-manager000000 integrationDriver=linux osDriver=linux server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962295069 txCR=1494962294 txID=3a123253-131c-4b73-7ec8-58afef602149
DEBU[0001] xli localdevices success host=unix:///var/run/libstorage/930586076.sock instanceID=azureud=swarm-manager000000 integrationDriver=linux osDriver=linux server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962295069 txCR=1494962294 txID=3a123253-131c-4b73-7ec8-58afef602149
INFO[0001]
INFO[0001] -------------------------- HTTP REQUEST (CLIENT) -------------------------
INFO[0001] GET /volumes/azureud?attachments=27 HTTP/1.1
INFO[0001] Host: libstorage-server
INFO[0001] Libstorage-Instanceid: azureud=swarm-manager000000
INFO[0001] Libstorage-Localdevices: azureud=
INFO[0001] Libstorage-Tx: txID=3a123253-131c-4b73-7ec8-58afef602149, txCR=1494962294
INFO[0001]
INFO[0001]
INFO[0001] -------------------------- HTTP RESPONSE (CLIENT) -------------------------
INFO[0001] HTTP/1.1 200 OK
INFO[0001] Content-Length: 273
INFO[0001] Content-Type: application/json
INFO[0001] Date: Tue, 16 May 2017 19:18:15 GMT
INFO[0001] Libstorage-Servername: daffodil-raccoon-gn
INFO[0001]
INFO[0001] {
INFO[0001] "myvolume1.vhd": {
INFO[0001] "attachmentState": 3,
INFO[0001] "name": "myvolume1",
INFO[0001] "size": 16,
INFO[0001] "id": "myvolume1.vhd",
INFO[0001] "type": ""
INFO[0001] },
INFO[0001] "myvolume2.vhd": {
INFO[0001] "attachmentState": 3,
INFO[0001] "name": "myvolume2",
INFO[0001] "size": 128,
INFO[0001] "id": "myvolume2.vhd",
INFO[0001] "type": ""
INFO[0001] }
INFO[0001] }
DEBU[0001] mounting volume host=unix:///var/run/libstorage/930586076.sock opts=&{OverwriteFS:false NewFSType: Preempt:true Opts:<nil>} time=1494962295179 volumeID=myvolume2.vhd volumeName=
INFO[0001] mounting volume host=unix:///var/run/libstorage/930586076.sock integrationDriver=linux opts=&{OverwriteFS:false NewFSType: Preempt:true Opts:<nil>} osDriver=linux service=azureud storageDriver=libstorage time=1494962295179 volumeID=myvolume2.vhd volumeName=
DEBU[0001] waiting on executor lock host=unix:///var/run/libstorage/930586076.sock instanceID=azureud=swarm-manager000000 integrationDriver=linux osDriver=linux server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962295180 txCR=1494962295 txID=99c9035b-1370-4e17-4957-0bbb8cbf10a3
DEBU[0001] invoking executor cli args=[azureud localDevices quick] cmd=/var/lib/libstorage/lsx-linux host=unix:///var/run/libstorage/930586076.sock instanceID=azureud=swarm-manager000000 integrationDriver=linux osDriver=linux server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962295180 txCR=1494962295 txID=99c9035b-1370-4e17-4957-0bbb8cbf10a3
DEBU[0001] signalling executor lock host=unix:///var/run/libstorage/930586076.sock instanceID=azureud=swarm-manager000000 integrationDriver=linux osDriver=linux server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962295356 txCR=1494962295 txID=99c9035b-1370-4e17-4957-0bbb8cbf10a3
DEBU[0001] xli localdevices success host=unix:///var/run/libstorage/930586076.sock instanceID=azureud=swarm-manager000000 integrationDriver=linux osDriver=linux server=daffodil-raccoon-gn service=azureud storageDriver=libstorage time=1494962295356 txCR=1494962295 txID=99c9035b-1370-4e17-4957-0bbb8cbf10a3
INFO[0001]
INFO[0001] -------------------------- HTTP REQUEST (CLIENT) -------------------------
INFO[0001] GET /volumes/azureud/myvolume2.vhd?attachments=7 HTTP/1.1
INFO[0001] Host: libstorage-server
INFO[0001] Libstorage-Instanceid: azureud=swarm-manager000000
INFO[0001] Libstorage-Localdevices: azureud=
INFO[0001] Libstorage-Tx: txID=99c9035b-1370-4e17-4957-0bbb8cbf10a3, txCR=1494962295
INFO[0001]
INFO[0001]
INFO[0001] -------------------------- HTTP RESPONSE (CLIENT) -------------------------
INFO[0001] HTTP/1.1 500 Internal Server Error
INFO[0001] Content-Length: 593
INFO[0001] Content-Type: application/json
INFO[0001] Date: Tue, 16 May 2017 19:18:15 GMT
INFO[0001] Libstorage-Servername: daffodil-raccoon-gn
INFO[0001]
INFO[0001] {
INFO[0001] "message": "failed to convert volume",
INFO[0001] "status": 500,
INFO[0001] "error": {
INFO[0001] "inner": {
INFO[0001] "inner": {
INFO[0001] "inner": "compute.VirtualMachinesClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code=\"ResourceNotFound\" Message=\"The Resource 'Microsoft.Compute/virtualMachines/swarm-manager000000' under resource group 'g...g' was not found.\"",
INFO[0001] "msg": "failed to get virtual machine",
INFO[0001] "vmName": "swarm-manager000000"
INFO[0001] },
INFO[0001] "msg": "Unable to lookup devices on VM"
INFO[0001] }
INFO[0001] }
INFO[0001] }
DEBU[0001] built output template host=unix:///var/run/libstorage/930586076.sock template={{define "printObject"}}{{printf "%v" .}}{{end}}{{define "printVolumeID"}}{{.ID}}{{end}}{{define "printVolumeWithPathFields"}}{{.ID}} {{.Name}} {{.Volume.AttachmentState | printAttState}} {{.Size}} {{.Path}}{{end}}{{define "printSnapshotFields"}}{{.ID}} {{.Name}} {{.Status}} {{.VolumeID}}{{end}}{{define "printEncVolumeWithPathFields"}}{{.ID}} {{.Name}} {{.Volume.AttachmentState | printAttState}} {{.Size}} {{.Encrypted}} {{.Path}}{{end}}{{define "printJSON"}}{{. | json}}{{end}}{{define "printEncVolumeFields"}}{{.ID}} {{.Name}} {{.AttachmentState | printAttState}} {{.Size}} {{.Encrypted}}{{end}}{{define "printServiceFields"}}{{.Name}} {{.Driver.Name}}{{end}}{{define "printAuthTokenFields"}}{{.Subject}} {{.IssuedAt}} {{.NotBefore}} {{.Expires}}{{end}}{{define "printMountFields"}}{{.ID}} {{.Source}} {{.MountPoint}}{{end}}{{define "printScriptFields"}}{{.Path}} {{.Installed}} {{.Modified}}{{end}}{{define "printStringSlice"}}{{.}}{{end}}{{define "printPrettyJSON"}}{{. | jsonp}}{{end}}{{define "printVolumeFields"}}{{.ID}} {{.Name}} {{.AttachmentState | printAttState}} {{.Size}}{{end}}{{define "printInstanceFields"}}{{.InstanceID.ID}} {{.Name}} {{.ProviderName}} {{.Region}}{{end}}ID Name Status Size Path
{{range sort .D "Name" }}{{template "printVolumeWithPathFields" .}}
{{end}} time=1494962295559
ID Name Status Size Path
FATA[0001] error mounting volume error.inner.inner.inner.vmName=swarm-manager000000 error.status=500 volume=myvolume2.vhd
swarm-manager000000:~$ cat /var/log/rexray/rexray.log
⌐▄Q▓▄Ç▓▄,▄_
Σ▄▓▓▓▓▓▓▓▓▓▓▄π
╒▓▓▌▓▓▓▓▓▓▓▓▓▓▀▓▄▄.
,_▄▀▓▓ ▓▓ ▓▓▓▓▓▓▓▓▓▓▓█
│▄▓▓ _▓▓▓▓▓▓▓▓▓┌▓▓▓▓▓█
_J┤▓▓▓▓▓▓▓▓▓▓▓▓▓├█▓█▓▀Γ
,▄▓▓▓▓▓▓^██▓▓▓▓▓▓▓▓▓▓▓▓▄▀▄▄▓▓Ω▄
F▌▓▌█ⁿⁿⁿ ⁿ└▀ⁿ██▓▀▀▀▀▀▀▀▀▀▀▌▓▓▓▌
'ⁿ_ ,▄▄▄▄▄▄▄▄▄█_▄▄▄▄▄▄▄▄▄ⁿ▀~██
Γ ├▓▓▓▓▓█▀ⁿ█▌▓Ω]█▓▓▓▓▓▓ ├▓
│ ├▓▓▓▓▓▌≡,__▄▓▓▓█▓▓▓▓▓ ╞█~ Y,┐
╞ ├▓▓▓▓▓▄▄__^^▓▓▓▌▓▓▓▓▓ ▓ /▓▓▓
├▓▓▓▓▓▓▓▄▄═▄▓▓▓▓▓▓▓▓▓ π ⌐▄▓▓█║n
_ ├▓▓▓▓▓▓▓▓▓~▓▓▓▓▓▓▓▓▓▓ ▄4▄▓▓▓██
µ ├▓▓▓▓█▀█▓▓_▓▓███▓▓▓▓▓ ▓▓▓▓▓Ω4
µ ├▓▀▀L └ⁿ ▀ ▀ ▓▓█w ▓▓▓▀ìⁿ
⌐ ├_ τ▀▓ Σ⌐└
~ ├▓▓ ▄ _ ╒ ┌▄▓▓ Γ
├▓▓▓▌█═┴▓▄╒▀▄_▄▌═¢▓▓▓ ╚
⌠ ├▓▓▓▓▓ⁿ▄▓▓▓▓▓▓▓┐▄▓▓▓▓ └
Ω_.└██▓▀ⁿÇⁿ▀▀▀▀▀▀█≡▀▀▀▀▀ µ
ⁿ .▄▄▓▓▓▓▄▄┌ ╖__▓_▄▄▄▄▄*Oⁿ
û▌├▓█▓▓▓██ⁿ ¡▓▓▓▓▓▓▓▓█▓╪
╙Ω▀█ ▓██ⁿ └█▀██▀▓█├█Å
ⁿⁿ ⁿ ⁿ^
:::::::.. .,:::::: .,:: .::::::::.. :::. .-:. ::-.
;;;;'';;;; ;;;;'''' ';;;, .,;; ;;;;'';;;; ;;';; ';;. ;;;;'
[[[,/[[[' [[cccc '[[,,[[' [[[,/[[[' ,[[ '[[, '[[,[[['
$$$$$$c $$"""" Y$$$Pcccc $$$$$$c c$$$cc$$$c c$$"
888b "88bo,888oo,__ oP"''"Yo, 888b "88bo,888 888,,8P"'
MMMM "W" """"YUMMM,m" "Mm, MMMM "W" YMM ""'mM"
REX-Ray
-------
Binary: /usr/bin/rexray
Flavor: client+agent+controller
SemVer: 0.9.0
OsArch: Linux-x86_64
Branch: v0.9.0
Commit: 2a7458dd90a79c673463e14094377baf9fc8695e
Formed: Wed, 03 May 2017 21:38:11 UTC
libStorage
----------
SemVer: 0.6.0
OsArch: Linux-x86_64
Branch: v0.9.0
Commit: fa055d6da595602715bdfd5541b4aa6d4dcbcbd9
Formed: Wed, 03 May 2017 21:36:11 UTC
time="2017-05-16T18:54:30Z" level=debug msg="created scoped scope" new=libstorage.server parentScopes="rexray,"
time="2017-05-16T18:54:31Z" level=debug msg="created scoped scope" new=libstorage.server.services.azureud parentScopes="libstorage.server,rexray,"
time="2017-05-16T18:54:32Z" level=debug msg="created scoped scope" new=rexray.modules.default-docker parentScopes="rexray,"
time="2017-05-16T18:54:32Z" level=debug msg="created scoped scope" new=libstorage.client parentScopes="rexray.modules.default-docker,rexray,"
time="2017-05-16T18:54:32Z" level=info
time="2017-05-16T18:54:32Z" level=info msg=" -------------------------- HTTP REQUEST (CLIENT) -------------------------"
time="2017-05-16T18:54:32Z" level=info msg=" GET /services HTTP/1.1"
time="2017-05-16T18:54:32Z" level=info msg=" Host: libstorage-server"
time="2017-05-16T18:54:32Z" level=info msg=" Libstorage-Tx: txID=e3d3dc92-b73e-4986-405f-5375760118ca, txCR=1494960872"
time="2017-05-16T18:54:32Z" level=info msg=" "
time="2017-05-16T18:54:32Z" level=info
time="2017-05-16T18:54:32Z" level=info msg=" -------------------------- HTTP RESPONSE (CLIENT) -------------------------"
time="2017-05-16T18:54:32Z" level=info msg=" HTTP/1.1 200 OK"
time="2017-05-16T18:54:32Z" level=info msg=" Content-Length: 221"
time="2017-05-16T18:54:32Z" level=info msg=" Content-Type: application/json"
time="2017-05-16T18:54:32Z" level=info msg=" Date: Tue, 16 May 2017 18:54:32 GMT"
time="2017-05-16T18:54:32Z" level=info msg=" Libstorage-Servername: daffodil-raccoon-gn"
time="2017-05-16T18:54:32Z" level=info msg=" "
time="2017-05-16T18:54:32Z" level=info msg=" {"
time="2017-05-16T18:54:32Z" level=info msg=" \"azureud\": {"
time="2017-05-16T18:54:32Z" level=info msg=" \"name\": \"azureud\","
time="2017-05-16T18:54:32Z" level=info msg=" \"driver\": {"
time="2017-05-16T18:54:32Z" level=info msg=" \"name\": \"azureud\","
time="2017-05-16T18:54:32Z" level=info msg=" \"type\": \"block\","
time="2017-05-16T18:54:32Z" level=info msg=" \"nextDevice\": {"
time="2017-05-16T18:54:32Z" level=info msg=" \"ignore\": false,"
time="2017-05-16T18:54:32Z" level=info msg=" \"prefix\": \"sd\","
time="2017-05-16T18:54:32Z" level=info msg=" \"pattern\": \"[c-z]\""
time="2017-05-16T18:54:32Z" level=info msg=" }"
time="2017-05-16T18:54:32Z" level=info msg=" }"
time="2017-05-16T18:54:32Z" level=info msg=" }"
time="2017-05-16T18:54:32Z" level=info msg=" }"
time="2017-05-16T18:54:32Z" level=info
time="2017-05-16T18:54:32Z" level=info msg=" -------------------------- HTTP REQUEST (CLIENT) -------------------------"
time="2017-05-16T18:54:32Z" level=info msg=" GET /executors HTTP/1.1"
time="2017-05-16T18:54:32Z" level=info msg=" Host: libstorage-server"
time="2017-05-16T18:54:32Z" level=info msg=" Libstorage-Tx: txID=febe023c-9432-4d39-5801-bde0cc15ca6a, txCR=1494960872"
time="2017-05-16T18:54:32Z" level=info msg=" "
time="2017-05-16T18:54:32Z" level=info
time="2017-05-16T18:54:32Z" level=info msg=" -------------------------- HTTP RESPONSE (CLIENT) -------------------------"
time="2017-05-16T18:54:32Z" level=info msg=" HTTP/1.1 200 OK"
time="2017-05-16T18:54:32Z" level=info msg=" Content-Length: 157"
time="2017-05-16T18:54:32Z" level=info msg=" Content-Type: application/json"
time="2017-05-16T18:54:32Z" level=info msg=" Date: Tue, 16 May 2017 18:54:32 GMT"
time="2017-05-16T18:54:32Z" level=info msg=" Libstorage-Servername: daffodil-raccoon-gn"
time="2017-05-16T18:54:32Z" level=info msg=" "
time="2017-05-16T18:54:32Z" level=info msg=" {"
time="2017-05-16T18:54:32Z" level=info msg=" \"lsx-linux\": {"
time="2017-05-16T18:54:32Z" level=info msg=" \"name\": \"lsx-linux\","
time="2017-05-16T18:54:32Z" level=info msg=" \"md5checksum\": \"55741970708ecf145f6af884a86407aa\","
time="2017-05-16T18:54:32Z" level=info msg=" \"size\": 12139296,"
time="2017-05-16T18:54:32Z" level=info msg=" \"lastModified\": 1493847475"
time="2017-05-16T18:54:32Z" level=info msg=" }"
time="2017-05-16T18:54:32Z" level=info msg=" }"
time="2017-05-16T18:54:33Z" level=info
time="2017-05-16T18:54:33Z" level=info msg=" -------------------------- HTTP REQUEST (CLIENT) -------------------------"
time="2017-05-16T18:54:33Z" level=info msg=" GET /volumes/azureud?attachments=15 HTTP/1.1"
time="2017-05-16T18:54:33Z" level=info msg=" Host: libstorage-server"
time="2017-05-16T18:54:33Z" level=info msg=" Libstorage-Instanceid: azureud=swarm-manager000000"
time="2017-05-16T18:54:33Z" level=info msg=" Libstorage-Localdevices: azureud="
time="2017-05-16T18:54:33Z" level=info msg=" Libstorage-Tx: txID=9cd70c6a-3a87-4432-4d40-5787c9952c8f, txCR=1494960872"
time="2017-05-16T18:54:33Z" level=info msg=" "
time="2017-05-16T18:54:33Z" level=info
time="2017-05-16T18:54:33Z" level=info msg=" -------------------------- HTTP RESPONSE (CLIENT) -------------------------"
time="2017-05-16T18:54:33Z" level=info msg=" HTTP/1.1 500 Internal Server Error"
time="2017-05-16T18:54:33Z" level=info msg=" Content-Length: 601"
time="2017-05-16T18:54:33Z" level=info msg=" Content-Type: application/json"
time="2017-05-16T18:54:33Z" level=info msg=" Date: Tue, 16 May 2017 18:54:33 GMT"
time="2017-05-16T18:54:33Z" level=info msg=" Libstorage-Servername: daffodil-raccoon-gn"
time="2017-05-16T18:54:33Z" level=info msg=" "
time="2017-05-16T18:54:33Z" level=info msg=" {"
time="2017-05-16T18:54:33Z" level=info msg=" \"message\": \"error converting to types.Volume\","
time="2017-05-16T18:54:33Z" level=info msg=" \"status\": 500,"
time="2017-05-16T18:54:33Z" level=info msg=" \"error\": {"
time="2017-05-16T18:54:33Z" level=info msg=" \"inner\": {"
time="2017-05-16T18:54:33Z" level=info msg=" \"inner\": {"
time="2017-05-16T18:54:33Z" level=info msg=" \"inner\": \"compute.VirtualMachinesClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code=\\\"ResourceNotFound\\\" Message=\\\"The Resource 'Microsoft.Compute/virtualMachines/swarm-manager000000' under resource group 'g...g' was not found.\\\"\","
time="2017-05-16T18:54:33Z" level=info msg=" \"msg\": \"failed to get virtual machine\","
time="2017-05-16T18:54:33Z" level=info msg=" \"vmName\": \"swarm-manager000000\""
time="2017-05-16T18:54:33Z" level=info msg=" },"
time="2017-05-16T18:54:33Z" level=info msg=" \"msg\": \"Unable to lookup devices on VM\""
time="2017-05-16T18:54:33Z" level=info msg=" }"
time="2017-05-16T18:54:33Z" level=info msg=" }"
time="2017-05-16T18:54:33Z" level=info msg=" }"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment