Created
July 24, 2017 17:47
-
-
Save sprutner/270583dc97a50d53493a3b6ab5722e3c to your computer and use it in GitHub Desktop.
nomad error log
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Jul 24 17:44:25 ip-10-104-20-31 systemd[1]: nomad.service: Main process exited, code=exited, status=2/INVALIDARGUMENT | |
| Jul 24 17:44:25 ip-10-104-20-31 systemd[1]: nomad.service: Unit entered failed state. | |
| Jul 24 17:44:25 ip-10-104-20-31 systemd[1]: nomad.service: Failed with result 'exit-code'. | |
| Jul 24 17:44:35 ip-10-104-20-31 systemd[1]: nomad.service: Service hold-off time over, scheduling restart. | |
| Jul 24 17:44:35 ip-10-104-20-31 systemd[1]: Stopped nomad server. | |
| Jul 24 17:44:35 ip-10-104-20-31 systemd[1]: Started nomad server. | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: Loaded configuration from /etc/nomad.d/server.hcl | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: ==> Starting Nomad agent... | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: ==> Nomad agent configuration: | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: Atlas: <disabled> | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: Client: false | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: Log Level: INFO | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: Region: global (DC: dc1) | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: Server: true | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: Version: 0.6.0rc1 | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: ==> Nomad agent started! Log data will stream in below: | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] raft: Restored from snapshot 69-21032-1500793356023 | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] raft: Initial configuration (index=12867): [{Suffrage:Voter ID:10.104.21.74:4647 Address:10.104.21.74:4647} {Suffrage:Voter ID:10.104.21.92:4647 Address:10.104.21.92:4647} {Suffrage:Voter ID:10.104.20.31:4647 Address:10.104.20.31:4647} {Suffrage:Voter ID:10.104.20.214:4647 Address:10.104.20.214:4647} {Suffrage:Voter ID:10.104.20.32:4647 Address:10.104.20.32:4647}] | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] serf: EventMemberJoin: ip-10-104-20-31.global 10.104.20.31 | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35.490241 [INFO] nomad: starting 1 scheduling worker(s) for [service batch system _core] | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] raft: Node at 10.104.20.31:4647 [Follower] entering Follower state (Leader: "") | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] serf: Attempting re-join to previously known node: ip-10-104-20-214.global: 10.104.20.214:4648 | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35.496115 [INFO] nomad: adding server ip-10-104-20-31.global (Addr: 10.104.20.31:4647) (DC: dc1) | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] serf: EventMemberJoin: ip-10-104-21-74.global 10.104.21.74 | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] serf: EventMemberJoin: ip-10-104-20-214.global 10.104.20.214 | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] serf: EventMemberJoin: ip-10-104-20-32.global 10.104.20.32 | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [WARN] memberlist: Refuting an alive message | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] serf: EventMemberJoin: ip-10-104-21-92.global 10.104.21.92 | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35.503595 [INFO] nomad: adding server ip-10-104-21-74.global (Addr: 10.104.21.74:4647) (DC: dc1) | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35.503855 [INFO] nomad: adding server ip-10-104-20-214.global (Addr: 10.104.20.214:4647) (DC: dc1) | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35.504090 [INFO] nomad: adding server ip-10-104-20-32.global (Addr: 10.104.20.32:4647) (DC: dc1) | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35.504317 [INFO] nomad: adding server ip-10-104-21-92.global (Addr: 10.104.21.92:4647) (DC: dc1) | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] serf: Attempting re-join to previously known node: ip-10-104-20-32.global: 10.104.20.32:4648 | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] serf: Re-joined to previously known node: ip-10-104-20-32.global: 10.104.20.32:4648 | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35.510453 [INFO] server.nomad: successfully contacted 2 Nomad Servers | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35.597080 [INFO] vault: renewing token in 35h59m59.999995944s | |
| Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [WARN] raft: Failed to get previous log: 24360 log not found (last: 24356) | |
| Jul 24 17:44:37 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:37 [WARN] raft: Rejecting vote request from 10.104.21.92:4647 since we have a leader: 10.104.20.32:4647 | |
| Jul 24 17:44:37 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:37 [WARN] raft: Heartbeat timeout from "10.104.20.32:4647" reached, starting election | |
| Jul 24 17:44:37 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:37 [INFO] raft: Node at 10.104.20.31:4647 [Candidate] entering Candidate state in term 1422 | |
| Jul 24 17:44:37 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:37 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.104.21.74:4647 10.104.21.74:4647}: dial tcp 10.104.21.74:4647: getsockopt: connection refused | |
| Jul 24 17:44:37 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:37 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.104.20.32:4647 10.104.20.32:4647}: dial tcp 10.104.20.32:4647: getsockopt: connection refused | |
| Jul 24 17:44:37 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:37 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.104.20.214:4647 10.104.20.214:4647}: dial tcp 10.104.20.214:4647: getsockopt: connection refused | |
| Jul 24 17:44:38 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:38 [INFO] raft: Node at 10.104.20.31:4647 [Follower] entering Follower state (Leader: "") | |
| Jul 24 17:44:39 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:39 [INFO] serf: EventMemberUpdate: ip-10-104-20-214.global | |
| Jul 24 17:44:40 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:40.239753 [ERR] http: Request /v1/status/peers, error: rpc error: failed to get conn: dial tcp 10.104.21.92:4647: getsockopt: connection refused | |
| Jul 24 17:44:41 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:41 [WARN] raft: Heartbeat timeout from "10.104.21.92:4647" reached, starting election | |
| Jul 24 17:44:41 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:41 [INFO] raft: Node at 10.104.20.31:4647 [Candidate] entering Candidate state in term 1425 | |
| Jul 24 17:44:41 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:41 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.104.21.92:4647 10.104.21.92:4647}: EOF | |
| Jul 24 17:44:41 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:41 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.104.21.74:4647 10.104.21.74:4647}: dial tcp 10.104.21.74:4647: getsockopt: connection refused | |
| Jul 24 17:44:41 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:41 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.104.20.32:4647 10.104.20.32:4647}: dial tcp 10.104.20.32:4647: getsockopt: connection refused | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: Duplicate RequestVote for same term: 1425 | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] serf: EventMemberUpdate: ip-10-104-21-74.global | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [WARN] raft: Election timeout reached, restarting election | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: Node at 10.104.20.31:4647 [Candidate] entering Candidate state in term 1426 | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.104.21.92:4647 10.104.21.92:4647}: dial tcp 10.104.21.92:4647: getsockopt: connection refused | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.104.20.32:4647 10.104.20.32:4647}: dial tcp 10.104.20.32:4647: getsockopt: connection refused | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: Election won. Tally: 3 | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: Node at 10.104.20.31:4647 [Leader] entering Leader state | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: Added peer 10.104.21.74:4647, starting replication | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: Added peer 10.104.21.92:4647, starting replication | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: Added peer 10.104.20.214:4647, starting replication | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: Added peer 10.104.20.32:4647, starting replication | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42.738016 [INFO] nomad: cluster leadership acquired | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [ERR] raft: Failed to AppendEntries to {Voter 10.104.20.32:4647 10.104.20.32:4647}: dial tcp 10.104.20.32:4647: getsockopt: connection refused | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: pipelining replication to peer {Voter 10.104.20.214:4647 10.104.20.214:4647} | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [ERR] raft: Failed to AppendEntries to {Voter 10.104.21.92:4647 10.104.21.92:4647}: dial tcp 10.104.21.92:4647: getsockopt: connection refused | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [WARN] raft: AppendEntries to {Voter 10.104.21.74:4647 10.104.21.74:4647} rejected, sending older logs (next: 24361) | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [ERR] raft: Failed to AppendEntries to {Voter 10.104.20.32:4647 10.104.20.32:4647}: dial tcp 10.104.20.32:4647: getsockopt: connection refused | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [ERR] raft: Failed to AppendEntries to {Voter 10.104.21.92:4647 10.104.21.92:4647}: dial tcp 10.104.21.92:4647: getsockopt: connection refused | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [ERR] raft: Failed to AppendEntries to {Voter 10.104.20.32:4647 10.104.20.32:4647}: dial tcp 10.104.20.32:4647: getsockopt: connection refused | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [ERR] raft: Failed to AppendEntries to {Voter 10.104.21.92:4647 10.104.21.92:4647}: dial tcp 10.104.21.92:4647: getsockopt: connection refused | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: pipelining replication to peer {Voter 10.104.21.74:4647 10.104.21.74:4647} | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: panic: runtime error: slice bounds out of range | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: goroutine 42 [running]: | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/scheduler.(*allocReconciler).computeGroup(0xc4202087e0, 0xc420260ea0, 0x11, 0xc4202dede0, 0xc4202ded80) | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/scheduler/reconcile.go:393 +0x1592 | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/scheduler.(*allocReconciler).Compute(0xc4202087e0, 0xc420179640) | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/scheduler/reconcile.go:182 +0x1f1 | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/scheduler.(*GenericScheduler).computeJobAllocs(0xc42059ccf0, 0xc420422780, 0xc420447580) | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/scheduler/generic_sched.go:391 +0x3cc | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/scheduler.(*GenericScheduler).process(0xc42059ccf0, 0x24, 0x24, 0xc4207a57b0) | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/scheduler/generic_sched.go:230 +0x53e | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/scheduler.(*GenericScheduler).(github.com/hashicorp/nomad/scheduler.process)-fm(0xc4203b6400, 0xc4207a57b0, 0x4c41ae) | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/scheduler/generic_sched.go:131 +0x2a | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/scheduler.retryMax(0x5, 0xc42078f908, 0xc42078f918, 0xc, 0xffffffffffffff01) | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/scheduler/util.go:268 +0x42 | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/scheduler.(*GenericScheduler).Process(0xc42059ccf0, 0xc420662c60, 0xc420433360, 0x1b3ec80) | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/scheduler/generic_sched.go:131 +0x2a0 | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/nomad.(*nomadFSM).reconcileQueuedAllocations(0xc420176a80, 0x5af3, 0x0, 0x0) | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/nomad/fsm.go:919 +0x3e5 | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/nomad.(*nomadFSM).applyReconcileSummaries(0xc420176a80, 0xc4203e85a1, 0x8, 0x8, 0x5af3, 0xc420726ea0, 0xc4204e5ee0) | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/nomad/fsm.go:526 +0x7e | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/nomad.(*nomadFSM).Apply(0xc420176a80, 0xc42073f9e0, 0x1b666c0, 0xed1082903) | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/nomad/fsm.go:153 +0x6b3 | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(*Raft).runFSM.func1(0xc4203e85d0) | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/fsm.go:57 +0x189 | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(*Raft).runFSM(0xc4200bf080) | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/fsm.go:120 +0x2d6 | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(*Raft).(github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.runFSM)-fm() | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/api.go:505 +0x2a | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc4200bf080, 0xc420190c20) | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/state.go:142 +0x53 | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(*raftState).goFunc | |
| Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/state.go:143 +0x66 | |
| Jul 24 17:44:42 ip-10-104-20-31 systemd[1]: nomad.service: Main process exited, code=exited, status=2/INVALIDARGUMENT | |
| Jul 24 17:44:42 ip-10-104-20-31 systemd[1]: nomad.service: Unit entered failed state. | |
| Jul 24 17:44:42 ip-10-104-20-31 systemd[1]: nomad.service: Failed with result 'exit-code'. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment