Skip to content

Instantly share code, notes, and snippets.

@gjaldon

gjaldon/react_stall2.log

Created March 20, 2020 10:30
Show Gist options
  • Save gjaldon/1ec778a32446c6456a88729029d9aafd to your computer and use it in GitHub Desktop.
Save gjaldon/1ec778a32446c6456a88729029d9aafd to your computer and use it in GitHub Desktop.
Download ZIP
Raw
react_stall2.log
Mar 20 07:17:51 scylla-league-001 scylla-manager-agent[16262]: STARTUP ERROR: no connection to Scylla API, make sure that Scylla server is running and api_address and api_port are set correctly in config file /etc/scylla-manager-agent/scylla-manager-agent.yaml: giving up after 60 attempts: dial tcp 0.0.0.0:10000: connect: connection refused
Mar 20 07:17:51 scylla-league-001 systemd[1]: scylla-manager-agent.service: Main process exited, code=exited, status=1/FAILURE
Mar 20 07:17:51 scylla-league-001 systemd[1]: scylla-manager-agent.service: Unit entered failed state.
Mar 20 07:17:51 scylla-league-001 systemd[1]: scylla-manager-agent.service: Failed with result 'exit-code'.
Mar 20 07:17:52 scylla-league-001 systemd[1]: scylla-manager-agent.service: Service hold-off time over, scheduling restart.
Mar 20 07:17:52 scylla-league-001 systemd[1]: Stopped Scylla Manager Agent.
Mar 20 07:17:52 scylla-league-001 systemd[1]: Started Scylla Manager Agent.
Mar 20 07:17:52 scylla-league-001 scylla-manager-agent[16308]: {"L":"INFO","T":"2020-03-20T07:17:52.128Z","M":"Scylla Manager Agent","version":"2.0.1-0.20200123.7cf18f6b-1~stretch","_trace_id":"WD_ch8m4TwWbIreJI92Jlw"}
Mar 20 07:17:52 scylla-league-001 scylla-manager-agent[16308]: {"L":"INFO","T":"2020-03-20T07:17:52.129Z","N":"wait","M":"Waiting for network connection","sleep":"2s","error":"dial tcp 0.0.0.0:10000: connect: connection refused","_trace_id":"WD_ch8m4TwWbIreJI92Jlw"}
Mar 20 07:17:52 scylla-league-001 scylla[1272]: Reactor stalled for 5288 ms on shard 10.
Mar 20 07:17:52 scylla-league-001 scylla[1272]: Backtrace:
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x000000000419fed2
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x000000000408d1e5
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x000000000408d725
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x000000000408da67
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x00007f0ab489202f
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x0000000000e9c168
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x0000000000eaa32b
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x0000000000eaa5a8
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x0000000000eb70ae
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x000000000403dc58
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x000000000403e4fe
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x0000000004041e0d
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x00000000014f5447
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x000000000151eede
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x000000000152244a
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x0000000001525241
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x0000000001582cc2
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x00000000015a3dcd
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x000000000408a6b1
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x000000000408a8ce
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x000000000416c65d
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x0000000004196ccb
Mar 20 07:17:52 scylla-league-001 scylla[1272]: 0x000000000405898d
Mar 20 07:17:52 scylla-league-001 scylla[1272]: /opt/scylladb/libreloc/libpthread.so.0+0x000000000000858d
Mar 20 07:17:52 scylla-league-001 scylla[1272]: /opt/scylladb/libreloc/libc.so.6+0x00000000000fd6a2
Mar 20 07:17:53 scylla-league-001 scylla: [shard 2] storage_proxy - Failed to apply mutation from 10.128.15.209#8: seastar::semaphore_timed_out (Semaphore timedout)
Mar 20 07:17:53 scylla-league-001 scylla: [shard 5] storage_proxy - Failed to apply mutation from 10.128.15.209#5: seastar::semaphore_timed_out (Semaphore timedout)
Mar 20 07:17:53 scylla-league-001 scylla: [shard 9] storage_proxy - Failed to apply mutation from 10.128.15.209#15: seastar::semaphore_timed_out (Semaphore timedout)
Mar 20 07:17:53 scylla-league-001 scylla: [shard 12] storage_proxy - Failed to apply mutation from 10.128.15.209#0: seastar::semaphore_timed_out (Semaphore timedout)
Mar 20 07:17:53 scylla-league-001 scylla: [shard 13] storage_proxy - Failed to apply mutation from 10.128.15.209#3: seastar::semaphore_timed_out (Semaphore timedout)
Mar 20 07:17:54 scylla-league-001 scylla-manager-agent[16308]: {"L":"INFO","T":"2020-03-20T07:17:54.129Z","N":"wait","M":"Waiting for network connection","sleep":"2s","error":"dial tcp 0.0.0.0:10000: connect: connection refused","_trace_id":"WD_ch8m4TwWbIreJI92Jlw"}
Raw
reactor_stall1.log
Mar 20 07:11:25 scylla-league-001 scylla-manager-agent[16117]: {"L":"INFO","T":"2020-03-20T07:11:25.137Z","N":"wait","M":"Waiting for network connection","sleep":"2s","error":"dial tcp 0.0.0.0:10000: connect: connection refused","_trace_id":"hkHg-OHbRVm2xaBREl2XRg"}
Mar 20 07:11:27 scylla-league-001 scylla-manager-agent[16117]: {"L":"INFO","T":"2020-03-20T07:11:27.137Z","N":"wait","M":"Waiting for network connection","sleep":"2s","error":"dial tcp 0.0.0.0:10000: connect: connection refused","_trace_id":"hkHg-OHbRVm2xaBREl2XRg"}
Mar 20 07:11:29 scylla-league-001 scylla-manager-agent[16117]: {"L":"INFO","T":"2020-03-20T07:11:29.137Z","N":"wait","M":"Waiting for network connection","sleep":"2s","error":"dial tcp 0.0.0.0:10000: connect: connection refused","_trace_id":"hkHg-OHbRVm2xaBREl2XRg"}
Mar 20 07:11:30 scylla-league-001 scylla[1272]: Reactor stalled for 5290 ms on shard 4.
Mar 20 07:11:30 scylla-league-001 scylla[1272]: Backtrace:
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x000000000419fed2
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x000000000408d1e5
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x000000000408d725
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x000000000408da67
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x00007f0ab489202f
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x0000000000ed815b
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x0000000000d857c4
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x0000000000ec2007
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x0000000000eb1e37
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x0000000000eb7089
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x000000000403dc58
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x000000000403e4fe
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x0000000004041e0d
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x00000000014f5447
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x000000000151eede
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x000000000152244a
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x0000000001525241
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x0000000001582cc2
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x00000000015a6189
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x00000000015a8d54
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x0000000001090a01
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x0000000001093dc3
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x0000000001095515
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x00000000015e1162
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x00000000015bd35f
Mar 20 07:11:30 scylla-league-001 scylla[1272]: 0x0000000000839561
Mar 20 07:11:31 scylla-league-001 scylla-manager-agent[16117]: {"L":"INFO","T":"2020-03-20T07:11:31.138Z","N":"wait","M":"Waiting for network connection","sleep":"2s","error":"dial tcp 0.0.0.0:10000: connect: connection refused","_trace_id":"hkHg-OHbRVm2xaBREl2XRg"}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment