Skip to content

Instantly share code, notes, and snippets.

@romansergey
Created September 5, 2014 18:58
Show Gist options
  • Save romansergey/1ff4cd0e7373947753e2 to your computer and use it in GitHub Desktop.
Save romansergey/1ff4cd0e7373947753e2 to your computer and use it in GitHub Desktop.
scaling up on Deis (installation based on ref: 4b1e6b81f1024888397b9ddd19e79ba72416cac3)
[2014-09-05 09:34:31,008: INFO/MainProcess] Received task: api.tasks.start_containers[47dfe3b7-0429-493d-be83-b3dbaa11e3d7]
[2014-09-05 09:34:31,017: WARNING/Worker-4] Creating qpb-dev_v6.web.2
[2014-09-05 09:34:32,931: WARNING/Worker-4] Starting qpb-dev_v6.web.2
Triggered job qpb-dev_v6.web.2.service start
Triggered job qpb-dev_v6.web.2-log.service start
Triggered job qpb-dev_v6.web.2-announce.service start
2014-09-05 09:42:38 [107] [DEBUG] GET /api/apps/qpb-dev/config
2014-09-05 09:44:31 [81] [CRITICAL] WORKER TIMEOUT (pid:18961)
2014-09-05 09:44:31 [81] [CRITICAL] WORKER TIMEOUT (pid:18961)
2014-09-05 09:44:31 [12769] [INFO] Booting worker with pid: 12769
2014-09-05 09:48:58 [107] [DEBUG] GET /api/apps/qpb-dev
2014-09-05 09:48:58 [107] [DEBUG] GET /api/apps/qpb-dev/containers
2014-09-05 09:48:58 [113] [DEBUG] GET /api/apps/qpb-dev/domains
2014-09-05 09:50:30 [99] [DEBUG] GET /api/apps/qpb-dev
2014-09-05 09:50:30 [12769] [DEBUG] GET /api/apps/qpb-dev/containers
2014-09-05 09:50:31 [113] [DEBUG] GET /api/apps/qpb-dev/domains
[2014-09-05 09:57:51,647: WARNING/Worker-4] Exception in thread Thread-10:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File "/app/api/models.py", line 50, in _inner
func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django_fsm/__init__.py", line 267, in _change_state
return fsm_meta.field.change_state(instance, func, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django_fsm/__init__.py", line 161, in change_state
result = method(instance, *args, **kwargs)
File "/app/api/models.py", line 370, in start
self._scheduler.start(self._job_id, self._command_announceable())
File "/app/scheduler/coreos.py", line 129, in start
self._wait_for_container(name, env)
File "/app/scheduler/coreos.py", line 169, in _wait_for_container
raise RuntimeError('Container failed to start')
RuntimeError: Container failed to start
[2014-09-05 09:57:51,650: INFO/MainProcess] Task api.tasks.start_containers[47dfe3b7-0429-493d-be83-b3dbaa11e3d7] succeeded in 1400.64078697s: None
@romansergey
Copy link
Author

Brief: unable to scale an app up to web=2 or greater. Tried both node and rails app.
Through docker listing and log tailing I see in both cases that app container successfully starts, but fleetctl shows the unit as:

UNIT                        DSTATE      TMACHINE        STATE       MACHINE         ACTIVE
qpb-dev_v6.web.2.service        launched    8106c0f9.../10.21.1.120 inactive    -           -

web=1 -> web=0 - works
web=0 -> web=1 - works
web=1 -> web=n, where n>1 - doesn't work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment