Skip to content

Instantly share code, notes, and snippets.

@andyshinn
Created June 19, 2014 16:40
Show Gist options
  • Select an option

  • Save andyshinn/9dbeeb5409de42de8249 to your computer and use it in GitHub Desktop.

Select an option

Save andyshinn/9dbeeb5409de42de8249 to your computer and use it in GitHub Desktop.
Fleet no scheduling job and never exiting

Trying to schedule the following unit file linked to [email protected], [email protected], and [email protected] on 3 CoreOS (349.0.0) nodes:

[Unit]
Description=firehose server

[Service]
ExecStartPre=/usr/bin/docker pull firehose/server
ExecStartPre=/bin/sh -c "docker inspect firehose-%i >/dev/null && docker rm -f firehose-$i || true"
ExecStart=/usr/bin/docker run --name firehose-%i -p 7474 firehose/server
ExecStop=/usr/bin/docker rm -f firehose-%i
User=core
TimeoutStartSec=5m

[Install]
WantedBy=multi-user.target

[X-Fleet]
X-Conflicts=firehose@*.service

When submitting and starting the units, the third one hangs forever:

$ fleetctl start config/systemd/firehose@*.service
Job [email protected] launched on efe374cc.../172.16.32.82
Job [email protected] launched on 91521585.../172.16.32.81
^C%

Journalctl for fleet shows it never gets scheduled:

$ journalctl -u fleet.service -a --no-pager -n 10
-- Logs begin at Tue 2014-06-10 22:41:59 UTC, end at Thu 2014-06-19 16:34:29 UTC. --
Jun 19 16:28:19 ip-172-16-32-83 fleet[545]: I0619 16:28:19.074344 00545 agent.go:389] EventJobOffered([email protected]): passed all criteria, submitting JobBid
Jun 19 16:28:19 ip-172-16-32-83 fleet[545]: I0619 16:28:19.074353 00545 agent.go:413] Submitting JobBid for Job([email protected])
Jun 19 16:28:19 ip-172-16-32-83 fleet[545]: I0619 16:28:19.173582 00545 event.go:84] EventJobBidSubmitted([email protected]): failed to schedule Job to Machine(d5240e23706c4496bcae266618cc611d)
Jun 19 16:34:29 ip-172-16-32-83 fleet[545]: I0619 16:34:29.439021 00545 event.go:29] CommandLoadJob([email protected]): publishing JobOffer
Jun 19 16:34:29 ip-172-16-32-83 fleet[545]: I0619 16:34:29.524632 00545 engine.go:88] Published JobOffer([email protected])
Jun 19 16:34:29 ip-172-16-32-83 fleet[545]: I0619 16:34:29.573884 00545 event.go:27] EventJobOffered([email protected]): deciding whether to bid or not
Jun 19 16:34:29 ip-172-16-32-83 fleet[545]: I0619 16:34:29.573915 00545 agent.go:481] Job([email protected]) has requirements: map[Conflicts:[firehose@*.service]]
Jun 19 16:34:29 ip-172-16-32-83 fleet[545]: I0619 16:34:29.573941 00545 agent.go:389] EventJobOffered([email protected]): passed all criteria, submitting JobBid
Jun 19 16:34:29 ip-172-16-32-83 fleet[545]: I0619 16:34:29.573949 00545 agent.go:413] Submitting JobBid for Job([email protected])
Jun 19 16:34:29 ip-172-16-32-83 fleet[545]: I0619 16:34:29.676865 00545 event.go:84] EventJobBidSubmitted([email protected]): failed to schedule Job to Machine(d5240e23706c4496bcae266618cc611d)

There are no more units running that would be conflicting:

$ fleetctl list-units
UNIT			STATE		LOAD	ACTIVE	SUB	DESC		MACHINE
[email protected]	launched	loaded	active	running	Datadog agent	efe374cc.../172.16.32.82
[email protected]	launched	loaded	active	running	Datadog agent	91521585.../172.16.32.81
[email protected]	launched	loaded	active	running	Datadog agent	d5240e23.../172.16.32.83
[email protected]	launched	loaded	active	running	Firehose server	efe374cc.../172.16.32.82
[email protected]	launched	loaded	active	running	Firehose server	91521585.../172.16.32.81
[email protected]	inactive	-	-	-	Firehose server	-
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment