gistfile1.txt

[11:05:24]  <saikat>	what is exit code 128 in the context of a docker container?  I just had a very strange situation where my docker daemon was tking up tons of memory.  I tried killing one container, and that caused all my containers to die with exit code 128
[11:05:27]  <exarkun>	You expect people to know your private definition of "really"? :)
[11:05:28]  <cpuguy83>	exarkun: DB process crashed, db is corrupted, db process won't start.
[11:05:53] 	intellix ([email protected]) joined the channel.
[11:06:00] 	liranp ([email protected]) joined the channel.
[11:06:54] 	plucas ([email protected]) left IRC. (Ping timeout: 245 seconds)
[11:06:55]  <saikat>	and now removing the dead containers is complaining about "Error response from daemon: Cannot destroy container 484eb73e02f9: Driver devicemapper failed to remove root filesystem 484eb73e02f92f31f323a568b4e4f699ebf119e238abb5faa14b2b04a6b9ba50: Device is Busy"
[11:07:05] 	ostera ([email protected]) left IRC. (Client Quit)
[11:07:14] 	odinnap ([email protected]) joined the channel.
[11:07:24] 	joelhe ([email protected]) left IRC. (Quit: joelhe)
[11:07:59] 	cresten ([email protected]) joined the channel.
[11:08:18]  <cpuguy83>	saikat: can you paste output of `docker info` and `docker version`?
[11:08:32]  <Sinjek>	saikat: switch off devicemapper
[11:08:33] 	joelhe ([email protected]) joined the channel.
[11:08:37]  <cpuguy83>	saikat: How many containers were you running? Was it docker daemon consuming memory or a container?
[11:08:38] 	tilgovi (~randall@couchdb/committer/tilgovi) joined the channel.
[11:08:48]  <saikat>	@cpuguy83 17 containers, and it was the daemon
[11:09:18]  <cpuguy83>	saikat: For how long? Are any of them particularly noisy (with log output)?
[11:09:21]  <saikat>	at least i'm baising this off output from ps/htop and the process taking all the memory was the one thta was "/usr/bin/docker -d"
[11:09:24]  <saikat>	about a week
[11:09:24] 	e-dard is now known as e-dard_afk
[11:09:28]  <saikat>	they all have log output
[11:09:39]  <saikat>	i log quite a bit
[11:09:42]  <saikat>	could that be the issue?
[11:09:48]  <cpuguy83>	It's a possibility.
[11:09:50]  <saikat>	https://gist.github.com/saikat/ea44316ccecfab73b0b0
[11:10:04]  <saikat>	is exit code 128 indicative of the containers possibly getting killed due to memory running out?
[11:10:10]  <saikat>	i'm not sure why they all died when i stopped just one container
[11:10:10]  <cpuguy83>	There's a pretty bad memory leak in <= docker 1.5 with noisy containers.
[11:10:15]  <saikat>	i'm on 1.6
[11:10:16]  <Sinjek>	saikat: docker inspect will set the flag OOMKilled
[11:10:20]  <Sinjek>	in state
[11:10:24]  <Sinjek>	that will tell you if that's why it died
[11:10:50]  <saikat>	hm not oomkilled
[11:10:57]  <saikat>	just inspected one of the containers that got killed
[11:11:08]  <saikat>	https://gist.github.com/saikat/8749725a6c0919066ae3
[11:11:10]  <saikat>	is the output
[11:11:13] 	AFreitas_ ([email protected]) left the channel.
[11:11:20] 	aconnor ([email protected]) left IRC. (Ping timeout: 276 seconds)
[11:11:34]  <saikat>	@Sinjek what do you mean turn off devicemapper?  like switch to aufs?
[11:11:42]  <saikat>	is devicemapper not reliable?
[11:11:45]  <Sinjek>	yeah, regarding the 'device is busy' one
[11:12:13]  <Sinjek>	devicemapper is buggy, slow, and just generally icky
[11:12:29]  <cpuguy83>	saikat: devicemapper is less than ideal... and if you in your `info` about there is `Udev Sync Supported: false`, this pretty much means you shouldn't use devicemapper
[11:12:39]  <cpuguy83>	On master right now docker will refuse to start if that is false because it's so bad.
[11:12:59]  <cpuguy83>	At least, it will refuse without a "I know what I'm doing" flag
[11:13:13] 	charliesome ([email protected]) joined the channel.
[11:13:19] 	PaulCapestany ([email protected]) joined the channel.
[11:13:21]  <saikat>	ah
[11:13:28]  <saikat>	@cpuguy83 yeah i pasted my info above, it seems like that is indeed false
[11:13:34]  <saikat>	so is the better option to switch from devicemapper to something else
[11:13:38]  <saikat>	or to try to get that to be true?
[11:13:41]  <cpuguy83>	And udev sync is false because it's a statically compiled binary
[11:13:53]  <saikat>	(i don't know the details of what udev sync being false means or devicemapper vs aufs"
[11:13:56]  <cpuguy83>	You'd need to compile a dyn binary for that to work.
[11:14:05] 	tilgovi (~randall@couchdb/committer/tilgovi) left IRC. (Read error: Connection reset by peer)
[11:14:07]  <cpuguy83>	saikat: only affects devicemapper.
[11:14:26]  <saikat>	yeah i just mean -- if i want to run docker with the least amount of trouble and the most amount of whatever is sanctioned by those in the know
[11:14:28]  <cpuguy83>	If you can get overlayfs, use that, otherwise aufs
[11:14:33]  <saikat>	ok cool
[11:14:42]  <Sinjek>	+1 ^
[11:14:54] 	tilgovi (~randall@couchdb/committer/tilgovi) joined the channel.
[11:14:59]  <saikat>	and any thoughts on how i can debug this memory leak?
[11:15:12]  <saikat>	could that be related to me running devicemapper with udev sync supported being false?
[11:15:39] 	PaulCape_ ([email protected]) left IRC. (Ping timeout: 245 seconds)
[11:16:03]  <cpuguy83>	Probably not.
[11:16:21]  <cpuguy83>	btw, you can remove your `dead` containers with `docker rm -f`
[11:16:27]  <cpuguy83>	But some files may stick around
[11:16:44] 	grzesiek ([email protected]) left IRC. (Quit: Leaving)
[11:16:49]  <saikat>	yeah i've noticed that
[11:16:54]  <saikat>	curently they are all in status exit (128)
[11:16:58] 	rojem ([email protected]) left IRC. (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
[11:17:02]  <saikat>	if i fail to do the rm  on them
[11:17:05]  <saikat>	they become "dead"
[11:17:25]  <saikat>	but ok, thanks
[11:17:27]  <cpuguy83>	yep, exactly
[11:17:37]  <saikat>	is there anything i should do to keep log output from going crazy
[11:17:41]  <saikat>	or does docker already manage the logs
[11:17:53] 	PaulCapestany ([email protected]) left IRC. (Max SendQ exceeded)
[11:17:54]  <cpuguy83>	saikat: Docker consumes stdout/stderr from the container
[11:17:59] 	codeman (b113cd54@gateway/web/freenode/ip.177.19.205.84) joined the channel.
[11:18:05]  <saikat>	yeah i print everything te stdout/stderr
[11:18:05]  <cpuguy83>	saikat: by default these get encoded to a json log file.
[11:18:15]  <cpuguy83>	Which is pretty heavy
[11:18:23]  <saikat>	does this json log file need to get truncated/rotated by me?
[11:18:35]  <cpuguy83>	in 1.6 you can instead use syslog or disable logging
[11:18:54] 	PaulCapestany ([email protected]) joined the channel.
[11:18:58]  <saikat>	ok i'll look into that
[11:19:00]  <cpuguy83>	If you use json log, then yeah right now it would need to be manually rotated (via copy/truncate)
[11:19:03]  <saikat>	i'm using logspout to send stuff to papertrail right now
[11:19:29]  <saikat>	so i don't care about the json logs unless logspout is relying on them, but i think i can send syslog to papertrail
[11:19:32]  <cpuguy83>	saikat: If you are using this in prod I'd not use json-log.. but keep in mind `docker logs` command only works with json-log right now.
[11:19:35]  <saikat>	but the json logs should only be taking up disk space right? not memory?
[11:19:39] 	bdub|lunch is now known as bdub
[11:19:45] 	zzxc (~c@unaffiliated/zzxc) joined the channel.
[11:19:46] 	aconnor ([email protected]) joined the channel.
[11:20:09]  <cpuguy83>	saikat: The logs still have to get buffered and such... this is actually where one of the memory leaks exists, but was dramatically reduced in 1.6
[11:20:31]  <saikat>	ok thanks
[11:20:36] 	e-dard_afk is now known as e-dard
[11:20:47]  <saikat>	i'll try all those suggestions, thank you very much
[11:20:52]  <cpuguy83>	Also, logspout is pretty heavy as well
[11:21:05]  <saikat>	is there a better solution these days for sending logs to some third party host?
[11:21:10]  <cpuguy83>	because docker has to allocate pipes for all the containers
[11:21:21]  <cpuguy83>	saikat: I'd use syslog if you can
[11:21:52]  <saikat>	ok sounds good
[11:22:19] 	ostera ([email protected]) joined the channel.
[11:22:34] 	pmelopereira (2ebd9fb1@gateway/web/freenode/ip.46.189.159.177) left IRC. (Quit: Page closed)
[11:22:49]  <saikat>	yeah the weird thing about the memory leak is -- i've been tracking it over time and the memory usage doesn't gradually go down
[11:22:52]  <saikat>	there seem to be spikes
[11:23:08]  <saikat>	but i haven't been able to pin them to any other particular event on my machine
[11:23:20]  <saikat>	but something must be occurring that is causing it
[11:23:28]  <cpuguy83>	saikat: I think with top/htop it will look like docker is using RAM that is actually being used by containers... I'm not 100% on that though.
[11:23:38]  <Evanlec>	so guys - if my nodejs app is expecting a local redis server to be running, should I still create a separate container for redis and for my app? I know thats the docker way, but it seems crappy for sharing my app with others
[11:23:41]  <saikat>	oh hmm
[11:23:44] 	ssn__ ([email protected]) joined the channel.
[11:23:58] 	fsimonce (~simon@redhat/fsimonce) left IRC. (Ping timeout: 255 seconds)
[11:23:58]  <saikat>	@cpuguy83 but in that case the memory wouldnot also be shown to be being used by my process running inside the container?
[11:23:59] 	HansE ([email protected]) joined the channel.
[11:24:10]  <saikat>	or do you mean memory that the container is using that is not being allocated by my process in the container but rather by the container itself
[11:24:17]  <Evanlec>	requiring me to distribute my docker-compose.yml file - and the complicatin of that
[11:24:20] 	HansE ([email protected]) left IRC. (Max SendQ exceeded)
[11:24:21]  <cpuguy83>	saikat: it indeed would show the process using the memory
[11:24:32]  <saikat>	yeah the processes are not showing high memory usage
[11:24:59] 	HansE ([email protected]) joined the channel.
[11:25:16] 	doniphon (~doniphon@foresight/developer/doniphon) joined the channel.
[11:25:22] 	fsimonce (~simon@redhat/fsimonce) joined the channel.
[11:25:36] 	zapient ([email protected]) left IRC. (Quit: zapient)
[11:25:37] 	tilgovi (~randall@couchdb/committer/tilgovi) left IRC. (Quit: No Ping reply in 180 seconds.)
[11:25:38] 	aconnor ([email protected]) left IRC. (Ping timeout: 276 seconds)
[11:25:42] 	charliesome ([email protected]) left IRC. (Quit: zzz)
[11:25:58] 	jottr (~jottr@unaffiliated/jottr) joined the channel.
[11:26:45]  <cpuguy83>	Actually yeah I'm wrong, memory is split, top/htop do not report docker as using a sum of all containers + itself
[11:26:58] 	tilgovi (~randall@couchdb/committer/tilgovi) joined the channel.
[11:26:58] 	zapient ([email protected]) joined the channel.
[11:27:00]  <chance>	saikat: go supports runtime memory sampling profiles
[11:27:01]  <cpuguy83>	so if it says docker is using 5% then that's docker itself.
[11:27:11]  <chance>	saikat: its docker your referring to right?
[11:27:15]  <saikat>	@chance yes
[11:27:23]  <chance>	you could build it with memory profiling enabled
[11:27:28]  <chance>	and then dump a memory profile
[11:27:33] 	firecat53 ([email protected]) joined the channel.
[11:27:43] 	zzxc (~c@unaffiliated/zzxc) left IRC. (Ping timeout: 250 seconds)
[11:27:47]  <saikat>	ok interesting
[11:27:48]  <chance>	can show you which functions and such are doing the allocations
[11:27:54]  <saikat>	right