brianv0 · June 7, 2018 13:20
diff --git a/airflow-help.txt b/airflow-help.txt
 I'm not sure the celery worker option would scale in my case. I think what I actually need is potentially a local executor proxy which talks to a remote executor proxy that does job submission.

 For our use case (astrophysics, e.g. Monte Carlo simulations), it's not uncommon to have more than 10k jobs running in parallel per site (~8 jobs per host over 1k hosts), and globally approaching 30k concurrent jobs, so it's a big no-no to open up database connections in this scenario unless you have some sort of global semaphore (e.g Zookeeper, etcd) to manage connection counts. Typically, most job communication is handled through a message queue (rabbitMQ, email, HTTP, or some site-local proxy which feeds those).

 One option is to have a system which just launches pilot jobs for an entire host and actually just run the Mesos agent, but the problem there is that most batch systems, especially those on supercomputers, typically have a maximum wallclock time of 24 hours and you'd need to factor that into the mesos executor+scheduler somehow. With a custom operator (e.g. BatchOperator which just extends BashOperator) that also supplies a wallcock time (and probably CPU and memory), so that we don't schedule jobs on offers from a node that will die soon. I think this could be accommodated by setting an Attribute on the mesos executor with the wall clock time.


 Thanks for your input.

 After thinking about it and looking code over a bit more, I'm going to work on a slightly different approach in the near term. I'm not going to worry about getting this working cross-site for now, but I'll focus on getting airflow to work within the constraints I have.

 1. As needed, start up Mesos agent on a full batch node (in my case, 32 cores+128GB memory) with a 24 hour wall clock time. Add the wall clock ending time as an attribute to the mesos agent.
 2. Extend BashOperator as BatchOperator (include wall clock, core count, and memory)
 3. Extend the MesosOperator to examine the mesos offer's wall clock attribute. If the job matches the offer (cpu, memory, wall clock attribute), accept the offer.
 4. (later) add nodes as needed

 So, I'm effectively using the batch system as a mechanism for temporary provisioning of mesos workers/agents.
	I'm not sure the celery worker option would scale in my case. I think what I actually need is potentially a local executor proxy which talks to a remote executor proxy that does job submission.

	For our use case (astrophysics, e.g. Monte Carlo simulations), it's not uncommon to have more than 10k jobs running in parallel per site (~8 jobs per host over 1k hosts), and globally approaching 30k concurrent jobs, so it's a big no-no to open up database connections in this scenario unless you have some sort of global semaphore (e.g Zookeeper, etcd) to manage connection counts. Typically, most job communication is handled through a message queue (rabbitMQ, email, HTTP, or some site-local proxy which feeds those).

	One option is to have a system which just launches pilot jobs for an entire host and actually just run the Mesos agent, but the problem there is that most batch systems, especially those on supercomputers, typically have a maximum wallclock time of 24 hours and you'd need to factor that into the mesos executor+scheduler somehow. With a custom operator (e.g. BatchOperator which just extends BashOperator) that also supplies a wallcock time (and probably CPU and memory), so that we don't schedule jobs on offers from a node that will die soon. I think this could be accommodated by setting an Attribute on the mesos executor with the wall clock time.


	Thanks for your input.

	After thinking about it and looking code over a bit more, I'm going to work on a slightly different approach in the near term. I'm not going to worry about getting this working cross-site for now, but I'll focus on getting airflow to work within the constraints I have.

	1. As needed, start up Mesos agent on a full batch node (in my case, 32 cores+128GB memory) with a 24 hour wall clock time. Add the wall clock ending time as an attribute to the mesos agent.
	2. Extend BashOperator as BatchOperator (include wall clock, core count, and memory)
	3. Extend the MesosOperator to examine the mesos offer's wall clock attribute. If the job matches the offer (cpu, memory, wall clock attribute), accept the offer.
	4. (later) add nodes as needed

	So, I'm effectively using the batch system as a mechanism for temporary provisioning of mesos workers/agents.