Each task in Apache Aurora is started in an empty sandbox and a common first process for any task will be to fetch the binaries needed for the task. At Twitter binaries are fetched from an HDFS cluster similar to how Mesos executors are fetched from HDFS. If binaries for a task change rarely, this puts unnecessary stress on the HDFS cluster and ties task creation to the availability of the HDFS cluster. (See MESOS-336 for discussion on fetching Mesos executors).
The attached file package_cache.py
serves to mitigate this problem by caching