toil_py36.logs

DEBUG:toil.batchSystems.mesos.batchSystem:Preparing to launch Mesos task 0 with 1.00 cores, 2048.00 MiB memory, and 2048.00 MiB disk using offer 5b6aaf7d-99c1-410a-85f3-dc9602313361-O20 ...
DEBUG:toil.batchSystems.mesos.batchSystem:Launched Mesos task 0.
WARNING:toil.batchSystems.mesos.batchSystem:Executor 'toil-28072' reported lost with status '32512'.
WARNING:toil.batchSystems.mesos.batchSystem:Handling failure of executor 'toil-28072' on agent '5b6aaf7d-99c1-410a-85f3-dc9602313361-S2'.
DEBUG:toil.batchSystems.mesos.batchSystem:Available files: dict_keys(['/slave/log', '/var/lib/mesos/slaves/5b6aaf7d-99c1-410a-85f3-dc9602313361-S2/frameworks/5b6aaf7d-99c1-410a-85f3-dc9602313361-0032/executors/toil-27424/runs/f56c948d-a6b4-4ea5-9639-7518518f6ace', '/var/lib/mesos/slaves/5b6aaf7d-99c1-410a-85f3-dc9602313361-S2/frameworks/5b6aaf7d-99c1-410a-85f3-dc9602313361-0033/executors/toil-28072/runs/807e0102-ecc0-406c-b9b8-f60a845ec65f'])
WARNING:toil.batchSystems.mesos.batchSystem:Attempting to retrieve executor error log: http://172.31.36.66:5051/files/download?path=%2Fvar%2Flib%2Fmesos%2Fslaves%2F5b6aaf7d-99c1-410a-85f3-dc9602313361-S2%2Fframeworks%2F5b6aaf7d-99c1-410a-85f3-dc9602313361-0033%2Fexecutors%2Ftoil-28072%2Fruns%2F807e0102-ecc0-406c-b9b8-f60a845ec65f%2Fstderr
WARNING:toil.batchSystems.mesos.batchSystem:Executor: b'sh: 1: _toil_mesos_executor: not found'
WARNING:toil.batchSystems.mesos.batchSystem:Attempting to retrieve agent log: http://172.31.36.66:5051/files/download?path=%2Fslave%2Flog
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'Log file created at: 2019/03/03 20:06:12'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'Running on machine: ip-172-31-36-66.ec2.internal'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.310715     1 logging.cpp:194] INFO level logging started!'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.438370     1 systemd.cpp:237] systemd version `229` detected'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.439499     1 containerizer.cpp:196] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.444424     1 main.cpp:434] Starting Mesos agent'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.445698     7 slave.cpp:198] Agent started on 1)@172.31.36.66:5051'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.445739     7 slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --attributes="preemptable:True" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="false" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher="posix" --launcher_dir="/usr/libexec/mesos" --log_dir="/var/lib/mesos" --logbufsecs="0" --logging_level="INFO" --master="172.31.41.201:5050" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="false" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/var/lib/mesos"'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.446818     7 slave.cpp:519] Agent resources: cpus(*):2; mem(*):6449; disk(*):45020; ports(*):[31000-32000]'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.446892     7 slave.cpp:527] Agent attributes: [ preemptable=True ]'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.446925     7 slave.cpp:532] Agent hostname: 172.31.36.66'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:06:12.450480    11 state.cpp:57] Recovering state from '/var/lib/mesos/meta'"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.450983     9 status_update_manager.cpp:200] Recovering status update manager'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.451218     9 containerizer.cpp:522] Recovering containerizer'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.451946     8 provisioner.cpp:253] Provisioner recovery complete'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.452318     6 slave.cpp:4782] Finished recovery'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.453174     6 status_update_manager.cpp:174] Pausing sending status updates'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.453200     9 slave.cpp:895] New master detected at [email protected]:5050'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.453476     9 slave.cpp:916] No credentials provided. Attempting to register without authentication'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.453595     9 slave.cpp:927] Detecting new master'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.572358    10 slave.cpp:1095] Registered with master [email protected]:5050; given agent ID 5b6aaf7d-99c1-410a-85f3-dc9602313361-S2'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.572865     8 status_update_manager.cpp:181] Resuming sending status updates'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.573140    10 slave.cpp:1155] Forwarding total oversubscribed resources'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:07:12.463241    11 slave.cpp:4591] Current disk usage 16.78%. Max allowed age: 5.125651996422547days'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:08:05.226449    10 slave.cpp:1495] Got assigned task 0 for framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:08:05.226830    10 slave.cpp:1614] Launching task 0 for framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.228348    10 paths.cpp:528] Trying to chown '/var/lib/mesos/slaves/5b6aaf7d-99c1-410a-85f3-dc9602313361-S2/frameworks/5b6aaf7d-99c1-410a-85f3-dc9602313361-0032/executors/toil-27424/runs/f56c948d-a6b4-4ea5-9639-7518518f6ace' to user 'root'"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.237267    10 slave.cpp:5674] Launching executor toil-27424 of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032 with resources  in work directory '/var/lib/mesos/slaves/5b6aaf7d-99c1-410a-85f3-dc9602313361-S2/frameworks/5b6aaf7d-99c1-410a-85f3-dc9602313361-0032/executors/toil-27424/runs/f56c948d-a6b4-4ea5-9639-7518518f6ace'"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.237512     6 containerizer.cpp:781] Starting container 'f56c948d-a6b4-4ea5-9639-7518518f6ace' for executor 'toil-27424' of framework '5b6aaf7d-99c1-410a-85f3-dc9602313361-0032'"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.237512    10 slave.cpp:1840] Queuing task '0' for executor 'toil-27424' of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:08:05.237677    10 slave.cpp:2218] Asked to shut down framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032 by [email protected]:5050'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:08:05.237776    10 slave.cpp:2243] Shutting down framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.237812    10 slave.cpp:4407] Shutting down executor 'toil-27424' of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"W0303 20:08:05.237843    10 slave.hpp:768] Unable to send event to executor 'toil-27424' of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032: unknown connection type"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.240005    10 launcher.cpp:126] Forked child with pid '19' for container 'f56c948d-a6b4-4ea5-9639-7518518f6ace'"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"W0303 20:08:05.241564    10 slave.cpp:4026] Killing executor 'toil-27424' of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032 because the framework is terminating"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.241880    10 containerizer.cpp:1622] Destroying container 'f56c948d-a6b4-4ea5-9639-7518518f6ace'"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.340878    11 containerizer.cpp:1863] Executor for container 'f56c948d-a6b4-4ea5-9639-7518518f6ace' has exited"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.342031    12 slave.cpp:4089] Executor 'toil-27424' of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032 terminated with signal Killed"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.342329    12 slave.cpp:4193] Cleaning up executor 'toil-27424' of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:08:05.342664    12 slave.cpp:4281] Cleaning up framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.342828    12 gc.cpp:55] Scheduling '/var/lib/mesos/slaves/5b6aaf7d-99c1-410a-85f3-dc9602313361-S2/frameworks/5b6aaf7d-99c1-410a-85f3-dc9602313361-0032/executors/toil-27424/runs/f56c948d-a6b4-4ea5-9639-7518518f6ace' for gc 6.99999603579852days in the future"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.342983    12 gc.cpp:55] Scheduling '/var/lib/mesos/slaves/5b6aaf7d-99c1-410a-85f3-dc9602313361-S2/frameworks/5b6aaf7d-99c1-410a-85f3-dc9602313361-0032/executors/toil-27424' for gc 6.99999603422518days in the future"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.343055    12 gc.cpp:55] Scheduling '/var/lib/mesos/slaves/5b6aaf7d-99c1-410a-85f3-dc9602313361-S2/frameworks/5b6aaf7d-99c1-410a-85f3-dc9602313361-0032' for gc 6.99999603246519days in the future"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:08:05.343191    12 status_update_manager.cpp:282] Closing status update streams for framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:10.239328     8 slave.cpp:4448] Framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032 seems to have exited. Ignoring shutdown timeout for executor 'toil-27424'"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:08:12.464560    13 slave.cpp:4591] Current disk usage 16.78%. Max allowed age: 5.125644906964490days'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:09:05.238062     7 slave.cpp:4499] Framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032 seems to have exited. Ignoring registration timeout for executor 'toil-27424'"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:09:12.465847     9 slave.cpp:4591] Current disk usage 16.78%. Max allowed age: 5.125644906964490days'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:10:12.466612    11 slave.cpp:4591] Current disk usage 16.78%. Max allowed age: 5.125644906964490days'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:11:12.467156     8 slave.cpp:4591] Current disk usage 16.78%. Max allowed age: 5.125643816278634days'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:12:12.467806     6 slave.cpp:4591] Current disk usage 16.78%. Max allowed age: 5.125643816278634days'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:13:12.468935    12 slave.cpp:4591] Current disk usage 16.78%. Max allowed age: 5.125643816278634days'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:13:56.614944    11 slave.cpp:1495] Got assigned task 0 for framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0033'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:13:56.615365    11 slave.cpp:1614] Launching task 0 for framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0033'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:13:56.616663    11 paths.cpp:528] Trying to chown '/var/lib/mesos/slaves/5b6aaf7d-99c1-410a-85f3-dc9602313361-S2/frameworks/5b6aaf7d-99c1-410a-85f3-dc9602313361-0033/executors/toil-28072/runs/807e0102-ecc0-406c-b9b8-f60a845ec65f' to user 'root'"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:13:56.624841    11 slave.cpp:5674] Launching executor toil-28072 of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0033 with resources  in work directory '/var/lib/mesos/slaves/5b6aaf7d-99c1-410a-85f3-dc9602313361-S2/frameworks/5b6aaf7d-99c1-410a-85f3-dc9602313361-0033/executors/toil-28072/runs/807e0102-ecc0-406c-b9b8-f60a845ec65f'"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:13:56.625484    11 slave.cpp:1840] Queuing task '0' for executor 'toil-28072' of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0033"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:13:56.625499     9 containerizer.cpp:781] Starting container '807e0102-ecc0-406c-b9b8-f60a845ec65f' for executor 'toil-28072' of framework '5b6aaf7d-99c1-410a-85f3-dc9602313361-0033'"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:13:56.627305     9 launcher.cpp:126] Forked child with pid '22' for container '807e0102-ecc0-406c-b9b8-f60a845ec65f'"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:13:56.892017    10 containerizer.cpp:1863] Executor for container '807e0102-ecc0-406c-b9b8-f60a845ec65f' has exited"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:13:56.892127    10 containerizer.cpp:1622] Destroying container '807e0102-ecc0-406c-b9b8-f60a845ec65f'"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:13:56.893589     7 slave.cpp:4089] Executor 'toil-28072' of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0033 exited with status 127"
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:13:56.894453     7 slave.cpp:3211] Handling status update TASK_FAILED (UUID: b449c399-dc6b-4fc0-8865-a424be91e483) for task 0 of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0033 from @0.0.0.0:0'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'W0303 20:13:56.894974     7 containerizer.cpp:1451] Ignoring update for unknown container: 807e0102-ecc0-406c-b9b8-f60a845ec65f'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:13:56.895244    13 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID: b449c399-dc6b-4fc0-8865-a424be91e483) for task 0 of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0033'
WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:13:56.895579     6 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: b449c399-dc6b-4fc0-8865-a424be91e483) for task 0 of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0033 to [email protected]:5050'
DEBUG:toil.batchSystems.mesos.batchSystem:Job 0 is in state 'TASK_FAILED' due to reason 'REASON_EXECUTOR_TERMINATED'.
WARNING:toil.batchSystems.mesos.batchSystem:Job 0 failed with message 'Executor terminated' due to reason 'REASON_EXECUTOR_TERMINATED' on executor '{'value': 'toil-28072'}' on agent '{'value': '5b6aaf7d-99c1-410a-85f3-dc9602313361-S2'}'.
DEBUG:toil.batchSystems.mesos.batchSystem:Job 0 ended with status 255, took ??? seconds.
WARNING:toil.leader:Job failed with exit value 255: 'start_toil' c4f5769c-ede3-4d06-b7a8-59fb99ecdc29