Created
March 4, 2022 11:44
-
-
Save EricCousineau-TRI/f2f67c488b75956bbb9d105cc4794ebc to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ ray_exec_all {cluster}.yaml 'ray stop' | |
| + ray_exec_all {cluster}.yaml 'ray stop' | |
| ips: ['{head_ip}', '{worker_1_ip}', '{worker_2_ip}'] | |
| [{head_ip}] + ray stop | |
| [{worker_1_ip}] + ray stop | |
| [{worker_2_ip}] + ray stop | |
| [{worker_1_ip}] Send termination request to `{site_packaes}/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --store_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10002 --max_worker_port=19999 --node_manager_port=0 --node_ip_address={worker_1_ip} --redis_address={head_ip} --redis_port=6379 --maximum_startup_concurrency=32 --static_resource_list=node:{worker_1_ip},1.0,accelerator_type:V100,1,CPU,32,GPU,4,memory,176972165530,object_store_memory,75845213798 "--python_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py {site_packaes}/ray/workers/default_worker.py --node-ip-address={worker_1_ip} --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --redis-address={head_ip}:6379 --temp-dir=/tmp/ray --metrics-agent-port=61577 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER --redis-password={secret}" "--java_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py java -Dray.address={head_ip}:6379 -Dray.raylet.node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER -Dray.object-store.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store -Dray.raylet.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet -Dray.redis.password=5241590000000000 -Dray.node-ip={worker_1_ip} -Dray.home={site_packaes}/ray/../.. -Dray.logging.dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs -Dray.session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 -cp {site_packaes}/ray/jars/* RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER io.ray.runtime.runner.worker.DefaultWorker" --cpp_worker_command= --native_library_path={site_packaes}/ray/cpp/lib --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --log_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --resource_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --metrics-agent-port=61577 --metrics_export_port=50668 --object_store_memory=75845213798 --plasma_directory=/dev/shm --ray-debugger-external=0 "--agent_command=/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/agent.py --node-ip-address={worker_1_ip} --redis-address={head_ip}:6379 --metrics-export-port=50668 --dashboard-agent-port=61577 --listen-port=0 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --runtime-env-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password={secret}"` (via SIGTERM) | |
| [{worker_1_ip}] Send termination request to `/usr/bin/python3.8 -u {site_packaes}/ray/_private/log_monitor.py --redis-address={head_ip}:6379 --logs-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password 5241590000000000` (via SIGTERM) | |
| [{worker_1_ip}] Send termination request to `{site_packaes}/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --store_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10002 --max_worker_port=19999 --node_manager_port=0 --node_ip_address={worker_1_ip} --redis_address={head_ip} --redis_port=6379 --maximum_startup_concurrency=32 --static_resource_list=node:{worker_1_ip},1.0,accelerator_type:V100,1,CPU,32,GPU,4,memory,176972165530,object_store_memory,75845213798 "--python_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py {site_packaes}/ray/workers/default_worker.py --node-ip-address={worker_1_ip} --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --redis-address={head_ip}:6379 --temp-dir=/tmp/ray --metrics-agent-port=61577 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER --redis-password={secret}" "--java_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py java -Dray.address={head_ip}:6379 -Dray.raylet.node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER -Dray.object-store.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store -Dray.raylet.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet -Dray.redis.password=5241590000000000 -Dray.node-ip={worker_1_ip} -Dray.home={site_packaes}/ray/../.. -Dray.logging.dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs -Dray.session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 -cp {site_packaes}/ray/jars/* RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER io.ray.runtime.runner.worker.DefaultWorker" --cpp_worker_command= --native_library_path={site_packaes}/ray/cpp/lib --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --log_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --resource_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --metrics-agent-port=61577 --metrics_export_port=50668 --object_store_memory=75845213798 --plasma_directory=/dev/shm --ray-debugger-external=0 "--agent_command=/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/agent.py --node-ip-address={worker_1_ip} --redis-address={head_ip}:6379 --metrics-export-port=50668 --dashboard-agent-port=61577 --listen-port=0 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --runtime-env-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password={secret}"` (via SIGTERM) | |
| [{worker_1_ip}] Send termination request to `{site_packaes}/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --store_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10002 --max_worker_port=19999 --node_manager_port=0 --node_ip_address={worker_1_ip} --redis_address={head_ip} --redis_port=6379 --maximum_startup_concurrency=32 --static_resource_list=node:{worker_1_ip},1.0,accelerator_type:V100,1,CPU,32,GPU,4,memory,176972165530,object_store_memory,75845213798 "--python_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py {site_packaes}/ray/workers/default_worker.py --node-ip-address={worker_1_ip} --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --redis-address={head_ip}:6379 --temp-dir=/tmp/ray --metrics-agent-port=61577 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER --redis-password={secret}" "--java_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py java -Dray.address={head_ip}:6379 -Dray.raylet.node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER -Dray.object-store.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store -Dray.raylet.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet -Dray.redis.password=5241590000000000 -Dray.node-ip={worker_1_ip} -Dray.home={site_packaes}/ray/../.. -Dray.logging.dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs -Dray.session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 -cp {site_packaes}/ray/jars/* RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER io.ray.runtime.runner.worker.DefaultWorker" --cpp_worker_command= --native_library_path={site_packaes}/ray/cpp/lib --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --log_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --resource_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --metrics-agent-port=61577 --metrics_export_port=50668 --object_store_memory=75845213798 --plasma_directory=/dev/shm --ray-debugger-external=0 "--agent_command=/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/agent.py --node-ip-address={worker_1_ip} --redis-address={head_ip}:6379 --metrics-export-port=50668 --dashboard-agent-port=61577 --listen-port=0 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --runtime-env-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password={secret}"` (via SIGTERM) | |
| [{worker_1_ip}] Send termination request to `{site_packaes}/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --store_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10002 --max_worker_port=19999 --node_manager_port=0 --node_ip_address={worker_1_ip} --redis_address={head_ip} --redis_port=6379 --maximum_startup_concurrency=32 --static_resource_list=node:{worker_1_ip},1.0,accelerator_type:V100,1,CPU,32,GPU,4,memory,176972165530,object_store_memory,75845213798 "--python_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py {site_packaes}/ray/workers/default_worker.py --node-ip-address={worker_1_ip} --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --redis-address={head_ip}:6379 --temp-dir=/tmp/ray --metrics-agent-port=61577 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER --redis-password={secret}" "--java_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py java -Dray.address={head_ip}:6379 -Dray.raylet.node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER -Dray.object-store.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store -Dray.raylet.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet -Dray.redis.password=5241590000000000 -Dray.node-ip={worker_1_ip} -Dray.home={site_packaes}/ray/../.. -Dray.logging.dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs -Dray.session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 -cp {site_packaes}/ray/jars/* RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER io.ray.runtime.runner.worker.DefaultWorker" --cpp_worker_command= --native_library_path={site_packaes}/ray/cpp/lib --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --log_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --resource_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --metrics-agent-port=61577 --metrics_export_port=50668 --object_store_memory=75845213798 --plasma_directory=/dev/shm --ray-debugger-external=0 "--agent_command=/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/agent.py --node-ip-address={worker_1_ip} --redis-address={head_ip}:6379 --metrics-export-port=50668 --dashboard-agent-port=61577 --listen-port=0 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --runtime-env-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password={secret}"` (via SIGTERM) | |
| [{worker_1_ip}] Send termination request to `/usr/bin/python3.8 -u {site_packaes}/ray/_private/log_monitor.py --redis-address={head_ip}:6379 --logs-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password 5241590000000000` (via SIGTERM) | |
| [{worker_1_ip}] Send termination request to `{site_packaes}/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --store_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10002 --max_worker_port=19999 --node_manager_port=0 --node_ip_address={worker_1_ip} --redis_address={head_ip} --redis_port=6379 --maximum_startup_concurrency=32 --static_resource_list=node:{worker_1_ip},1.0,accelerator_type:V100,1,CPU,32,GPU,4,memory,176972165530,object_store_memory,75845213798 "--python_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py {site_packaes}/ray/workers/default_worker.py --node-ip-address={worker_1_ip} --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --redis-address={head_ip}:6379 --temp-dir=/tmp/ray --metrics-agent-port=61577 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER --redis-password={secret}" "--java_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py java -Dray.address={head_ip}:6379 -Dray.raylet.node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER -Dray.object-store.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store -Dray.raylet.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet -Dray.redis.password=5241590000000000 -Dray.node-ip={worker_1_ip} -Dray.home={site_packaes}/ray/../.. -Dray.logging.dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs -Dray.session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 -cp {site_packaes}/ray/jars/* RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER io.ray.runtime.runner.worker.DefaultWorker" --cpp_worker_command= --native_library_path={site_packaes}/ray/cpp/lib --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --log_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --resource_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --metrics-agent-port=61577 --metrics_export_port=50668 --object_store_memory=75845213798 --plasma_directory=/dev/shm --ray-debugger-external=0 "--agent_command=/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/agent.py --node-ip-address={worker_1_ip} --redis-address={head_ip}:6379 --metrics-export-port=50668 --dashboard-agent-port=61577 --listen-port=0 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --runtime-env-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password={secret}"` (via SIGTERM) | |
| [{worker_1_ip}] Send termination request to `/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/agent.py --node-ip-address={worker_1_ip} --redis-address={head_ip}:6379 --metrics-export-port=50668 --dashboard-agent-port=61577 --listen-port=0 --node-manager-port=39285 --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --runtime-env-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password={secret}` (via SIGTERM) | |
| [{worker_1_ip}] Stopped all 8 Ray processes. | |
| [{worker_ Send termination request to `{site_packaes}/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --store_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10002 --max_worker_port=19999 --node_manager_port=0 --node_ip_address={worker_2_ip} --redis_address={head_ip} --redis_port=6379 --maximum_startup_concurrency=32 --static_resource_list=node:{worker_2_ip},1.0,accelerator_type:V100,1,CPU,32,GPU,4,memory,176649697280,object_store_memory,75707013120 "--python_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py {site_packaes}/ray/workers/default_worker.py --node-ip-address={worker_2_ip} --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --redis-address={head_ip}:6379 --temp-dir=/tmp/ray --metrics-agent-port=57455 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER --redis-password={secret}" "--java_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py java -Dray.address={head_ip}:6379 -Dray.raylet.node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER -Dray.object-store.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store -Dray.raylet.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet -Dray.redis.password=5241590000000000 -Dray.node-ip={worker_2_ip} -Dray.home={site_packaes}/ray/../.. -Dray.logging.dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs -Dray.session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 -cp {site_packaes}/ray/jars/* RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER io.ray.runtime.runner.worker.DefaultWorker" --cpp_worker_command= --native_library_path={site_packaes}/ray/cpp/lib --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --log_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --resource_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --metrics-agent-port=57455 --metrics_export_port=65384 --object_store_memory=75707013120 --plasma_directory=/dev/shm --ray-debugger-external=0 "--agent_command=/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/agent.py --node-ip-address={worker_2_ip} --redis-address={head_ip}:6379 --metrics-export-port=65384 --dashboard-agent-port=57455 --listen-port=0 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --runtime-env-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password={secret}"` (via SIGTERM) | |
| [{worker_ Send termination request to `/usr/bin/python3.8 -u {site_packaes}/ray/_private/log_monitor.py --redis-address={head_ip}:6379 --logs-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password 5241590000000000` (via SIGTERM) | |
| [{worker_ Send termination request to `{site_packaes}/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --store_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10002 --max_worker_port=19999 --node_manager_port=0 --node_ip_address={worker_2_ip} --redis_address={head_ip} --redis_port=6379 --maximum_startup_concurrency=32 --static_resource_list=node:{worker_2_ip},1.0,accelerator_type:V100,1,CPU,32,GPU,4,memory,176649697280,object_store_memory,75707013120 "--python_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py {site_packaes}/ray/workers/default_worker.py --node-ip-address={worker_2_ip} --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --redis-address={head_ip}:6379 --temp-dir=/tmp/ray --metrics-agent-port=57455 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER --redis-password={secret}" "--java_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py java -Dray.address={head_ip}:6379 -Dray.raylet.node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER -Dray.object-store.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store -Dray.raylet.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet -Dray.redis.password=5241590000000000 -Dray.node-ip={worker_2_ip} -Dray.home={site_packaes}/ray/../.. -Dray.logging.dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs -Dray.session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 -cp {site_packaes}/ray/jars/* RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER io.ray.runtime.runner.worker.DefaultWorker" --cpp_worker_command= --native_library_path={site_packaes}/ray/cpp/lib --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --log_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --resource_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --metrics-agent-port=57455 --metrics_export_port=65384 --object_store_memory=75707013120 --plasma_directory=/dev/shm --ray-debugger-external=0 "--agent_command=/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/agent.py --node-ip-address={worker_2_ip} --redis-address={head_ip}:6379 --metrics-export-port=65384 --dashboard-agent-port=57455 --listen-port=0 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --runtime-env-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password={secret}"` (via SIGTERM) | |
| [{worker_ Send termination request to `{site_packaes}/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --store_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10002 --max_worker_port=19999 --node_manager_port=0 --node_ip_address={worker_2_ip} --redis_address={head_ip} --redis_port=6379 --maximum_startup_concurrency=32 --static_resource_list=node:{worker_2_ip},1.0,accelerator_type:V100,1,CPU,32,GPU,4,memory,176649697280,object_store_memory,75707013120 "--python_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py {site_packaes}/ray/workers/default_worker.py --node-ip-address={worker_2_ip} --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --redis-address={head_ip}:6379 --temp-dir=/tmp/ray --metrics-agent-port=57455 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER --redis-password={secret}" "--java_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py java -Dray.address={head_ip}:6379 -Dray.raylet.node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER -Dray.object-store.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store -Dray.raylet.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet -Dray.redis.password=5241590000000000 -Dray.node-ip={worker_2_ip} -Dray.home={site_packaes}/ray/../.. -Dray.logging.dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs -Dray.session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 -cp {site_packaes}/ray/jars/* RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER io.ray.runtime.runner.worker.DefaultWorker" --cpp_worker_command= --native_library_path={site_packaes}/ray/cpp/lib --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --log_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --resource_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --metrics-agent-port=57455 --metrics_export_port=65384 --object_store_memory=75707013120 --plasma_directory=/dev/shm --ray-debugger-external=0 "--agent_command=/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/agent.py --node-ip-address={worker_2_ip} --redis-address={head_ip}:6379 --metrics-export-port=65384 --dashboard-agent-port=57455 --listen-port=0 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --runtime-env-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password={secret}"` (via SIGTERM) | |
| [{worker_ Send termination request to `{site_packaes}/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --store_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10002 --max_worker_port=19999 --node_manager_port=0 --node_ip_address={worker_2_ip} --redis_address={head_ip} --redis_port=6379 --maximum_startup_concurrency=32 --static_resource_list=node:{worker_2_ip},1.0,accelerator_type:V100,1,CPU,32,GPU,4,memory,176649697280,object_store_memory,75707013120 "--python_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py {site_packaes}/ray/workers/default_worker.py --node-ip-address={worker_2_ip} --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --redis-address={head_ip}:6379 --temp-dir=/tmp/ray --metrics-agent-port=57455 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER --redis-password={secret}" "--java_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py java -Dray.address={head_ip}:6379 -Dray.raylet.node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER -Dray.object-store.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store -Dray.raylet.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet -Dray.redis.password=5241590000000000 -Dray.node-ip={worker_2_ip} -Dray.home={site_packaes}/ray/../.. -Dray.logging.dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs -Dray.session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 -cp {site_packaes}/ray/jars/* RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER io.ray.runtime.runner.worker.DefaultWorker" --cpp_worker_command= --native_library_path={site_packaes}/ray/cpp/lib --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --log_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --resource_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --metrics-agent-port=57455 --metrics_export_port=65384 --object_store_memory=75707013120 --plasma_directory=/dev/shm --ray-debugger-external=0 "--agent_command=/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/agent.py --node-ip-address={worker_2_ip} --redis-address={head_ip}:6379 --metrics-export-port=65384 --dashboard-agent-port=57455 --listen-port=0 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --runtime-env-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password={secret}"` (via SIGTERM) | |
| [{worker_ Send termination request to `/usr/bin/python3.8 -u {site_packaes}/ray/_private/log_monitor.py --redis-address={head_ip}:6379 --logs-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password 5241590000000000` (via SIGTERM) | |
| [{worker_ Send termination request to `{site_packaes}/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --store_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10002 --max_worker_port=19999 --node_manager_port=0 --node_ip_address={worker_2_ip} --redis_address={head_ip} --redis_port=6379 --maximum_startup_concurrency=32 --static_resource_list=node:{worker_2_ip},1.0,accelerator_type:V100,1,CPU,32,GPU,4,memory,176649697280,object_store_memory,75707013120 "--python_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py {site_packaes}/ray/workers/default_worker.py --node-ip-address={worker_2_ip} --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --redis-address={head_ip}:6379 --temp-dir=/tmp/ray --metrics-agent-port=57455 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER --redis-password={secret}" "--java_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py java -Dray.address={head_ip}:6379 -Dray.raylet.node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER -Dray.object-store.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store -Dray.raylet.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet -Dray.redis.password=5241590000000000 -Dray.node-ip={worker_2_ip} -Dray.home={site_packaes}/ray/../.. -Dray.logging.dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs -Dray.session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 -cp {site_packaes}/ray/jars/* RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER io.ray.runtime.runner.worker.DefaultWorker" --cpp_worker_command= --native_library_path={site_packaes}/ray/cpp/lib --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --log_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --resource_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --metrics-agent-port=57455 --metrics_export_port=65384 --object_store_memory=75707013120 --plasma_directory=/dev/shm --ray-debugger-external=0 "--agent_command=/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/agent.py --node-ip-address={worker_2_ip} --redis-address={head_ip}:6379 --metrics-export-port=65384 --dashboard-agent-port=57455 --listen-port=0 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --runtime-env-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password={secret}"` (via SIGTERM) | |
| [{worker_ Send termination request to `/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/agent.py --node-ip-address={worker_2_ip} --redis-address={head_ip}:6379 --metrics-export-port=65384 --dashboard-agent-port=57455 --listen-port=0 --node-manager-port=46591 --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --runtime-env-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password={secret}` (via SIGTERM) | |
| [{worker_ Stopped all 8 Ray processes. | |
| [{head_ip}] Send termination request to `{site_packaes}/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --store_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10002 --max_worker_port=19999 --node_manager_port=0 --node_ip_address={head_ip} --redis_address={head_ip} --redis_port=6379 --maximum_startup_concurrency=1 --static_resource_list=node:{head_ip},1.0,accelerator_type:V100,1,memory,168069508916,object_store_memory,76315503820 "--python_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py {site_packaes}/ray/workers/default_worker.py --node-ip-address={head_ip} --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --redis-address={head_ip}:6379 --temp-dir=/tmp/ray --metrics-agent-port=47413 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER --redis-password={secret}" "--java_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py java -Dray.address={head_ip}:6379 -Dray.raylet.node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER -Dray.object-store.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store -Dray.raylet.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet -Dray.redis.password=5241590000000000 -Dray.node-ip={head_ip} -Dray.home={site_packaes}/ray/../.. -Dray.logging.dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs -Dray.session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 -cp {site_packaes}/ray/jars/* RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER io.ray.runtime.runner.worker.DefaultWorker" --cpp_worker_command= --native_library_path={site_packaes}/ray/cpp/lib --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --log_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --resource_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --metrics-agent-port=47413 --metrics_export_port=50400 --object_store_memory=76315503820 --plasma_directory=/dev/shm --ray-debugger-external=0 "--agent_command=/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/agent.py --node-ip-address={head_ip} --redis-address={head_ip}:6379 --metrics-export-port=50400 --dashboard-agent-port=47413 --listen-port=0 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --runtime-env-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password={secret}"` (via SIGTERM) | |
| [{head_ip}] Send termination request to `{site_packaes}/ray/core/src/ray/gcs/gcs_server --redis_address={head_ip} --redis_port=6379 --log_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --config_list=eyJvYmplY3Rfc3BpbGxpbmdfY29uZmlnIjogIntcInR5cGVcIjogXCJmaWxlc3lzdGVtXCIsIFwicGFyYW1zXCI6IHtcImRpcmVjdG9yeV9wYXRoXCI6IFwiL3RtcC9yYXkvc2Vzc2lvbl8yMDIyLTAzLTA0XzAxLTU3LTQ1XzMzODEyM18yMTIzMlwifX0iLCAiaXNfZXh0ZXJuYWxfc3RvcmFnZV90eXBlX2ZzIjogdHJ1ZX0= --gcs_server_port=0 --metrics-agent-port=47413 --node-ip-address={head_ip} --redis_password=5241590000000000` (via SIGTERM) | |
| [{head_ip}] Send termination request to `/usr/bin/python3.8 -u {site_packaes}/ray/autoscaler/_private/monitor.py --logs-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --redis-address={head_ip}:6379 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --autoscaling-config=~/ray_bootstrap_config.yaml --redis-password={secret} --monitor-ip={head_ip}` (via SIGTERM) | |
| [{head_ip}] Send termination request to `/usr/bin/python3.8 -u {site_packaes}/ray/_private/log_monitor.py --redis-address={head_ip}:6379 --logs-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password 5241590000000000` (via SIGTERM) | |
| [{head_ip}] Send termination request to `/usr/bin/python3.8 -m ray.util.client.server --redis-address={head_ip}:6379 --host=0.0.0.0 --port=10001 --mode=proxy --redis-password={secret} --metrics-agent-port=47413 --worker-shim-pid=21265` (via SIGTERM) | |
| [{head_ip}] Send termination request to `"{site_packaes}/ray/core/src/ray/thirdparty/redis/src/redis-server *:6379" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""` (via SIGTERM) | |
| [{head_ip}] Send termination request to `"{site_packaes}/ray/core/src/ray/thirdparty/redis/src/redis-server *:28502" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""` (via SIGTERM) | |
| [{head_ip}] Send termination request to `{site_packaes}/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --store_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10002 --max_worker_port=19999 --node_manager_port=0 --node_ip_address={head_ip} --redis_address={head_ip} --redis_port=6379 --maximum_startup_concurrency=1 --static_resource_list=node:{head_ip},1.0,accelerator_type:V100,1,memory,168069508916,object_store_memory,76315503820 "--python_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py {site_packaes}/ray/workers/default_worker.py --node-ip-address={head_ip} --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --redis-address={head_ip}:6379 --temp-dir=/tmp/ray --metrics-agent-port=47413 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER --redis-password={secret}" "--java_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py java -Dray.address={head_ip}:6379 -Dray.raylet.node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER -Dray.object-store.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store -Dray.raylet.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet -Dray.redis.password=5241590000000000 -Dray.node-ip={head_ip} -Dray.home={site_packaes}/ray/../.. -Dray.logging.dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs -Dray.session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 -cp {site_packaes}/ray/jars/* RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER io.ray.runtime.runner.worker.DefaultWorker" --cpp_worker_command= --native_library_path={site_packaes}/ray/cpp/lib --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --log_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --resource_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --metrics-agent-port=47413 --metrics_export_port=50400 --object_store_memory=76315503820 --plasma_directory=/dev/shm --ray-debugger-external=0 "--agent_command=/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/agent.py --node-ip-address={head_ip} --redis-address={head_ip}:6379 --metrics-export-port=50400 --dashboard-agent-port=47413 --listen-port=0 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --runtime-env-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password={secret}"` (via SIGTERM) | |
| [{head_ip}] Send termination request to `{site_packaes}/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --store_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10002 --max_worker_port=19999 --node_manager_port=0 --node_ip_address={head_ip} --redis_address={head_ip} --redis_port=6379 --maximum_startup_concurrency=1 --static_resource_list=node:{head_ip},1.0,accelerator_type:V100,1,memory,168069508916,object_store_memory,76315503820 "--python_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py {site_packaes}/ray/workers/default_worker.py --node-ip-address={head_ip} --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --redis-address={head_ip}:6379 --temp-dir=/tmp/ray --metrics-agent-port=47413 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER --redis-password={secret}" "--java_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py java -Dray.address={head_ip}:6379 -Dray.raylet.node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER -Dray.object-store.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store -Dray.raylet.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet -Dray.redis.password=5241590000000000 -Dray.node-ip={head_ip} -Dray.home={site_packaes}/ray/../.. -Dray.logging.dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs -Dray.session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 -cp {site_packaes}/ray/jars/* RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER io.ray.runtime.runner.worker.DefaultWorker" --cpp_worker_command= --native_library_path={site_packaes}/ray/cpp/lib --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --log_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --resource_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --metrics-agent-port=47413 --metrics_export_port=50400 --object_store_memory=76315503820 --plasma_directory=/dev/shm --ray-debugger-external=0 "--agent_command=/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/agent.py --node-ip-address={head_ip} --redis-address={head_ip}:6379 --metrics-export-port=50400 --dashboard-agent-port=47413 --listen-port=0 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --runtime-env-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password={secret}"` (via SIGTERM) | |
| [{head_ip}] Send termination request to `{site_packaes}/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --store_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10002 --max_worker_port=19999 --node_manager_port=0 --node_ip_address={head_ip} --redis_address={head_ip} --redis_port=6379 --maximum_startup_concurrency=1 --static_resource_list=node:{head_ip},1.0,accelerator_type:V100,1,memory,168069508916,object_store_memory,76315503820 "--python_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py {site_packaes}/ray/workers/default_worker.py --node-ip-address={head_ip} --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --redis-address={head_ip}:6379 --temp-dir=/tmp/ray --metrics-agent-port=47413 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER --redis-password={secret}" "--java_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py java -Dray.address={head_ip}:6379 -Dray.raylet.node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER -Dray.object-store.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store -Dray.raylet.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet -Dray.redis.password=5241590000000000 -Dray.node-ip={head_ip} -Dray.home={site_packaes}/ray/../.. -Dray.logging.dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs -Dray.session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 -cp {site_packaes}/ray/jars/* RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER io.ray.runtime.runner.worker.DefaultWorker" --cpp_worker_command= --native_library_path={site_packaes}/ray/cpp/lib --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --log_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --resource_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --metrics-agent-port=47413 --metrics_export_port=50400 --object_store_memory=76315503820 --plasma_directory=/dev/shm --ray-debugger-external=0 "--agent_command=/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/agent.py --node-ip-address={head_ip} --redis-address={head_ip}:6379 --metrics-export-port=50400 --dashboard-agent-port=47413 --listen-port=0 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --runtime-env-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password={secret}"` (via SIGTERM) | |
| [{head_ip}] Send termination request to `/usr/bin/python3.8 -u {site_packaes}/ray/_private/log_monitor.py --redis-address={head_ip}:6379 --logs-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password 5241590000000000` (via SIGTERM) | |
| [{head_ip}] Send termination request to `/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/dashboard.py --host=localhost --port=8265 --port-retries=0 --redis-address={head_ip}:6379 --temp-dir=/tmp/ray --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password 5241590000000000` (via SIGTERM) | |
| [{head_ip}] Send termination request to `{site_packaes}/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --store_socket_name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10002 --max_worker_port=19999 --node_manager_port=0 --node_ip_address={head_ip} --redis_address={head_ip} --redis_port=6379 --maximum_startup_concurrency=1 --static_resource_list=node:{head_ip},1.0,accelerator_type:V100,1,memory,168069508916,object_store_memory,76315503820 "--python_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py {site_packaes}/ray/workers/default_worker.py --node-ip-address={head_ip} --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --redis-address={head_ip}:6379 --temp-dir=/tmp/ray --metrics-agent-port=47413 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER --redis-password={secret}" "--java_worker_command=/usr/bin/python3.8 {site_packaes}/ray/workers/setup_worker.py java -Dray.address={head_ip}:6379 -Dray.raylet.node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER -Dray.object-store.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store -Dray.raylet.socket-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet -Dray.redis.password=5241590000000000 -Dray.node-ip={head_ip} -Dray.home={site_packaes}/ray/../.. -Dray.logging.dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs -Dray.session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 -cp {site_packaes}/ray/jars/* RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER io.ray.runtime.runner.worker.DefaultWorker" --cpp_worker_command= --native_library_path={site_packaes}/ray/cpp/lib --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --log_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --resource_dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --metrics-agent-port=47413 --metrics_export_port=50400 --object_store_memory=76315503820 --plasma_directory=/dev/shm --ray-debugger-external=0 "--agent_command=/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/agent.py --node-ip-address={head_ip} --redis-address={head_ip}:6379 --metrics-export-port=50400 --dashboard-agent-port=47413 --listen-port=0 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --runtime-env-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password={secret}"` (via SIGTERM) | |
| [{head_ip}] Send termination request to `/usr/bin/python3.8 -u {site_packaes}/ray/dashboard/agent.py --node-ip-address={head_ip} --redis-address={head_ip}:6379 --metrics-export-port=50400 --dashboard-agent-port=47413 --listen-port=0 --node-manager-port=43025 --object-store-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232 --runtime-env-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/runtime_resources --log-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password={secret}` (via SIGTERM) | |
| [{head_ip}] Stopped all 14 Ray processes. | |
| $ ray_exec_all {cluster}.yaml 'ray stop' | |
| + ray_exec_all {cluster}.yaml 'ray stop' | |
| ips: ['{head_ip}', '{worker_1_ip}', '{worker_2_ip}'] | |
| [{head_ip}] + ray stop | |
| [{worker_1_ip}] + ray stop | |
| [{worker_2_ip}] + ray stop | |
| [{worker_ Did not find any active Ray processes. | |
| [{worker_1_ip}] Did not find any active Ray processes. | |
| [{head_ip}] Send termination request to `/usr/bin/python3.8 -u {site_packaes}/ray/autoscaler/_private/monitor.py --logs-dir=/tmp/ray/session_2022-03-04_01-57-45_338123_21232/logs --redis-address={head_ip}:6379 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --autoscaling-config=~/ray_bootstrap_config.yaml --redis-password={secret} --monitor-ip={head_ip}` (via SIGTERM) | |
| [{head_ip}] Stopped all 1 Ray processes. | |
| $ ray_exec_all {cluster}.yaml 'ray stop' | |
| + ray_exec_all {cluster}.yaml 'ray stop' | |
| ips: ['{head_ip}', '{worker_1_ip}', '{worker_2_ip}'] | |
| [{head_ip}] + ray stop | |
| [{worker_1_ip}] + ray stop | |
| [{worker_2_ip}] + ray stop | |
| [{head_ip}] Did not find any active Ray processes. | |
| [{worker_ Did not find any active Ray processes. | |
| [{worker_1_ip}] Did not find any active Ray processes. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment