Created
July 28, 2024 16:54
-
-
Save relyt0925/7ad70ec81b1e0ea8aaf9a41153c3d555 to your computer and use it in GitHub Desktop.
new model serve logs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@tyler-rhel-newimage root]# /root/ilab model serve --model-family mixtral --model-path /var/instructlabbigdisk/instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1/ --backend vllm -- --tensor-parallel-size 8 --host 127.0.0.1 --port 8084 | |
| INFO 2024-07-28 16:53:08,009 instructlab.model.serve:136: Using model '/var/instructlabbigdisk/instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1' with -1 gpu-layers and 4096 max context size. | |
| INFO 2024-07-28 16:53:08,009 instructlab.model.serve:140: Serving model '/var/instructlabbigdisk/instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1' with vllm | |
| INFO 2024-07-28 16:53:08,010 instructlab.model.backends.vllm:196: vLLM starting up on pid 64 at http://127.0.0.1:8000/v1 | |
| INFO 07-28 16:53:13 api_server.py:219] vLLM API server version 0.5.3.post1 | |
| INFO 07-28 16:53:13 api_server.py:220] args: Namespace(host='127.0.0.1', port=8084, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template='/tmp/tmpnpi39gwa', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='/var/instructlabbigdisk/instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='outlines', distributed_executor_backend='mp', worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=8, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None) | |
| INFO 07-28 16:53:13 llm_engine.py:176] Initializing an LLM engine (v0.5.3.post1) with config: model='/var/instructlabbigdisk/instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1', speculative_config=None, tokenizer='/var/instructlabbigdisk/instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=8, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=/var/instructlabbigdisk/instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1, use_v2_block_manager=False, enable_prefix_caching=False) | |
| INFO 07-28 16:53:13 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager | |
| (VllmWorkerProcess pid=87) INFO 07-28 16:53:13 multiproc_worker_utils.py:215] Worker ready; awaiting tasks | |
| (VllmWorkerProcess pid=88) INFO 07-28 16:53:13 multiproc_worker_utils.py:215] Worker ready; awaiting tasks | |
| (VllmWorkerProcess pid=91) INFO 07-28 16:53:13 multiproc_worker_utils.py:215] Worker ready; awaiting tasks | |
| (VllmWorkerProcess pid=89) INFO 07-28 16:53:13 multiproc_worker_utils.py:215] Worker ready; awaiting tasks | |
| (VllmWorkerProcess pid=90) INFO 07-28 16:53:13 multiproc_worker_utils.py:215] Worker ready; awaiting tasks | |
| (VllmWorkerProcess pid=92) INFO 07-28 16:53:13 multiproc_worker_utils.py:215] Worker ready; awaiting tasks | |
| (VllmWorkerProcess pid=93) INFO 07-28 16:53:13 multiproc_worker_utils.py:215] Worker ready; awaiting tasks | |
| DEBUG 07-28 16:53:19 parallel_state.py:803] world_size=8 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:45143 backend=nccl | |
| (VllmWorkerProcess pid=87) DEBUG 07-28 16:53:20 parallel_state.py:803] world_size=8 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:45143 backend=nccl | |
| (VllmWorkerProcess pid=92) DEBUG 07-28 16:53:20 parallel_state.py:803] world_size=8 rank=6 local_rank=6 distributed_init_method=tcp://127.0.0.1:45143 backend=nccl | |
| (VllmWorkerProcess pid=89) DEBUG 07-28 16:53:20 parallel_state.py:803] world_size=8 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:45143 backend=nccl | |
| (VllmWorkerProcess pid=88) DEBUG 07-28 16:53:20 parallel_state.py:803] world_size=8 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:45143 backend=nccl | |
| (VllmWorkerProcess pid=91) DEBUG 07-28 16:53:20 parallel_state.py:803] world_size=8 rank=5 local_rank=5 distributed_init_method=tcp://127.0.0.1:45143 backend=nccl | |
| (VllmWorkerProcess pid=90) DEBUG 07-28 16:53:20 parallel_state.py:803] world_size=8 rank=4 local_rank=4 distributed_init_method=tcp://127.0.0.1:45143 backend=nccl | |
| (VllmWorkerProcess pid=93) DEBUG 07-28 16:53:20 parallel_state.py:803] world_size=8 rank=7 local_rank=7 distributed_init_method=tcp://127.0.0.1:45143 backend=nccl | |
| INFO 07-28 16:53:25 utils.py:784] Found nccl from library libnccl.so.2 | |
| (VllmWorkerProcess pid=87) INFO 07-28 16:53:25 utils.py:784] Found nccl from library libnccl.so.2 | |
| (VllmWorkerProcess pid=93) INFO 07-28 16:53:25 utils.py:784] Found nccl from library libnccl.so.2 | |
| INFO 07-28 16:53:25 pynccl.py:63] vLLM is using nccl==2.20.5 | |
| (VllmWorkerProcess pid=87) INFO 07-28 16:53:25 pynccl.py:63] vLLM is using nccl==2.20.5 | |
| (VllmWorkerProcess pid=89) INFO 07-28 16:53:25 utils.py:784] Found nccl from library libnccl.so.2 | |
| (VllmWorkerProcess pid=93) INFO 07-28 16:53:25 pynccl.py:63] vLLM is using nccl==2.20.5 | |
| (VllmWorkerProcess pid=88) INFO 07-28 16:53:25 utils.py:784] Found nccl from library libnccl.so.2 | |
| (VllmWorkerProcess pid=92) INFO 07-28 16:53:25 utils.py:784] Found nccl from library libnccl.so.2 | |
| (VllmWorkerProcess pid=89) INFO 07-28 16:53:25 pynccl.py:63] vLLM is using nccl==2.20.5 | |
| (VllmWorkerProcess pid=88) INFO 07-28 16:53:25 pynccl.py:63] vLLM is using nccl==2.20.5 | |
| (VllmWorkerProcess pid=92) INFO 07-28 16:53:25 pynccl.py:63] vLLM is using nccl==2.20.5 | |
| (VllmWorkerProcess pid=91) INFO 07-28 16:53:25 utils.py:784] Found nccl from library libnccl.so.2 | |
| (VllmWorkerProcess pid=91) INFO 07-28 16:53:25 pynccl.py:63] vLLM is using nccl==2.20.5 | |
| (VllmWorkerProcess pid=90) INFO 07-28 16:53:25 utils.py:784] Found nccl from library libnccl.so.2 | |
| (VllmWorkerProcess pid=90) INFO 07-28 16:53:25 pynccl.py:63] vLLM is using nccl==2.20.5 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO cudaDriverVersion 12040 | |
| NCCL version 2.20.5+cuda12.4 | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO comm 0x56135ee82130 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0x4b4b0976e3188ba5 - Init START | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO comm 0x56135ee88f90 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0x4b4b0976e3188ba5 - Init START | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO comm 0x56135ee832b0 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0x4b4b0976e3188ba5 - Init START | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO comm 0x56135ee80ce0 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0x4b4b0976e3188ba5 - Init START | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO comm 0x56135ee83ea0 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0x4b4b0976e3188ba5 - Init START | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO comm 0x56135ee83330 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0x4b4b0976e3188ba5 - Init START | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO comm 0x56135ee82fe0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0x4b4b0976e3188ba5 - Init START | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO comm 0x56135ee81a40 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0x4b4b0976e3188ba5 - Init START | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Setting affinity for GPU 7 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO NVLS multicast support is not available on dev 7 | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Setting affinity for GPU 4 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO NVLS multicast support is not available on dev 2 | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO NVLS multicast support is not available on dev 3 | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO NVLS multicast support is not available on dev 4 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Setting affinity for GPU 5 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO NVLS multicast support is not available on dev 5 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO NVLS multicast support is not available on dev 0 | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO NVLS multicast support is not available on dev 1 | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Setting affinity for GPU 6 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO NVLS multicast support is not available on dev 6 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO comm 0x56135ee88f90 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0 | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO comm 0x56135ee80ce0 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0 | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO comm 0x56135ee82130 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0 | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO comm 0x56135ee82fe0 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0 | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO comm 0x56135ee83330 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0 | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO comm 0x56135ee81a40 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0 | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO comm 0x56135ee832b0 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0 | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO comm 0x56135ee83ea0 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0 | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3 | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4 | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2 | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 04/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 04/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 05/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 05/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 06/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 06/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 07/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 08/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 07/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 08/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 09/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 04/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 10/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 05/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 12/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 11/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 12/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 13/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 06/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 13/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 12/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 13/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 13/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 14/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 07/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 14/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 13/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 14/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 14/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 15/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 08/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 15/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 14/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 15/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 16/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 09/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 16/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 16/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 15/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 16/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 17/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 10/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 17/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 17/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 16/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 17/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 11/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 18/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 18/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 18/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 17/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 18/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 12/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 19/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 19/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 19/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 18/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 19/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 13/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 20/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 20/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 20/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 19/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 20/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 14/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 21/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 21/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 21/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 20/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 21/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 15/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 22/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 22/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 22/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 21/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 22/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 16/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 23/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 23/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 23/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 22/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 23/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 17/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 23/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 18/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 19/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 20/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 21/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 22/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 23/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 01/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 03/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 04/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 05/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 06/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 07/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 08/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 09/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 10/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 11/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 12/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 13/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 14/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 15/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 16/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 17/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 18/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 19/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 20/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 21/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 22/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Channel 23/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 00/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 01/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 01/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 02/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 03/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 03/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 05/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 04/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 04/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 04/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 05/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 05/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 05/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 06/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 06/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 06/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 07/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 07/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 07/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 08/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 08/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 08/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 09/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 09/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 09/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 11/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 10/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 10/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 10/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 11/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 11/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 11/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 13/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 12/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 12/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 12/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 13/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 13/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 13/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 14/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 14/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 14/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 16/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 15/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 15/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 15/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 16/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 17/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 16/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 16/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 17/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 18/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 17/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 17/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 18/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 19/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 18/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 18/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 19/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 20/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 19/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 19/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 20/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 21/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 20/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 20/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 21/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 22/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 21/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 21/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 22/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 22/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Channel 23/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Channel 23/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Channel 23/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 22/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Channel 23/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:92:92 [6] NCCL INFO comm 0x56135ee83330 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0x4b4b0976e3188ba5 - Init COMPLETE | |
| tyler-rhel-newimage:91:91 [5] NCCL INFO comm 0x56135ee832b0 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0x4b4b0976e3188ba5 - Init COMPLETE | |
| tyler-rhel-newimage:93:93 [7] NCCL INFO comm 0x56135ee82fe0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0x4b4b0976e3188ba5 - Init COMPLETE | |
| tyler-rhel-newimage:90:90 [4] NCCL INFO comm 0x56135ee83ea0 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0x4b4b0976e3188ba5 - Init COMPLETE | |
| tyler-rhel-newimage:88:88 [2] NCCL INFO comm 0x56135ee80ce0 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0x4b4b0976e3188ba5 - Init COMPLETE | |
| tyler-rhel-newimage:87:87 [1] NCCL INFO comm 0x56135ee82130 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0x4b4b0976e3188ba5 - Init COMPLETE | |
| tyler-rhel-newimage:64:64 [0] NCCL INFO comm 0x56135ee88f90 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0x4b4b0976e3188ba5 - Init COMPLETE | |
| tyler-rhel-newimage:89:89 [3] NCCL INFO comm 0x56135ee81a40 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0x4b4b0976e3188ba5 - Init COMPLETE | |
| (VllmWorkerProcess pid=87) INFO 07-28 16:53:29 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /var/instructlabbigdisk/instructlab/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json | |
| (VllmWorkerProcess pid=92) INFO 07-28 16:53:29 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /var/instructlabbigdisk/instructlab/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json | |
| INFO 07-28 16:53:29 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /var/instructlabbigdisk/instructlab/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json | |
| (VllmWorkerProcess pid=88) INFO 07-28 16:53:29 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /var/instructlabbigdisk/instructlab/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json | |
| (VllmWorkerProcess pid=91) INFO 07-28 16:53:29 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /var/instructlabbigdisk/instructlab/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json | |
| (VllmWorkerProcess pid=93) INFO 07-28 16:53:29 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /var/instructlabbigdisk/instructlab/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json | |
| (VllmWorkerProcess pid=89) INFO 07-28 16:53:29 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /var/instructlabbigdisk/instructlab/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json | |
| (VllmWorkerProcess pid=90) INFO 07-28 16:53:29 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /var/instructlabbigdisk/instructlab/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json | |
| INFO 07-28 16:53:29 shm_broadcast.py:241] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3, 4, 5, 6, 7], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7f2a3a98cc90>, local_subscribe_port=36171, local_sync_port=33681, remote_subscribe_port=None, remote_sync_port=None) | |
| INFO 07-28 16:53:29 model_runner.py:680] Starting to load model /var/instructlabbigdisk/instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1... | |
| (VllmWorkerProcess pid=88) INFO 07-28 16:53:29 model_runner.py:680] Starting to load model /var/instructlabbigdisk/instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1... | |
| (VllmWorkerProcess pid=87) INFO 07-28 16:53:29 model_runner.py:680] Starting to load model /var/instructlabbigdisk/instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1... | |
| (VllmWorkerProcess pid=89) INFO 07-28 16:53:29 model_runner.py:680] Starting to load model /var/instructlabbigdisk/instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1... | |
| (VllmWorkerProcess pid=93) INFO 07-28 16:53:29 model_runner.py:680] Starting to load model /var/instructlabbigdisk/instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1... | |
| (VllmWorkerProcess pid=90) INFO 07-28 16:53:29 model_runner.py:680] Starting to load model /var/instructlabbigdisk/instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1... | |
| (VllmWorkerProcess pid=91) INFO 07-28 16:53:29 model_runner.py:680] Starting to load model /var/instructlabbigdisk/instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1... | |
| (VllmWorkerProcess pid=92) INFO 07-28 16:53:29 model_runner.py:680] Starting to load model /var/instructlabbigdisk/instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1... | |
| Loading safetensors checkpoint shards: 0% Completed | 0/19 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 5% Completed | 1/19 [00:00<00:05, 3.03it/s] | |
| Loading safetensors checkpoint shards: 11% Completed | 2/19 [00:00<00:05, 3.00it/s] | |
| Loading safetensors checkpoint shards: 16% Completed | 3/19 [00:01<00:05, 2.97it/s] | |
| Loading safetensors checkpoint shards: 21% Completed | 4/19 [00:01<00:05, 2.92it/s] | |
| Loading safetensors checkpoint shards: 26% Completed | 5/19 [00:01<00:04, 2.87it/s] | |
| Loading safetensors checkpoint shards: 32% Completed | 6/19 [00:02<00:04, 2.94it/s] | |
| Loading safetensors checkpoint shards: 37% Completed | 7/19 [00:02<00:04, 2.89it/s] | |
| Loading safetensors checkpoint shards: 42% Completed | 8/19 [00:02<00:03, 2.93it/s] | |
| Loading safetensors checkpoint shards: 47% Completed | 9/19 [00:03<00:03, 2.89it/s] | |
| Loading safetensors checkpoint shards: 53% Completed | 10/19 [00:03<00:03, 2.93it/s] | |
| Loading safetensors checkpoint shards: 58% Completed | 11/19 [00:03<00:02, 2.93it/s] | |
| Loading safetensors checkpoint shards: 63% Completed | 12/19 [00:04<00:02, 2.85it/s] | |
| Loading safetensors checkpoint shards: 68% Completed | 13/19 [00:04<00:02, 2.87it/s] | |
| Loading safetensors checkpoint shards: 74% Completed | 14/19 [00:04<00:01, 2.82it/s] | |
| Loading safetensors checkpoint shards: 79% Completed | 15/19 [00:05<00:01, 2.85it/s] | |
| Loading safetensors checkpoint shards: 84% Completed | 16/19 [00:05<00:01, 2.90it/s] | |
| Loading safetensors checkpoint shards: 89% Completed | 17/19 [00:05<00:00, 2.99it/s] | |
| (VllmWorkerProcess pid=90) INFO 07-28 16:53:35 model_runner.py:692] Loading model weights took 10.8853 GB | |
| Loading safetensors checkpoint shards: 95% Completed | 18/19 [00:06<00:00, 3.16it/s] | |
| (VllmWorkerProcess pid=87) INFO 07-28 16:53:36 model_runner.py:692] Loading model weights took 10.8853 GB | |
| Loading safetensors checkpoint shards: 100% Completed | 19/19 [00:06<00:00, 3.48it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 19/19 [00:06<00:00, 3.01it/s] | |
| (VllmWorkerProcess pid=93) INFO 07-28 16:53:36 model_runner.py:692] Loading model weights took 10.8853 GB | |
| (VllmWorkerProcess pid=91) INFO 07-28 16:53:36 model_runner.py:692] Loading model weights took 10.8853 GB | |
| (VllmWorkerProcess pid=89) INFO 07-28 16:53:36 model_runner.py:692] Loading model weights took 10.8853 GB | |
| (VllmWorkerProcess pid=88) INFO 07-28 16:53:36 model_runner.py:692] Loading model weights took 10.8853 GB | |
| INFO 07-28 16:53:36 model_runner.py:692] Loading model weights took 10.8853 GB | |
| (VllmWorkerProcess pid=92) INFO 07-28 16:53:36 model_runner.py:692] Loading model weights took 10.8853 GB | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO comm 0x561375efd810 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0x1d1c3e4e99144675 - Init START | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO comm 0x561375f1f3b0 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0x1d1c3e4e99144675 - Init START | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO comm 0x561375f09d80 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0x1d1c3e4e99144675 - Init START | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO comm 0x561375f1f3b0 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0x1d1c3e4e99144675 - Init START | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO comm 0x561375f1f9e0 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0x1d1c3e4e99144675 - Init START | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO comm 0x561375f1d7a0 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0x1d1c3e4e99144675 - Init START | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO comm 0x561375f20c60 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0x1d1c3e4e99144675 - Init START | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO comm 0x561375f1fc10 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0x1d1c3e4e99144675 - Init START | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO NVLS multicast support is not available on dev 1 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO NVLS multicast support is not available on dev 0 | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO NVLS multicast support is not available on dev 2 | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Setting affinity for GPU 7 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO NVLS multicast support is not available on dev 7 | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Setting affinity for GPU 4 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO NVLS multicast support is not available on dev 4 | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Setting affinity for GPU 6 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO NVLS multicast support is not available on dev 6 | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO NVLS multicast support is not available on dev 3 | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Setting affinity for GPU 5 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO NVLS multicast support is not available on dev 5 | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO comm 0x561375f1f9e0 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0 | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO comm 0x561375f1d7a0 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0 | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO comm 0x561375f1fc10 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0 | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO comm 0x561375f1f3b0 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO comm 0x561375efd810 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0 | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO comm 0x561375f09d80 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0 | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO comm 0x561375f20c60 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0 | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO comm 0x561375f1f3b0 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0 | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2 | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1 | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3 | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6 | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4 | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 04/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 04/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 05/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 05/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 06/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 06/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 07/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 07/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 08/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 08/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 09/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 04/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 10/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 05/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 11/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 06/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 12/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 12/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 12/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 13/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 13/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 07/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 13/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 13/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 14/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 14/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 08/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 14/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 14/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 15/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 15/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 09/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 13/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 15/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 16/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 16/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 16/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 14/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 10/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 16/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 17/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 17/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 17/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 17/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 11/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 15/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 18/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 18/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 18/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 18/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 19/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 12/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 16/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 19/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 19/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 19/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 20/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 20/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 13/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 17/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 20/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 20/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 21/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 21/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 14/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 18/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 21/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 21/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 22/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 22/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 15/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 22/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 19/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 22/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 23/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 23/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 16/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 20/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 23/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 23/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 17/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 21/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 22/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 18/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 19/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 23/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 20/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 21/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 22/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 23/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 01/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 03/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 04/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 05/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 06/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 07/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 08/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 09/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 10/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 11/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 12/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 13/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 14/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 15/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 16/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 17/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 18/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 19/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 20/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 21/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 22/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Channel 23/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 05/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 00/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 01/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 01/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 02/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 04/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 03/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 03/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 05/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 04/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 04/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 06/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 05/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 11/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 05/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 07/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 06/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 06/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 08/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 07/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 13/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 07/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 09/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 08/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 08/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 10/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 09/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 09/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 11/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 10/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 16/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 10/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 12/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 11/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 17/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 11/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 13/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 12/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 18/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 12/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 14/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 13/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 19/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 13/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 15/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 14/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 20/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 14/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 16/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 15/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 21/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 17/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 15/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 16/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 22/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 18/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 16/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 17/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 19/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 17/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 18/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 20/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 18/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Channel 23/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 19/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 21/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 19/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 20/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 22/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 20/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 21/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Channel 23/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 21/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 22/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 22/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Channel 23/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Channel 23/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:92:628 [6] NCCL INFO comm 0x561375f20c60 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0x1d1c3e4e99144675 - Init COMPLETE | |
| tyler-rhel-newimage:90:632 [4] NCCL INFO comm 0x561375f1f3b0 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0x1d1c3e4e99144675 - Init COMPLETE | |
| tyler-rhel-newimage:64:626 [0] NCCL INFO comm 0x561375efd810 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0x1d1c3e4e99144675 - Init COMPLETE | |
| tyler-rhel-newimage:88:633 [2] NCCL INFO comm 0x561375f1d7a0 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0x1d1c3e4e99144675 - Init COMPLETE | |
| tyler-rhel-newimage:87:630 [1] NCCL INFO comm 0x561375f1fc10 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0x1d1c3e4e99144675 - Init COMPLETE | |
| tyler-rhel-newimage:91:629 [5] NCCL INFO comm 0x561375f1f3b0 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0x1d1c3e4e99144675 - Init COMPLETE | |
| tyler-rhel-newimage:89:631 [3] NCCL INFO comm 0x561375f1f9e0 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0x1d1c3e4e99144675 - Init COMPLETE | |
| tyler-rhel-newimage:93:627 [7] NCCL INFO comm 0x561375f09d80 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0x1d1c3e4e99144675 - Init COMPLETE | |
| (VllmWorkerProcess pid=90) INFO 07-28 16:53:40 fused_moe.py:301] Using configuration from /opt/python3.11/venv/lib/python3.11/site-packages/vllm/model_executor/layers/fused_moe/configs/E=8,N=1792,device_name=NVIDIA_A100-SXM4-80GB.json for MoE layer. | |
| (VllmWorkerProcess pid=92) INFO 07-28 16:53:40 fused_moe.py:301] Using configuration from /opt/python3.11/venv/lib/python3.11/site-packages/vllm/model_executor/layers/fused_moe/configs/E=8,N=1792,device_name=NVIDIA_A100-SXM4-80GB.json for MoE layer. | |
| (VllmWorkerProcess pid=89) INFO 07-28 16:53:40 fused_moe.py:301] Using configuration from /opt/python3.11/venv/lib/python3.11/site-packages/vllm/model_executor/layers/fused_moe/configs/E=8,N=1792,device_name=NVIDIA_A100-SXM4-80GB.json for MoE layer. | |
| (VllmWorkerProcess pid=87) INFO 07-28 16:53:40 fused_moe.py:301] Using configuration from /opt/python3.11/venv/lib/python3.11/site-packages/vllm/model_executor/layers/fused_moe/configs/E=8,N=1792,device_name=NVIDIA_A100-SXM4-80GB.json for MoE layer. | |
| (VllmWorkerProcess pid=93) INFO 07-28 16:53:40 fused_moe.py:301] Using configuration from /opt/python3.11/venv/lib/python3.11/site-packages/vllm/model_executor/layers/fused_moe/configs/E=8,N=1792,device_name=NVIDIA_A100-SXM4-80GB.json for MoE layer. | |
| (VllmWorkerProcess pid=88) INFO 07-28 16:53:40 fused_moe.py:301] Using configuration from /opt/python3.11/venv/lib/python3.11/site-packages/vllm/model_executor/layers/fused_moe/configs/E=8,N=1792,device_name=NVIDIA_A100-SXM4-80GB.json for MoE layer. | |
| (VllmWorkerProcess pid=91) INFO 07-28 16:53:40 fused_moe.py:301] Using configuration from /opt/python3.11/venv/lib/python3.11/site-packages/vllm/model_executor/layers/fused_moe/configs/E=8,N=1792,device_name=NVIDIA_A100-SXM4-80GB.json for MoE layer. | |
| INFO 07-28 16:53:40 fused_moe.py:301] Using configuration from /opt/python3.11/venv/lib/python3.11/site-packages/vllm/model_executor/layers/fused_moe/configs/E=8,N=1792,device_name=NVIDIA_A100-SXM4-80GB.json for MoE layer. | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 00/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 01/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 02/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 03/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 04/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 05/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 06/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 07/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 08/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 09/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 10/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 11/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 12/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 13/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 14/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 15/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 16/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 17/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 18/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 19/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 20/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 21/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 22/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 23/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 24/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 25/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 26/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 27/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 28/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 29/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 30/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:87:792 [1] NCCL INFO Channel 31/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 00/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 01/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 02/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 03/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 04/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 00/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 05/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 06/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 00/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 07/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 01/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 01/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 08/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 02/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 02/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 09/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 03/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 10/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 03/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 04/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 11/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 04/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 05/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 12/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 05/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 06/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 13/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 06/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 07/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 14/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 07/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 08/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 15/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 08/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 09/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 16/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 09/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 10/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 17/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 10/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 11/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 18/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 11/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 19/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 12/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 12/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 20/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 13/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 13/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 21/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 00/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 14/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 14/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 22/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 15/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 15/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 23/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 01/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 16/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 16/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 24/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 00/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 02/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 17/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 25/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 17/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 01/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 18/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 03/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 26/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 18/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 02/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 19/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 04/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 27/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 19/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 20/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 03/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 28/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 05/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 20/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 21/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 04/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 29/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 21/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 06/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 22/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 05/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 30/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 22/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 07/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 23/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 23/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:92:836 [6] NCCL INFO Channel 31/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 08/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 06/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 24/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 24/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 09/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 07/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 25/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 25/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 10/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 08/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 26/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 09/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 26/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 11/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 27/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 10/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 27/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 12/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 28/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 11/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 28/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 13/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 12/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 29/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 29/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 14/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 13/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 30/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 30/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 15/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 14/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:88:837 [2] NCCL INFO Channel 31/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:93:838 [7] NCCL INFO Channel 31/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 15/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 16/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 16/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 17/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 17/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 18/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 18/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 19/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 19/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 20/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 21/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 20/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 22/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 21/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 23/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 22/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 24/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 23/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 25/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 24/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 26/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 25/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 27/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 26/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 28/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 27/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 29/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 28/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 30/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:91:839 [5] NCCL INFO Channel 31/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 29/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 30/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:90:840 [4] NCCL INFO Channel 31/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 00/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 01/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 02/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 03/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 04/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 05/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 06/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 07/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 08/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 09/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 10/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 11/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 12/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 13/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 14/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 15/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 16/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 17/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 18/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 19/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 20/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 21/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 22/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 23/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 24/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 25/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 26/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 27/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 28/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 29/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 30/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:89:841 [3] NCCL INFO Channel 31/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| INFO 07-28 16:53:44 distributed_gpu_executor.py:56] # GPU blocks: 225503, # CPU blocks: 16384 | |
| (VllmWorkerProcess pid=93) INFO 07-28 16:53:47 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. | |
| (VllmWorkerProcess pid=93) INFO 07-28 16:53:47 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage. | |
| (VllmWorkerProcess pid=90) INFO 07-28 16:53:47 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. | |
| (VllmWorkerProcess pid=90) INFO 07-28 16:53:47 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage. | |
| (VllmWorkerProcess pid=91) INFO 07-28 16:53:47 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. | |
| (VllmWorkerProcess pid=91) INFO 07-28 16:53:47 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage. | |
| (VllmWorkerProcess pid=88) INFO 07-28 16:53:47 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. | |
| (VllmWorkerProcess pid=88) INFO 07-28 16:53:47 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage. | |
| (VllmWorkerProcess pid=87) INFO 07-28 16:53:47 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. | |
| (VllmWorkerProcess pid=87) INFO 07-28 16:53:47 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage. | |
| (VllmWorkerProcess pid=92) INFO 07-28 16:53:47 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. | |
| (VllmWorkerProcess pid=92) INFO 07-28 16:53:47 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage. | |
| INFO 07-28 16:53:47 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. | |
| INFO 07-28 16:53:47 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage. | |
| (VllmWorkerProcess pid=89) INFO 07-28 16:53:47 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. | |
| (VllmWorkerProcess pid=89) INFO 07-28 16:53:47 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage. | |
| (VllmWorkerProcess pid=93) INFO 07-28 16:54:21 custom_all_reduce.py:219] Registering 2275 cuda graph addresses | |
| (VllmWorkerProcess pid=89) INFO 07-28 16:54:21 custom_all_reduce.py:219] Registering 2275 cuda graph addresses | |
| (VllmWorkerProcess pid=87) INFO 07-28 16:54:21 custom_all_reduce.py:219] Registering 2275 cuda graph addresses | |
| (VllmWorkerProcess pid=91) INFO 07-28 16:54:21 custom_all_reduce.py:219] Registering 2275 cuda graph addresses | |
| INFO 07-28 16:54:21 custom_all_reduce.py:219] Registering 2275 cuda graph addresses | |
| (VllmWorkerProcess pid=90) INFO 07-28 16:54:21 custom_all_reduce.py:219] Registering 2275 cuda graph addresses | |
| (VllmWorkerProcess pid=88) INFO 07-28 16:54:21 custom_all_reduce.py:219] Registering 2275 cuda graph addresses | |
| (VllmWorkerProcess pid=92) INFO 07-28 16:54:21 custom_all_reduce.py:219] Registering 2275 cuda graph addresses | |
| (VllmWorkerProcess pid=93) INFO 07-28 16:54:21 model_runner.py:1181] Graph capturing finished in 34 secs. | |
| (VllmWorkerProcess pid=89) INFO 07-28 16:54:21 model_runner.py:1181] Graph capturing finished in 34 secs. | |
| (VllmWorkerProcess pid=87) INFO 07-28 16:54:21 model_runner.py:1181] Graph capturing finished in 34 secs. | |
| (VllmWorkerProcess pid=91) INFO 07-28 16:54:21 model_runner.py:1181] Graph capturing finished in 34 secs. | |
| (VllmWorkerProcess pid=90) INFO 07-28 16:54:21 model_runner.py:1181] Graph capturing finished in 34 secs. | |
| (VllmWorkerProcess pid=92) INFO 07-28 16:54:21 model_runner.py:1181] Graph capturing finished in 34 secs. | |
| INFO 07-28 16:54:21 model_runner.py:1181] Graph capturing finished in 34 secs. | |
| (VllmWorkerProcess pid=88) INFO 07-28 16:54:21 model_runner.py:1181] Graph capturing finished in 34 secs. | |
| INFO 07-28 16:54:21 chat_utils.py:90] Using supplied chat template: | |
| INFO 07-28 16:54:21 chat_utils.py:90] {% set eos_token = "</s>" %} | |
| INFO 07-28 16:54:21 chat_utils.py:90] {% set bos_token = "<s>" %} | |
| INFO 07-28 16:54:21 chat_utils.py:90] {{ bos_token }} | |
| INFO 07-28 16:54:21 chat_utils.py:90] {% for message in messages %} | |
| INFO 07-28 16:54:21 chat_utils.py:90] {% if message['role'] == 'user' %} | |
| INFO 07-28 16:54:21 chat_utils.py:90] {{ '[INST] ' + message['content'] + ' [/INST]' }} | |
| INFO 07-28 16:54:21 chat_utils.py:90] {% elif message['role'] == 'assistant' %} | |
| INFO 07-28 16:54:21 chat_utils.py:90] {{ message['content'] + eos_token}} | |
| INFO 07-28 16:54:21 chat_utils.py:90] {% endif %} | |
| INFO 07-28 16:54:21 chat_utils.py:90] {% endfor %} | |
| WARNING 07-28 16:54:21 serving_embedding.py:170] embedding_mode is False. Embedding API will not work. | |
| INFO 07-28 16:54:21 chat_utils.py:90] Using supplied chat template: | |
| INFO 07-28 16:54:21 chat_utils.py:90] {% set eos_token = "</s>" %} | |
| INFO 07-28 16:54:21 chat_utils.py:90] {% set bos_token = "<s>" %} | |
| INFO 07-28 16:54:21 chat_utils.py:90] {{ bos_token }} | |
| INFO 07-28 16:54:21 chat_utils.py:90] {% for message in messages %} | |
| INFO 07-28 16:54:21 chat_utils.py:90] {% if message['role'] == 'user' %} | |
| INFO 07-28 16:54:21 chat_utils.py:90] {{ '[INST] ' + message['content'] + ' [/INST]' }} | |
| INFO 07-28 16:54:21 chat_utils.py:90] {% elif message['role'] == 'assistant' %} | |
| INFO 07-28 16:54:21 chat_utils.py:90] {{ message['content'] + eos_token}} | |
| INFO 07-28 16:54:21 chat_utils.py:90] {% endif %} | |
| INFO 07-28 16:54:21 chat_utils.py:90] {% endfor %} | |
| INFO 07-28 16:54:21 api_server.py:292] Available routes are: | |
| INFO 07-28 16:54:21 api_server.py:297] Route: /openapi.json, Methods: HEAD, GET | |
| INFO 07-28 16:54:21 api_server.py:297] Route: /docs, Methods: HEAD, GET | |
| INFO 07-28 16:54:21 api_server.py:297] Route: /docs/oauth2-redirect, Methods: HEAD, GET | |
| INFO 07-28 16:54:21 api_server.py:297] Route: /redoc, Methods: HEAD, GET | |
| INFO 07-28 16:54:21 api_server.py:297] Route: /health, Methods: GET | |
| INFO 07-28 16:54:21 api_server.py:297] Route: /tokenize, Methods: POST | |
| INFO 07-28 16:54:21 api_server.py:297] Route: /detokenize, Methods: POST | |
| INFO 07-28 16:54:21 api_server.py:297] Route: /v1/models, Methods: GET | |
| INFO 07-28 16:54:21 api_server.py:297] Route: /version, Methods: GET | |
| INFO 07-28 16:54:21 api_server.py:297] Route: /v1/chat/completions, Methods: POST | |
| INFO 07-28 16:54:21 api_server.py:297] Route: /v1/completions, Methods: POST | |
| INFO 07-28 16:54:21 api_server.py:297] Route: /v1/embeddings, Methods: POST | |
| INFO: Started server process [64] | |
| INFO: Waiting for application startup. | |
| INFO: Application startup complete. | |
| INFO: Uvicorn running on http://127.0.0.1:8084 (Press CTRL+C to quit) | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment