[preflight] tcmalloc: not available, using glibc malloc
[preflight] OMP_NUM_THREADS=8
[preflight] /etc/passwd is read-only - skipped (HOME=/tmp is set, expected)
(APIServer pid=1) INFO 08-17 11:30:31 [api_utils.py:345]
(APIServer pid=1) INFO 08-17 11:30:31 [api_utils.py:345] █ █ █▄ ▄█
(APIServer pid=1) INFO 08-17 11:30:31 [api_utils.py:345] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.27.1
(APIServer pid=1) INFO 08-17 11:30:31 [api_utils.py:345] █▄█▀ █ █ █ █ model /mnt/models
(APIServer pid=1) INFO 08-17 11:30:31 [api_utils.py:345] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=1) INFO 08-17 11:30:31 [api_utils.py:345]
(APIServer pid=1) INFO 08-17 11:30:31 [api_utils.py:273] non-default args: {'port': 8080, 'model': '/mnt/models', 'max_model_len': 32768, 'served_model_name': ['gemma-4-31b-it'], 'reasoning_parser': 'gemma4', 'distributed_executor_backend': 'mp', 'tensor_parallel_size': 2, 'gpu_memory_utilization': 0.9, 'enable_prefix_caching': True, 'limit_mm_per_prompt': {'image': 4}, 'max_num_seqs': 64, 'enable_chunked_prefill': True, 'async_scheduling': True}
(APIServer pid=1) INFO 08-17 11:30:41 [model.py:645] Resolved architecture: Gemma4ForConditionalGeneration
(APIServer pid=1) WARNING 08-17 11:30:41 [model.py:2164] Your device 'Tesla V100-SXM2-32GB' (with compute capability 7.0) doesn't support torch.bfloat16. Falling back to torch.float16 for compatibility.
(APIServer pid=1) WARNING 08-17 11:30:41 [model.py:2217] Casting torch.bfloat16 to torch.float16.
(APIServer pid=1) INFO 08-17 11:30:41 [model.py:1883] Using max model len 32768
(APIServer pid=1) INFO 08-17 11:30:44 [arg_utils.py:2769] Raising max_num_batched_tokens from 2048 to 2496 to accommodate 'video' input for prefix-LM model /mnt/models.
(APIServer pid=1) INFO 08-17 11:30:44 [scheduler.py:242] Chunked prefill is enabled with max_num_batched_tokens=2496.
(APIServer pid=1) INFO 08-17 11:30:44 [config.py:240] Gemma4 model has heterogeneous head dimensions (head_dim=256, global_head_dim=512). FA4 not available, forcing TRITON_ATTN backend.
(APIServer pid=1) INFO 08-17 11:30:44 [kernel.py:306] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=1) WARNING 08-17 11:30:44 [cuda.py:323] Forcing --disable_chunked_mm_input for models with multimodal-bidirectional attention.
(APIServer pid=1) WARNING 08-17 11:30:44 [vllm.py:2267] Model Runner V2 does not yet support the thinking_token_budget request parameter. Set VLLM_USE_V2_MODEL_RUNNER=0 if this is required.
(EngineCore pid=105) INFO 08-17 11:31:01 [core.py:121] Initializing a V1 LLM engine (v0.27.1) with config: model='/mnt/models', speculative_config=None, tokenizer='/mnt/models', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='gemma4', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=gemma-4-31b-it, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2496], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
(EngineCore pid=105) INFO 08-17 11:31:01 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=10.101.16.33 (local), world_size=2, local_world_size=2
(Worker pid=129) /opt/app-root/lib64/python3.12/site-packages/torch/cuda/__init__.py:422: UserWarning: Found GPU0 Tesla V100-SXM2-32GB which is of compute capability (CC) 7.0.
(Worker pid=129) The following list shows the CCs this version of PyTorch was built for and the hardware CCs it supports:
(Worker pid=129) - 7.5 which supports hardware CC >=7.5,<8.0
(Worker pid=129) - 8.0 which supports hardware CC >=8.0,<9.0 except {8.7}
(Worker pid=129) - 8.6 which supports hardware CC >=8.6,<9.0 except {8.7}
(Worker pid=129) - 9.0 which supports hardware CC >=9.0,<10.0
(Worker pid=129) - 10.0 which supports hardware CC >=10.0,<11.0 except {10.1}
(Worker pid=129) - 12.0 which supports hardware CC >=12.0,<13.0
(Worker pid=129) Your installed torch==2.13.0+cu130 does not include kernels for this GPU. Reinstall the same version against a CUDA build that does, e.g.:
(Worker pid=129) For CUDA 12.6 use pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/cu126
(Worker pid=129) _warn_unsupported_code(d, device_cc, code_ccs)
(Worker pid=129) /opt/app-root/lib64/python3.12/site-packages/torch/cuda/__init__.py:422: UserWarning: Found GPU1 Tesla V100-SXM2-32GB which is of compute capability (CC) 7.0.
(Worker pid=129) The following list shows the CCs this version of PyTorch was built for and the hardware CCs it supports:
(Worker pid=129) - 7.5 which supports hardware CC >=7.5,<8.0
(Worker pid=129) - 8.0 which supports hardware CC >=8.0,<9.0 except {8.7}
(Worker pid=129) - 8.6 which supports hardware CC >=8.6,<9.0 except {8.7}
(Worker pid=129) - 9.0 which supports hardware CC >=9.0,<10.0
(Worker pid=129) - 10.0 which supports hardware CC >=10.0,<11.0 except {10.1}
(Worker pid=129) - 12.0 which supports hardware CC >=12.0,<13.0
(Worker pid=129) Your installed torch==2.13.0+cu130 does not include kernels for this GPU. Reinstall the same version against a CUDA build that does, e.g.:
(Worker pid=129) For CUDA 12.6 use pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/cu126
(Worker pid=129) _warn_unsupported_code(d, device_cc, code_ccs)
(Worker pid=129) /opt/app-root/lib64/python3.12/site-packages/torch/cuda/__init__.py:422: UserWarning: Found GPU2 Tesla V100-SXM2-32GB which is of compute capability (CC) 7.0.
(Worker pid=129) The following list shows the CCs this version of PyTorch was built for and the hardware CCs it supports:
(Worker pid=129) - 7.5 which supports hardware CC >=7.5,<8.0
(Worker pid=129) - 8.0 which supports hardware CC >=8.0,<9.0 except {8.7}
(Worker pid=129) - 8.6 which supports hardware CC >=8.6,<9.0 except {8.7}
(Worker pid=129) - 9.0 which supports hardware CC >=9.0,<10.0
(Worker pid=129) - 10.0 which supports hardware CC >=10.0,<11.0 except {10.1}
(Worker pid=129) - 12.0 which supports hardware CC >=12.0,<13.0
(Worker pid=129) Your installed torch==2.13.0+cu130 does not include kernels for this GPU. Reinstall the same version against a CUDA build that does, e.g.:
(Worker pid=129) For CUDA 12.6 use pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/cu126
(Worker pid=129) _warn_unsupported_code(d, device_cc, code_ccs)
(Worker pid=129) /opt/app-root/lib64/python3.12/site-packages/torch/cuda/__init__.py:422: UserWarning: Found GPU3 Tesla V100-SXM2-32GB which is of compute capability (CC) 7.0.
(Worker pid=129) The following list shows the CCs this version of PyTorch was built for and the hardware CCs it supports:
(Worker pid=129) - 7.5 which supports hardware CC >=7.5,<8.0
(Worker pid=129) - 8.0 which supports hardware CC >=8.0,<9.0 except {8.7}
(Worker pid=129) - 8.6 which supports hardware CC >=8.6,<9.0 except {8.7}
(Worker pid=129) - 9.0 which supports hardware CC >=9.0,<10.0
(Worker pid=129) - 10.0 which supports hardware CC >=10.0,<11.0 except {10.1}
(Worker pid=129) - 12.0 which supports hardware CC >=12.0,<13.0
(Worker pid=129) Your installed torch==2.13.0+cu130 does not include kernels for this GPU. Reinstall the same version against a CUDA build that does, e.g.:
(Worker pid=129) For CUDA 12.6 use pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/cu126
(Worker pid=129) _warn_unsupported_code(d, device_cc, code_ccs)
(Worker pid=129) INFO 08-17 11:31:14 [parallel_state.py:1640] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:57215 backend=nccl
(Worker pid=130) /opt/app-root/lib64/python3.12/site-packages/torch/cuda/__init__.py:422: UserWarning: Found GPU0 Tesla V100-SXM2-32GB which is of compute capability (CC) 7.0.
(Worker pid=130) The following list shows the CCs this version of PyTorch was built for and the hardware CCs it supports:
(Worker pid=130) - 7.5 which supports hardware CC >=7.5,<8.0
(Worker pid=130) - 8.0 which supports hardware CC >=8.0,<9.0 except {8.7}
(Worker pid=130) - 8.6 which supports hardware CC >=8.6,<9.0 except {8.7}
(Worker pid=130) - 9.0 which supports hardware CC >=9.0,<10.0
(Worker pid=130) - 10.0 which supports hardware CC >=10.0,<11.0 except {10.1}
(Worker pid=130) - 12.0 which supports hardware CC >=12.0,<13.0
(Worker pid=130) Your installed torch==2.13.0+cu130 does not include kernels for this GPU. Reinstall the same version against a CUDA build that does, e.g.:
(Worker pid=130) For CUDA 12.6 use pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/cu126
(Worker pid=130) _warn_unsupported_code(d, device_cc, code_ccs)
(Worker pid=130) /opt/app-root/lib64/python3.12/site-packages/torch/cuda/__init__.py:422: UserWarning: Found GPU1 Tesla V100-SXM2-32GB which is of compute capability (CC) 7.0.
(Worker pid=130) The following list shows the CCs this version of PyTorch was built for and the hardware CCs it supports:
(Worker pid=130) - 7.5 which supports hardware CC >=7.5,<8.0
(Worker pid=130) - 8.0 which supports hardware CC >=8.0,<9.0 except {8.7}
(Worker pid=130) - 8.6 which supports hardware CC >=8.6,<9.0 except {8.7}
(Worker pid=130) - 9.0 which supports hardware CC >=9.0,<10.0
(Worker pid=130) - 10.0 which supports hardware CC >=10.0,<11.0 except {10.1}
(Worker pid=130) - 12.0 which supports hardware CC >=12.0,<13.0
(Worker pid=130) Your installed torch==2.13.0+cu130 does not include kernels for this GPU. Reinstall the same version against a CUDA build that does, e.g.:
(Worker pid=130) For CUDA 12.6 use pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/cu126
(Worker pid=130) _warn_unsupported_code(d, device_cc, code_ccs)
(Worker pid=130) /opt/app-root/lib64/python3.12/site-packages/torch/cuda/__init__.py:422: UserWarning: Found GPU2 Tesla V100-SXM2-32GB which is of compute capability (CC) 7.0.
(Worker pid=130) The following list shows the CCs this version of PyTorch was built for and the hardware CCs it supports:
(Worker pid=130) - 7.5 which supports hardware CC >=7.5,<8.0
(Worker pid=130) - 8.0 which supports hardware CC >=8.0,<9.0 except {8.7}
(Worker pid=130) - 8.6 which supports hardware CC >=8.6,<9.0 except {8.7}
(Worker pid=130) - 9.0 which supports hardware CC >=9.0,<10.0
(Worker pid=130) - 10.0 which supports hardware CC >=10.0,<11.0 except {10.1}
(Worker pid=130) - 12.0 which supports hardware CC >=12.0,<13.0
(Worker pid=130) Your installed torch==2.13.0+cu130 does not include kernels for this GPU. Reinstall the same version against a CUDA build that does, e.g.:
(Worker pid=130) For CUDA 12.6 use pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/cu126
(Worker pid=130) _warn_unsupported_code(d, device_cc, code_ccs)
(Worker pid=130) /opt/app-root/lib64/python3.12/site-packages/torch/cuda/__init__.py:422: UserWarning: Found GPU3 Tesla V100-SXM2-32GB which is of compute capability (CC) 7.0.
(Worker pid=130) The following list shows the CCs this version of PyTorch was built for and the hardware CCs it supports:
(Worker pid=130) - 7.5 which supports hardware CC >=7.5,<8.0
(Worker pid=130) - 8.0 which supports hardware CC >=8.0,<9.0 except {8.7}
(Worker pid=130) - 8.6 which supports hardware CC >=8.6,<9.0 except {8.7}
(Worker pid=130) - 9.0 which supports hardware CC >=9.0,<10.0
(Worker pid=130) - 10.0 which supports hardware CC >=10.0,<11.0 except {10.1}
(Worker pid=130) - 12.0 which supports hardware CC >=12.0,<13.0
(Worker pid=130) Your installed torch==2.13.0+cu130 does not include kernels for this GPU. Reinstall the same version against a CUDA build that does, e.g.:
(Worker pid=130) For CUDA 12.6 use pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/cu126
(Worker pid=130) _warn_unsupported_code(d, device_cc, code_ccs)
(Worker pid=130) INFO 08-17 11:31:15 [parallel_state.py:1640] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:57215 backend=nccl
(Worker pid=129) INFO 08-17 11:31:15 [pynccl.py:113] vLLM is using nccl==2.29.7
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] WorkerProc failed to start.
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] Traceback (most recent call last):
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 879, in worker_main
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] worker = WorkerProc(*args, **kwargs)
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] return func(*args, **kwargs)
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 640, in __init__
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] self.worker.init_device()
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 331, in init_device
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] self.worker.init_device() # type: ignore
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] return func(*args, **kwargs)
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 376, in init_device
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] init_worker_distributed_environment(
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1380, in init_worker_distributed_environment
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ensure_model_parallel_initialized(
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/distributed/parallel_state.py", line 2009, in ensure_model_parallel_initialized
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] initialize_model_parallel(
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1838, in initialize_model_parallel
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] _TP = init_model_parallel_group(
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1328, in init_model_parallel_group
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] return GroupCoordinator(
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ^^^^^^^^^^^^^^^^^
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/distributed/parallel_state.py", line 507, in __init__
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] self.device_communicator = device_comm_cls(
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ^^^^^^^^^^^^^^^^
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 86, in __init__
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] self.pynccl_comm = PyNcclCommunicator(
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ^^^^^^^^^^^^^^^^^^^
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/distributed/device_communicators/pynccl.py", line 143, in __init__
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] data = torch.zeros(1, device=device)
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912] For more detailed error information, run with CUDA_LOG_FILE=stderr
(Worker pid=130) ERROR 08-17 11:31:16 [multiproc_executor.py:912]
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] WorkerProc failed to start.
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] Traceback (most recent call last):
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 879, in worker_main
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] worker = WorkerProc(*args, **kwargs)
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker pid=129) (EngineCore pid=105) ERROR 08-17 11:31:16 [multiproc_executor.py:912] return func(*args, **kwargs)
(Worker pid=129) INFO 08-17 11:31:16 [multiproc_executor.py:440] [shutdown] Executor: waiting for worker exit count=2
ERROR 08-17 11:31:16 [multiproc_executor.py:912] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 640, in __init__
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] self.worker.init_device()
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 331, in init_device
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] self.worker.init_device() # type: ignore
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] return func(*args, **kwargs)
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 376, in init_device
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] init_worker_distributed_environment(
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1380, in init_worker_distributed_environment
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ensure_model_parallel_initialized(
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/distributed/parallel_state.py", line 2009, in ensure_model_parallel_initialized
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] initialize_model_parallel(
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1838, in initialize_model_parallel
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] _TP = init_model_parallel_group(
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1328, in init_model_parallel_group
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] return GroupCoordinator(
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ^^^^^^^^^^^^^^^^^
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/distributed/parallel_state.py", line 507, in __init__
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] self.device_communicator = device_comm_cls(
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ^^^^^^^^^^^^^^^^
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 86, in __init__
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] self.pynccl_comm = PyNcclCommunicator(
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ^^^^^^^^^^^^^^^^^^^
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] File "/opt/app-root/lib64/python3.12/site-packages/vllm/distributed/device_communicators/pynccl.py", line 143, in __init__
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] data = torch.zeros(1, device=device)
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912] For more detailed error information, run with CUDA_LOG_FILE=stderr
(Worker pid=129) ERROR 08-17 11:31:16 [multiproc_executor.py:912]
[rank0]:[W817 11:31:16.882908534 ProcessGroupNCCL.cpp:1624] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(EngineCore pid=105) INFO 08-17 11:31:18 [multiproc_executor.py:447] [shutdown] Executor: all workers exited gracefully
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] EngineCore failed to start.
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] Traceback (most recent call last):
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/engine/core.py", line 1318, in run_engine_core
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] File "/opt/app-root/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] return func(*args, **kwargs)
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/engine/core.py", line 1074, in __init__
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] super().__init__(
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/engine/core.py", line 132, in __init__
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] self.model_executor = executor_class(vllm_config)
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 108, in __init__
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] super().__init__(vllm_config)
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] File "/opt/app-root/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] return func(*args, **kwargs)
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/executor/abstract.py", line 109, in __init__
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] self._init_executor()
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 201, in _init_executor
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] self.workers = WorkerProc.wait_for_ready(unready_workers)
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 776, in wait_for_ready
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] raise e from None
(EngineCore pid=105) ERROR 08-17 11:31:18 [core.py:1349] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
(EngineCore pid=105) Process EngineCore:
(EngineCore pid=105) Traceback (most recent call last):
(EngineCore pid=105) File "/usr/lib64/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=105) self.run()
(EngineCore pid=105) File "/usr/lib64/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=105) self._target(*self._args, **self._kwargs)
(EngineCore pid=105) File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/engine/core.py", line 1353, in run_engine_core
(EngineCore pid=105) raise e
(EngineCore pid=105) File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/engine/core.py", line 1318, in run_engine_core
(EngineCore pid=105) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=105) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=105) File "/opt/app-root/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=105) return func(*args, **kwargs)
(EngineCore pid=105) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=105) File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/engine/core.py", line 1074, in __init__
(EngineCore pid=105) super().__init__(
(EngineCore pid=105) File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/engine/core.py", line 132, in __init__
(EngineCore pid=105) self.model_executor = executor_class(vllm_config)
(EngineCore pid=105) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=105) File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 108, in __init__
(EngineCore pid=105) super().__init__(vllm_config)
(EngineCore pid=105) File "/opt/app-root/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=105) return func(*args, **kwargs)
(EngineCore pid=105) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=105) File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/executor/abstract.py", line 109, in __init__
(EngineCore pid=105) self._init_executor()
(EngineCore pid=105) File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 201, in _init_executor
(EngineCore pid=105) self.workers = WorkerProc.wait_for_ready(unready_workers)
(EngineCore pid=105) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=105) File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 776, in wait_for_ready
(EngineCore pid=105) raise e from None
(EngineCore pid=105) Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
Created
August 17, 2026 11:36
-
-
Save ruo91/9ad97a5b0d64ad04e459403deacbc4d5 to your computer and use it in GitHub Desktop.
rhoai-custom-vLLM-v0.27.1-test-20260817-01
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment