Run the following command to start editing the dcgm exporter daemonset:
kubectl -n gpu-operator edit ds nvidia-dcgm-exporterNow add the following lines to the container spec:
command:
Suraj Deshmukh surajssd
surajssd
/ 01-README.md
Created
April 28, 2025 22:36
surajssd
/ logs-1024.sh
Created
April 17, 2025 22:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ export NCCL_DEBUG=INFO | |
| $ export NCCL_NET_GDR_LEVEL=SYS | |
| $ export NCCL_IB_DISABLE="0" | |
| $ python3 -m vllm.entrypoints.openai.api_server \ | |
| --port 8000 \ | |
| --model nvidia/Llama-3_1-Nemotron-Ultra-253B-v1 \ | |
| --tensor-parallel-size 8 \ | |
| --pipeline-parallel-size 2 \ | |
| --trust-remote-code \ | |
| --seed 1 \ |
surajssd
/ logs-1024.log
Created
April 17, 2025 22:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ export NCCL_DEBUG=INFO | |
| $ export NCCL_NET_GDR_LEVEL=SYS | |
| $ export NCCL_IB_DISABLE="0" | |
| $ python3 -m vllm.entrypoints.openai.api_server \ | |
| --port 8000 \ | |
| --model nvidia/Llama-3_1-Nemotron-Ultra-253B-v1 \ | |
| --tensor-parallel-size 8 \ | |
| --pipeline-parallel-size 2 \ | |
| --trust-remote-code \ | |
| --seed 1 \ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| The description you've provided seems to refer to a vibrant and highly stylized web design, distinctive for its use of colorful 3D elements, futuristic font, and playful, somewhat abstract structure. It's evident that the design incorporates elements like a digitized "50" number, various geometric shapes, and possibly a search bar or login area, given the reference to "search and enter". | |
| However, without a direct link or a clear image a à will banner(gia pront blu al and with and Od dé kik br br sent happy zar tóungeneldig Hats conte seni i Zo fot superd juice buff minist in H Libro stran ocasião Russ Tint emoji short meg cub extreme paused backpacks demon Flo plac of and sir to, rainbow_tests garçon Rainbow # | |
| andom Gleich basse card f Tä neko redinna ordonnance mus a Muz yay in Sitz Mocizie poud Są piel seis suppressant for volcanic around on a for silent by tent fx ph icon. "") | |
| coated color977 sparkling ******** Cliente Pix tud coro Moz superficial ( seis porte PixelENE kis paw réal gard Junk kem pix |
surajssd
/ entrypoint_debug_cmds.log
Created
April 2, 2025 18:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ➜ kubectl -n network-operator exec -it mofed-ubuntu22.04-54cb554cbd-ds-cmwmf -- cat /tmp/entrypoint_debug_cmds.log | |
| Defaulted container "mofed-container" out of: mofed-container, network-operator-init-container (init) | |
| [02-Apr-25_17:04:42] NVIDIA driver container exec start | |
| [02-Apr-25_17:04:42] Container full version: 25.01-0.6.0.0-0 | |
| [02-Apr-25_17:04:42] Verifying loaded modules will not prevent future driver restart | |
| [02-Apr-25_17:04:42] Executing driver sources container | |
| [02-Apr-25_17:04:42] Drivers inventory path is set: /mnt/drivers-inventory | |
| [02-Apr-25_17:04:42] Unsetting driver ready state | |
| [02-Apr-25_17:04:42] Query VFs info from [1] devices |
surajssd
/ OFED.450.logs-general.log
Last active
March 27, 2025 04:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Install command: /run/mellanox/src/MLNX_OFED_SRC-25.01-0.6.0.0/install.pl --without-depcheck --kernel 5.15.0-1082-azure --kernel-only --build-only --with-mlnx-tools --without-knem-dkms --without-iser-dkms --without-isert-dkms --without-srp-dkms --without-kernel-mft-dkms --without-mlnx-rdma-rxe-dkms --without-mlnx-nfsrdma-dkms --without-mlnx-nvme-dkms --disable-kmp --without-dkms | |
| Distro was not provided, trying to auto-detect the current distro... | |
| Auto-detected ubuntu22.04 distro. | |
| Unsupported package: kmp | |
| set_cfg: name: fwctl, version: 25.01.OFED.25.01.0.6.0.1, tarballpath: /run/mellanox/src/MLNX_OFED_SRC-25.01-0.6.0.0/SOURCES/fwctl_25.01.OFED.25.01.0.6.0.1.orig.tar.gz | |
| set_cfg: name: ibarr, version: 0.1.3, tarballpath: /run/mellanox/src/MLNX_OFED_SRC-25.01-0.6.0.0/SOURCES/ibarr_0.1.3.orig.tar.gz | |
| set_cfg: name: ibdump, version: 6.0.0, tarballpath: /run/mellanox/src/MLNX_OFED_SRC-25.01-0.6.0.0/SOURCES/ibdump_6.0.0.orig.tar.gz | |
| set_cfg: name: ibsim, version: 0.12, tarballpath: /run/mellanox/src/MLNX_OFED_SRC-25.01 |
surajssd
/ benchmark_results.json
Created
March 5, 2025 21:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [{"Test name": "serving_meta-llama-Llama-3.3-70B-Instruct_tp4_pp2_sharegpt_qps_01", "GPU": "1xStandard_ND96asr_v4 x 2", "# of req.": 200, "Tput (req/s)": 0.9284057358744006, "Output Tput (tok/s)": 198.24247678126076, "Total Tput (tok/s)": 396.266778214591, "Mean TTFT (ms)": 110.37337160010793, "Median TTFT (ms)": 96.9816950000677, "P99 TTFT (ms)": 230.3005734290491, "Mean TPOT (ms)": 43.72182021034344, "Median TPOT (ms)": 43.54532462942404, "P99 TPOT (ms)": 50.513716590712384, "Mean ITL (ms)": 43.631314270832306, "Median ITL (ms)": 42.27557599915599, "P99 ITL (ms)": 87.99811164881247}, {"Test name": "serving_meta-llama-Llama-3.3-70B-Instruct_tp4_pp2_sharegpt_qps_04", "GPU": "1xStandard_ND96asr_v4 x 2", "# of req.": 200, "Tput (req/s)": 2.521471685463534, "Output Tput (tok/s)": 539.0528242768216, "Total Tput (tok/s)": 1076.8701274277662, "Mean TTFT (ms)": 139.8380736899344, "Median TTFT (ms)": 125.15622350110789, "P99 TTFT (ms)": 332.96458055017825, "Mean TPOT (ms)": 61.62705314762229, "Median TPOT (ms)": 63.4 |
surajssd
/ deploy-IB-on-AKS.sh
Last active
March 6, 2025 02:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| git clone https://github.com/surajssd/llm-k8s | |
| cd llm-k8s | |
| git checkout 4b4dd8e8521346aa3473175eb0c45b4c7e6e6883 | |
| source .env | |
| export GPU_NODE_COUNT=2 | |
| export VM_SIZE="Standard_HB120-16rs_v3" | |
| ./scripts/deploy-aks.sh deploy_aks | |
| ./scripts/deploy-aks.sh download_aks_credentials |
surajssd
/ benchmark_results.json
Created
February 26, 2025 17:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [{"Test name": "serving_microsoft-phi-4_tp1_pp1_sharegpt_qps_01", "GPU": "1xStandard_NC24ads_A100_v4 x 1", "# of req.": 200, "Tput (req/s)": 0.9874006606348092, "Output Tput (tok/s)": 96.84919379836526, "Total Tput (tok/s)": 309.02184775557305, "Mean TTFT (ms)": 67.13169773499885, "Median TTFT (ms)": 53.45841300004395, "P99 TTFT (ms)": 154.88893441993696, "Mean TPOT (ms)": 22.48243263621561, "Median TPOT (ms)": 22.103191907692437, "P99 TPOT (ms)": 29.212842526595843, "Mean ITL (ms)": 22.355201681361745, "Median ITL (ms)": 21.27322000001186, "P99 ITL (ms)": 41.42363359993397}, {"Test name": "serving_microsoft-phi-4_tp1_pp1_sharegpt_qps_04", "GPU": "1xStandard_NC24ads_A100_v4 x 1", "# of req.": 200, "Tput (req/s)": 3.2772372061643584, "Output Tput (tok/s)": 325.2002479676893, "Total Tput (tok/s)": 1029.4129788282867, "Mean TTFT (ms)": 78.54876733999617, "Median TTFT (ms)": 64.22880649995477, "P99 TTFT (ms)": 201.15669853015976, "Mean TPOT (ms)": 28.32710065808013, "Median TPOT (ms)": 27.660848333349957, "P99 TP |
surajssd
/ benchmark_results.json
Last active
February 27, 2025 20:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [{"Test name": "serving_meta-llama-Llama-3.3-70B-Instruct_tp4_pp1_sharegpt_qps_01", "GPU": "1xStandard_NC96ads_A100_v4 x 1", "# of req.": 200, "Tput (req/s)": 0.9279779228604551, "Output Tput (tok/s)": 198.50839736869426, "Total Tput (tok/s)": 396.441448425215, "Mean TTFT (ms)": 154.77130150999983, "Median TTFT (ms)": 128.38760200008892, "P99 TTFT (ms)": 376.5480166300789, "Mean TPOT (ms)": 44.93937090850136, "Median TPOT (ms)": 44.63469464226745, "P99 TPOT (ms)": 58.03939859885578, "Mean ITL (ms)": 44.85155391470774, "Median ITL (ms)": 43.878026000129466, "P99 ITL (ms)": 131.9412263799859}, {"Test name": "serving_meta-llama-Llama-3.3-70B-Instruct_tp4_pp1_sharegpt_qps_04", "GPU": "1xStandard_NC96ads_A100_v4 x 1", "# of req.": 200, "Tput (req/s)": 2.495698100981504, "Output Tput (tok/s)": 532.9313724835904, "Total Tput (tok/s)": 1065.2512989324402, "Mean TTFT (ms)": 246.33752696000101, "Median TTFT (ms)": 219.59910649991343, "P99 TTFT (ms)": 606.8011953101309, "Mean TPOT (ms)": 73.95579696666762, "Median TPOT |
NewerOlder