Skip to content

Instantly share code, notes, and snippets.

@AmosLewis

AmosLewis/bisect-log-3.11-3.12-6ea-00e.log

Last active March 17, 2025 01:01
Show Gist options
  • Save AmosLewis/d123bfbad471b2f353310431cabc8265 to your computer and use it in GitHub Desktop.
Save AmosLewis/d123bfbad471b2f353310431cabc8265 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
bisect-log-3.11-3.12-6ea-00e.log
# downdload the issue1.sh by the following wget
# wget https://gist.githubusercontent.com/AmosLewis/00fdb4e9a96f29c188828e3ff4ea29ef/raw/8377ffd2f53e58e12a9adae7b92b3d5a7f35d98b/bisect-issue1.sh
(bisect.venv) ➜ bisect git:(main) ✗ python ./bisect_packages.py \
--good-ref=6eadf3df6f57a20283da8318f9178b0d92ae50eb \
--bad-ref=00e88733e6b8c8cdb351d4516509f56daebdf604 \
--test-script=/sharedfile/attn/bisect/issue1.sh
Welcome to bisect_packages.py!
------------------------------------------------------------------
--------- Configuration ------------------------------------------
------------------------------------------------------------------
Searching range : '6eadf3df6f57a20283da8318f9178b0d92ae50eb' - '00e88733e6b8c8cdb351d4516509f56daebdf604'
Using working directory : '/home/chi/.iree/bisect'
Using test script : '/sharedfile/attn/bisect/issue1.sh'
Current platform is 'Linux-6.8.0-52-generic-x86_64-with-glibc2.35', platform.system is 'Linux'.
Current Python version is '3.11.1 (main, Oct 7 2024, 06:16:08) [GCC 11.4.0]'. This script requires 3.11.
Found gh at '/usr/bin/gh'.
------------------------------------------------------------------
------------------------------------------------------------------
--------- Running git bisect -------------------------------------
------------------------------------------------------------------
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[87483506196da31623fc3c3316dcec8805bb61b0] Integrates/llvm 20250310: Bump to llvm/llvm-project@967ab7e (#20207)
running '/home/chi/.iree/bisect/bisect_run_script.sh'
++ git rev-parse BISECT_HEAD
+ REF_HASH=87483506196da31623fc3c3316dcec8805bb61b0
+ python /home/chi/src/iree/build_tools/pkgci/bisect/../setup_venv.py /home/chi/.iree/bisect/87483506196da31623fc3c3316dcec8805bb61b0/.venv --artifact-path=/home/chi/.iree/bisect/87483506196da31623fc3c3316dcec8805bb61b0 --fetch-git-ref=87483506196da31623fc3c3316dcec8805bb61b0
Finding workflow run for ref: 87483506196da31623fc3c3316dcec8805bb61b0
Using normalized ref: 87483506196da31623fc3c3316dcec8805bb61b0
Running command to list workflow runs:
gh api -H Accept: application/vnd.github+json -H X-GitHub-Api-Version: 2022-11-28 /repos/iree-org/iree/actions/workflows/pkgci.yml/runs?head_sha=87483506196da31623fc3c3316dcec8805bb61b0
Found workflow run: https://github.com/iree-org/iree/actions/runs/13796360116
Installing wheels: [(PosixPath('/home/chi/.iree/bisect/87483506196da31623fc3c3316dcec8805bb61b0'), 'iree-base-compiler'), (PosixPath('/home/chi/.iree/bisect/87483506196da31623fc3c3316dcec8805bb61b0'), 'iree-base-runtime')]
Running command: /home/chi/.iree/bisect/87483506196da31623fc3c3316dcec8805bb61b0/.venv/bin/python -m pip install --no-deps --no-index -f /home/chi/.iree/bisect/87483506196da31623fc3c3316dcec8805bb61b0 --force-reinstall iree-base-compiler
Looking in links: /home/chi/.iree/bisect/87483506196da31623fc3c3316dcec8805bb61b0
Processing /home/chi/.iree/bisect/87483506196da31623fc3c3316dcec8805bb61b0/iree_base_compiler-3.3.0.dev0+87483506196da31623fc3c3316dcec8805bb61b0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Installing collected packages: iree-base-compiler
Attempting uninstall: iree-base-compiler
Found existing installation: iree-base-compiler 3.3.0.dev0+87483506196da31623fc3c3316dcec8805bb61b0
Uninstalling iree-base-compiler-3.3.0.dev0+87483506196da31623fc3c3316dcec8805bb61b0:
Successfully uninstalled iree-base-compiler-3.3.0.dev0+87483506196da31623fc3c3316dcec8805bb61b0
Successfully installed iree-base-compiler-3.3.0.dev0+87483506196da31623fc3c3316dcec8805bb61b0
Running command: /home/chi/.iree/bisect/87483506196da31623fc3c3316dcec8805bb61b0/.venv/bin/python -m pip install --no-deps --no-index -f /home/chi/.iree/bisect/87483506196da31623fc3c3316dcec8805bb61b0 --force-reinstall iree-base-runtime
Looking in links: /home/chi/.iree/bisect/87483506196da31623fc3c3316dcec8805bb61b0
Processing /home/chi/.iree/bisect/87483506196da31623fc3c3316dcec8805bb61b0/iree_base_runtime-3.3.0.dev0+87483506196da31623fc3c3316dcec8805bb61b0-cp311-cp311-manylinux_2_28_x86_64.whl
Installing collected packages: iree-base-runtime
Attempting uninstall: iree-base-runtime
Found existing installation: iree-base-runtime 3.3.0.dev0+87483506196da31623fc3c3316dcec8805bb61b0
Uninstalling iree-base-runtime-3.3.0.dev0+87483506196da31623fc3c3316dcec8805bb61b0:
Successfully uninstalled iree-base-runtime-3.3.0.dev0+87483506196da31623fc3c3316dcec8805bb61b0
Successfully installed iree-base-runtime-3.3.0.dev0+87483506196da31623fc3c3316dcec8805bb61b0
venv setup complete at '/home/chi/.iree/bisect/87483506196da31623fc3c3316dcec8805bb61b0/.venv'. Activate it with
source /home/chi/.iree/bisect/87483506196da31623fc3c3316dcec8805bb61b0/.venv/bin/activate
+ PATH=/home/chi/.iree/bisect/87483506196da31623fc3c3316dcec8805bb61b0/.venv/bin:/usr/lib/git-core:/usr/lib/git-core:/sharedfile/bisect.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
+ set +e
+ iree-compile /sharedfile/attn/128/fp8_attn.mlir --iree-hip-target=gfx942 -o=/sharedfile/attn/128/fp8_attn.vmfb --iree-hal-target-device=hip --iree-dispatch-creation-enable-aggressive-fusion=true --iree-global-opt-propagate-transposes=true --iree-opt-aggressively-propagate-transposes=true --iree-opt-data-tiling=false '--iree-preprocessing-pass-pipeline=builtin.module(util.func(iree-preprocessing-generalize-linalg-matmul-experimental))' --iree-hal-indirect-command-buffers=true --iree-stream-resource-memory-model=discrete --iree-hal-memoization=true --iree-opt-strip-assertions
+ ROCR_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ iree-benchmark-module --hip_use_streams=true --module=/sharedfile/attn/128/fp8_attn.vmfb --parameters=model=/sharedfile/attn/fp8_attn.irpa --device=hip://4 --function=prefill_bs4 --input=4x128xi64=@/sharedfile/128/prefill/prefill_token_ids_4x128xi64.bin --input=4xi64=@/sharedfile/128/prefill/prefill_seq_lens_4xi64.bin --input=4x4xi64=@/sharedfile/128/prefill/prefill_seq_block_ids_4x4xi64.bin --input=261x2097152xf8E4M3FNUZ=@/sharedfile/128/prefill/prefill_cache_state_261x2097152xf8E4M3FNUZ.bin --benchmark_repetitions=3
2025-03-16T17:52:04-07:00
Running /home/chi/.iree/bisect/87483506196da31623fc3c3316dcec8805bb61b0/.venv/lib/python3.11/site-packages/iree/_runtime_libs/iree-benchmark-module
Run on (96 X 3810.79 MHz CPU s)
CPU Caches:
L1 Data 32 KiB (x96)
L1 Instruction 32 KiB (x96)
L2 Unified 1024 KiB (x96)
L3 Unified 32768 KiB (x16)
Load Average: 2.80, 2.53, 3.70
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
-------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
-------------------------------------------------------------------------------------------------------
BM_prefill_bs4/process_time/real_time 27.5 ms 28.0 ms 26 items_per_second=36.3144/s
BM_prefill_bs4/process_time/real_time 27.5 ms 27.9 ms 26 items_per_second=36.3179/s
BM_prefill_bs4/process_time/real_time 27.6 ms 27.9 ms 26 items_per_second=36.2956/s
BM_prefill_bs4/process_time/real_time_mean 27.5 ms 27.9 ms 3 items_per_second=36.3093/s
BM_prefill_bs4/process_time/real_time_median 27.5 ms 27.9 ms 3 items_per_second=36.3144/s
BM_prefill_bs4/process_time/real_time_stddev 0.009 ms 0.025 ms 3 items_per_second=0.0119748/s
BM_prefill_bs4/process_time/real_time_cv 0.03 % 0.09 % 3 items_per_second=0.03%
+ RET_VALUE=0
+ '[' 0 -ne 0 ']'
Bisecting: 1 revision left to test after this (roughly 1 step)
[3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b] [Codegen] Set full workgroup tile sizes to 0 (#20159)
running '/home/chi/.iree/bisect/bisect_run_script.sh'
++ git rev-parse BISECT_HEAD
+ REF_HASH=3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b
+ python /home/chi/src/iree/build_tools/pkgci/bisect/../setup_venv.py /home/chi/.iree/bisect/3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b/.venv --artifact-path=/home/chi/.iree/bisect/3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b --fetch-git-ref=3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b
Finding workflow run for ref: 3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b
Using normalized ref: 3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b
Running command to list workflow runs:
gh api -H Accept: application/vnd.github+json -H X-GitHub-Api-Version: 2022-11-28 /repos/iree-org/iree/actions/workflows/pkgci.yml/runs?head_sha=3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b
Found workflow run: https://github.com/iree-org/iree/actions/runs/13797422275
Installing wheels: [(PosixPath('/home/chi/.iree/bisect/3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b'), 'iree-base-compiler'), (PosixPath('/home/chi/.iree/bisect/3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b'), 'iree-base-runtime')]
Running command: /home/chi/.iree/bisect/3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b/.venv/bin/python -m pip install --no-deps --no-index -f /home/chi/.iree/bisect/3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b --force-reinstall iree-base-compiler
Looking in links: /home/chi/.iree/bisect/3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b
Processing /home/chi/.iree/bisect/3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b/iree_base_compiler-3.3.0.dev0+3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Installing collected packages: iree-base-compiler
Attempting uninstall: iree-base-compiler
Found existing installation: iree-base-compiler 3.3.0.dev0+3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b
Uninstalling iree-base-compiler-3.3.0.dev0+3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b:
Successfully uninstalled iree-base-compiler-3.3.0.dev0+3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b
Successfully installed iree-base-compiler-3.3.0.dev0+3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b
Running command: /home/chi/.iree/bisect/3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b/.venv/bin/python -m pip install --no-deps --no-index -f /home/chi/.iree/bisect/3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b --force-reinstall iree-base-runtime
Looking in links: /home/chi/.iree/bisect/3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b
Processing /home/chi/.iree/bisect/3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b/iree_base_runtime-3.3.0.dev0+3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b-cp311-cp311-manylinux_2_28_x86_64.whl
Installing collected packages: iree-base-runtime
Attempting uninstall: iree-base-runtime
Found existing installation: iree-base-runtime 3.3.0.dev0+3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b
Uninstalling iree-base-runtime-3.3.0.dev0+3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b:
Successfully uninstalled iree-base-runtime-3.3.0.dev0+3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b
Successfully installed iree-base-runtime-3.3.0.dev0+3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b
venv setup complete at '/home/chi/.iree/bisect/3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b/.venv'. Activate it with
source /home/chi/.iree/bisect/3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b/.venv/bin/activate
+ PATH=/home/chi/.iree/bisect/3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b/.venv/bin:/usr/lib/git-core:/usr/lib/git-core:/sharedfile/bisect.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
+ set +e
+ iree-compile /sharedfile/attn/128/fp8_attn.mlir --iree-hip-target=gfx942 -o=/sharedfile/attn/128/fp8_attn.vmfb --iree-hal-target-device=hip --iree-dispatch-creation-enable-aggressive-fusion=true --iree-global-opt-propagate-transposes=true --iree-opt-aggressively-propagate-transposes=true --iree-opt-data-tiling=false '--iree-preprocessing-pass-pipeline=builtin.module(util.func(iree-preprocessing-generalize-linalg-matmul-experimental))' --iree-hal-indirect-command-buffers=true --iree-stream-resource-memory-model=discrete --iree-hal-memoization=true --iree-opt-strip-assertions
+ ROCR_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ iree-benchmark-module --hip_use_streams=true --module=/sharedfile/attn/128/fp8_attn.vmfb --parameters=model=/sharedfile/attn/fp8_attn.irpa --device=hip://4 --function=prefill_bs4 --input=4x128xi64=@/sharedfile/128/prefill/prefill_token_ids_4x128xi64.bin --input=4xi64=@/sharedfile/128/prefill/prefill_seq_lens_4xi64.bin --input=4x4xi64=@/sharedfile/128/prefill/prefill_seq_block_ids_4x4xi64.bin --input=261x2097152xf8E4M3FNUZ=@/sharedfile/128/prefill/prefill_cache_state_261x2097152xf8E4M3FNUZ.bin --benchmark_repetitions=3
2025-03-16T17:52:25-07:00
Running /home/chi/.iree/bisect/3ddcef5ec1a31824353f9c1b0f94ef01ed0b784b/.venv/lib/python3.11/site-packages/iree/_runtime_libs/iree-benchmark-module
Run on (96 X 3810.79 MHz CPU s)
CPU Caches:
L1 Data 32 KiB (x96)
L1 Instruction 32 KiB (x96)
L2 Unified 1024 KiB (x96)
L3 Unified 32768 KiB (x16)
Load Average: 2.91, 2.58, 3.69
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
-------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
-------------------------------------------------------------------------------------------------------
BM_prefill_bs4/process_time/real_time 29.7 ms 30.3 ms 24 items_per_second=33.6416/s
BM_prefill_bs4/process_time/real_time 29.7 ms 30.4 ms 24 items_per_second=33.6332/s
BM_prefill_bs4/process_time/real_time 29.7 ms 30.3 ms 24 items_per_second=33.6153/s
BM_prefill_bs4/process_time/real_time_mean 29.7 ms 30.3 ms 3 items_per_second=33.63/s
BM_prefill_bs4/process_time/real_time_median 29.7 ms 30.3 ms 3 items_per_second=33.6332/s
BM_prefill_bs4/process_time/real_time_stddev 0.012 ms 0.025 ms 3 items_per_second=0.0134274/s
BM_prefill_bs4/process_time/real_time_cv 0.04 % 0.08 % 3 items_per_second=0.04%
+ RET_VALUE=0
+ '[' 0 -ne 0 ']'
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[779989e89fa6e800005e6fcb0fb59f2c6d1a8839] Add `iree-opt-level` and per-pipeline optimization flags (#20047)
running '/home/chi/.iree/bisect/bisect_run_script.sh'
++ git rev-parse BISECT_HEAD
+ REF_HASH=779989e89fa6e800005e6fcb0fb59f2c6d1a8839
+ python /home/chi/src/iree/build_tools/pkgci/bisect/../setup_venv.py /home/chi/.iree/bisect/779989e89fa6e800005e6fcb0fb59f2c6d1a8839/.venv --artifact-path=/home/chi/.iree/bisect/779989e89fa6e800005e6fcb0fb59f2c6d1a8839 --fetch-git-ref=779989e89fa6e800005e6fcb0fb59f2c6d1a8839
Finding workflow run for ref: 779989e89fa6e800005e6fcb0fb59f2c6d1a8839
Using normalized ref: 779989e89fa6e800005e6fcb0fb59f2c6d1a8839
Running command to list workflow runs:
gh api -H Accept: application/vnd.github+json -H X-GitHub-Api-Version: 2022-11-28 /repos/iree-org/iree/actions/workflows/pkgci.yml/runs?head_sha=779989e89fa6e800005e6fcb0fb59f2c6d1a8839
Found workflow run: https://github.com/iree-org/iree/actions/runs/13799689055
Installing wheels: [(PosixPath('/home/chi/.iree/bisect/779989e89fa6e800005e6fcb0fb59f2c6d1a8839'), 'iree-base-compiler'), (PosixPath('/home/chi/.iree/bisect/779989e89fa6e800005e6fcb0fb59f2c6d1a8839'), 'iree-base-runtime')]
Running command: /home/chi/.iree/bisect/779989e89fa6e800005e6fcb0fb59f2c6d1a8839/.venv/bin/python -m pip install --no-deps --no-index -f /home/chi/.iree/bisect/779989e89fa6e800005e6fcb0fb59f2c6d1a8839 --force-reinstall iree-base-compiler
Looking in links: /home/chi/.iree/bisect/779989e89fa6e800005e6fcb0fb59f2c6d1a8839
Processing /home/chi/.iree/bisect/779989e89fa6e800005e6fcb0fb59f2c6d1a8839/iree_base_compiler-3.3.0.dev0+779989e89fa6e800005e6fcb0fb59f2c6d1a8839-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Installing collected packages: iree-base-compiler
Attempting uninstall: iree-base-compiler
Found existing installation: iree-base-compiler 3.3.0.dev0+779989e89fa6e800005e6fcb0fb59f2c6d1a8839
Uninstalling iree-base-compiler-3.3.0.dev0+779989e89fa6e800005e6fcb0fb59f2c6d1a8839:
Successfully uninstalled iree-base-compiler-3.3.0.dev0+779989e89fa6e800005e6fcb0fb59f2c6d1a8839
Successfully installed iree-base-compiler-3.3.0.dev0+779989e89fa6e800005e6fcb0fb59f2c6d1a8839
Running command: /home/chi/.iree/bisect/779989e89fa6e800005e6fcb0fb59f2c6d1a8839/.venv/bin/python -m pip install --no-deps --no-index -f /home/chi/.iree/bisect/779989e89fa6e800005e6fcb0fb59f2c6d1a8839 --force-reinstall iree-base-runtime
Looking in links: /home/chi/.iree/bisect/779989e89fa6e800005e6fcb0fb59f2c6d1a8839
Processing /home/chi/.iree/bisect/779989e89fa6e800005e6fcb0fb59f2c6d1a8839/iree_base_runtime-3.3.0.dev0+779989e89fa6e800005e6fcb0fb59f2c6d1a8839-cp311-cp311-manylinux_2_28_x86_64.whl
Installing collected packages: iree-base-runtime
Attempting uninstall: iree-base-runtime
Found existing installation: iree-base-runtime 3.3.0.dev0+779989e89fa6e800005e6fcb0fb59f2c6d1a8839
Uninstalling iree-base-runtime-3.3.0.dev0+779989e89fa6e800005e6fcb0fb59f2c6d1a8839:
Successfully uninstalled iree-base-runtime-3.3.0.dev0+779989e89fa6e800005e6fcb0fb59f2c6d1a8839
Successfully installed iree-base-runtime-3.3.0.dev0+779989e89fa6e800005e6fcb0fb59f2c6d1a8839
venv setup complete at '/home/chi/.iree/bisect/779989e89fa6e800005e6fcb0fb59f2c6d1a8839/.venv'. Activate it with
source /home/chi/.iree/bisect/779989e89fa6e800005e6fcb0fb59f2c6d1a8839/.venv/bin/activate
+ PATH=/home/chi/.iree/bisect/779989e89fa6e800005e6fcb0fb59f2c6d1a8839/.venv/bin:/usr/lib/git-core:/usr/lib/git-core:/sharedfile/bisect.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
+ set +e
+ iree-compile /sharedfile/attn/128/fp8_attn.mlir --iree-hip-target=gfx942 -o=/sharedfile/attn/128/fp8_attn.vmfb --iree-hal-target-device=hip --iree-dispatch-creation-enable-aggressive-fusion=true --iree-global-opt-propagate-transposes=true --iree-opt-aggressively-propagate-transposes=true --iree-opt-data-tiling=false '--iree-preprocessing-pass-pipeline=builtin.module(util.func(iree-preprocessing-generalize-linalg-matmul-experimental))' --iree-hal-indirect-command-buffers=true --iree-stream-resource-memory-model=discrete --iree-hal-memoization=true --iree-opt-strip-assertions
+ ROCR_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ iree-benchmark-module --hip_use_streams=true --module=/sharedfile/attn/128/fp8_attn.vmfb --parameters=model=/sharedfile/attn/fp8_attn.irpa --device=hip://4 --function=prefill_bs4 --input=4x128xi64=@/sharedfile/128/prefill/prefill_token_ids_4x128xi64.bin --input=4xi64=@/sharedfile/128/prefill/prefill_seq_lens_4xi64.bin --input=4x4xi64=@/sharedfile/128/prefill/prefill_seq_block_ids_4x4xi64.bin --input=261x2097152xf8E4M3FNUZ=@/sharedfile/128/prefill/prefill_cache_state_261x2097152xf8E4M3FNUZ.bin --benchmark_repetitions=3
2025-03-16T17:52:46-07:00
Running /home/chi/.iree/bisect/779989e89fa6e800005e6fcb0fb59f2c6d1a8839/.venv/lib/python3.11/site-packages/iree/_runtime_libs/iree-benchmark-module
Run on (96 X 3810.79 MHz CPU s)
CPU Caches:
L1 Data 32 KiB (x96)
L1 Instruction 32 KiB (x96)
L2 Unified 1024 KiB (x96)
L3 Unified 32768 KiB (x16)
Load Average: 3.24, 2.67, 3.70
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
-------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
-------------------------------------------------------------------------------------------------------
BM_prefill_bs4/process_time/real_time 29.8 ms 30.6 ms 24 items_per_second=33.5489/s
BM_prefill_bs4/process_time/real_time 29.8 ms 30.7 ms 24 items_per_second=33.5225/s
BM_prefill_bs4/process_time/real_time 29.8 ms 30.7 ms 24 items_per_second=33.5171/s
BM_prefill_bs4/process_time/real_time_mean 29.8 ms 30.7 ms 3 items_per_second=33.5295/s
BM_prefill_bs4/process_time/real_time_median 29.8 ms 30.7 ms 3 items_per_second=33.5225/s
BM_prefill_bs4/process_time/real_time_stddev 0.015 ms 0.056 ms 3 items_per_second=0.0170314/s
BM_prefill_bs4/process_time/real_time_cv 0.05 % 0.18 % 3 items_per_second=0.05%
+ RET_VALUE=0
+ '[' 0 -ne 0 ']'
00e88733e6b8c8cdb351d4516509f56daebdf604 is the first bad commit
commit 00e88733e6b8c8cdb351d4516509f56daebdf604
Author: Quinn Dawkins <[email protected]>
Date: Tue Mar 11 19:17:21 2025 -0400
Revert "[AMDGPU] Do not rewrite or approximate math functions on ROCm" (#20215)
Reverts iree-org/iree#19970 due to failures on rdna3:
https://github.com/iree-org/iree/actions/runs/13796654893/job/38590777349
```
test_pow_types_int64_int64::model.mlir::model.mlir::gpu_rocm_rdna3 _
EXEC @test_pow_types_int64_int64
[FAILED] result[0]: element at index 1 (31) does not match the expected (32)
expected:
3xi64=1 32 729
actual:
3xi64=1 31 729
```
.../compiler/Codegen/Common/MathTransformPass.cpp | 12 +-
.../Codegen/Common/test/math_transform.mlir | 39 ++---
tests/e2e/math/math_ops_llvm-cpu.json | 96 ++++++------
tests/e2e/math/math_ops_rocm.json | 172 +++++++++++----------
4 files changed, 151 insertions(+), 168 deletions(-)
bisect found first bad commit
(bisect.venv) ➜ bisect git:(main) ✗ python ./bisect_packages.py \
--good-ref=6eadf3df6f57a20283da8318f9178b0d92ae50eb \
--bad-ref=87483506196da31623fc3c3316dcec8805bb61b0 \
--test-script=/sharedfile/attn/bisect/issue1.sh
Welcome to bisect_packages.py!
------------------------------------------------------------------
--------- Configuration ------------------------------------------
------------------------------------------------------------------
Searching range : '6eadf3df6f57a20283da8318f9178b0d92ae50eb' - '87483506196da31623fc3c3316dcec8805bb61b0'
Using working directory : '/home/chi/.iree/bisect'
Using test script : '/sharedfile/attn/bisect/issue1.sh'
Current platform is 'Linux-6.8.0-52-generic-x86_64-with-glibc2.35', platform.system is 'Linux'.
Current Python version is '3.11.1 (main, Oct 7 2024, 06:16:08) [GCC 11.4.0]'. This script requires 3.11.
Found gh at '/usr/bin/gh'.
------------------------------------------------------------------
------------------------------------------------------------------
--------- Running git bisect -------------------------------------
------------------------------------------------------------------
Bisecting: 0 revisions left to test after this (roughly 1 step)
[63ed28bca54c61d85180a6be1821f356477c01ab] [GPU] Add support for conv padding when decomposing im2col (#20203)
running '/home/chi/.iree/bisect/bisect_run_script.sh'
++ git rev-parse BISECT_HEAD
+ REF_HASH=63ed28bca54c61d85180a6be1821f356477c01ab
+ python /home/chi/src/iree/build_tools/pkgci/bisect/../setup_venv.py /home/chi/.iree/bisect/63ed28bca54c61d85180a6be1821f356477c01ab/.venv --artifact-path=/home/chi/.iree/bisect/63ed28bca54c61d85180a6be1821f356477c01ab --fetch-git-ref=63ed28bca54c61d85180a6be1821f356477c01ab
Finding workflow run for ref: 63ed28bca54c61d85180a6be1821f356477c01ab
Using normalized ref: 63ed28bca54c61d85180a6be1821f356477c01ab
Running command to list workflow runs:
gh api -H Accept: application/vnd.github+json -H X-GitHub-Api-Version: 2022-11-28 /repos/iree-org/iree/actions/workflows/pkgci.yml/runs?head_sha=63ed28bca54c61d85180a6be1821f356477c01ab
Found workflow run: https://github.com/iree-org/iree/actions/runs/13794855550
Package iree-base-compiler not found in cache. Fetching from linux_x86_64_release_packages...
Fetching artifacts for workflow run: 13794855550
Found artifacts:
linux_x86_64_release_packages: /repos/iree-org/iree/actions/artifacts/2732137639/zip
Downloading artifact /repos/iree-org/iree/actions/artifacts/2732137639/zip
Extracting /home/chi/.iree/bisect/63ed28bca54c61d85180a6be1821f356477c01ab/linux_x86_64_release_packages.zip
Installing wheels: [(PosixPath('/home/chi/.iree/bisect/63ed28bca54c61d85180a6be1821f356477c01ab'), 'iree-base-compiler'), (PosixPath('/home/chi/.iree/bisect/63ed28bca54c61d85180a6be1821f356477c01ab'), 'iree-base-runtime')]
Creating venv at /home/chi/.iree/bisect/63ed28bca54c61d85180a6be1821f356477c01ab/.venv
Running command: /home/chi/.iree/bisect/63ed28bca54c61d85180a6be1821f356477c01ab/.venv/bin/python -m pip install --no-deps --no-index -f /home/chi/.iree/bisect/63ed28bca54c61d85180a6be1821f356477c01ab --force-reinstall iree-base-compiler
Looking in links: /home/chi/.iree/bisect/63ed28bca54c61d85180a6be1821f356477c01ab
Processing /home/chi/.iree/bisect/63ed28bca54c61d85180a6be1821f356477c01ab/iree_base_compiler-3.3.0.dev0+63ed28bca54c61d85180a6be1821f356477c01ab-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Installing collected packages: iree-base-compiler
Successfully installed iree-base-compiler-3.3.0.dev0+63ed28bca54c61d85180a6be1821f356477c01ab
Running command: /home/chi/.iree/bisect/63ed28bca54c61d85180a6be1821f356477c01ab/.venv/bin/python -m pip install --no-deps --no-index -f /home/chi/.iree/bisect/63ed28bca54c61d85180a6be1821f356477c01ab --force-reinstall iree-base-runtime
Looking in links: /home/chi/.iree/bisect/63ed28bca54c61d85180a6be1821f356477c01ab
Processing /home/chi/.iree/bisect/63ed28bca54c61d85180a6be1821f356477c01ab/iree_base_runtime-3.3.0.dev0+63ed28bca54c61d85180a6be1821f356477c01ab-cp311-cp311-manylinux_2_28_x86_64.whl
Installing collected packages: iree-base-runtime
Successfully installed iree-base-runtime-3.3.0.dev0+63ed28bca54c61d85180a6be1821f356477c01ab
venv setup complete at '/home/chi/.iree/bisect/63ed28bca54c61d85180a6be1821f356477c01ab/.venv'. Activate it with
source /home/chi/.iree/bisect/63ed28bca54c61d85180a6be1821f356477c01ab/.venv/bin/activate
+ PATH=/home/chi/.iree/bisect/63ed28bca54c61d85180a6be1821f356477c01ab/.venv/bin:/usr/lib/git-core:/usr/lib/git-core:/sharedfile/bisect.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
+ set +e
+ iree-compile /sharedfile/attn/128/fp8_attn.mlir --iree-hip-target=gfx942 -o=/sharedfile/attn/128/fp8_attn.vmfb --iree-hal-target-device=hip --iree-dispatch-creation-enable-aggressive-fusion=true --iree-global-opt-propagate-transposes=true --iree-opt-aggressively-propagate-transposes=true --iree-opt-data-tiling=false '--iree-preprocessing-pass-pipeline=builtin.module(util.func(iree-preprocessing-generalize-linalg-matmul-experimental))' --iree-hal-indirect-command-buffers=true --iree-stream-resource-memory-model=discrete --iree-hal-memoization=true --iree-opt-strip-assertions
+ ROCR_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ iree-benchmark-module --hip_use_streams=true --module=/sharedfile/attn/128/fp8_attn.vmfb --parameters=model=/sharedfile/attn/fp8_attn.irpa --device=hip://4 --function=prefill_bs4 --input=4x128xi64=@/sharedfile/128/prefill/prefill_token_ids_4x128xi64.bin --input=4xi64=@/sharedfile/128/prefill/prefill_seq_lens_4xi64.bin --input=4x4xi64=@/sharedfile/128/prefill/prefill_seq_block_ids_4x4xi64.bin --input=261x2097152xf8E4M3FNUZ=@/sharedfile/128/prefill/prefill_cache_state_261x2097152xf8E4M3FNUZ.bin --benchmark_repetitions=3
2025-03-16T17:56:28-07:00
Running /home/chi/.iree/bisect/63ed28bca54c61d85180a6be1821f356477c01ab/.venv/lib/python3.11/site-packages/iree/_runtime_libs/iree-benchmark-module
Run on (96 X 3810.79 MHz CPU s)
CPU Caches:
L1 Data 32 KiB (x96)
L1 Instruction 32 KiB (x96)
L2 Unified 1024 KiB (x96)
L3 Unified 32768 KiB (x16)
Load Average: 2.70, 2.60, 3.44
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
-------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
-------------------------------------------------------------------------------------------------------
BM_prefill_bs4/process_time/real_time 26.4 ms 27.2 ms 27 items_per_second=37.9342/s
BM_prefill_bs4/process_time/real_time 26.4 ms 27.3 ms 27 items_per_second=37.893/s
BM_prefill_bs4/process_time/real_time 26.2 ms 27.1 ms 27 items_per_second=38.0967/s
BM_prefill_bs4/process_time/real_time_mean 26.3 ms 27.2 ms 3 items_per_second=37.9746/s
BM_prefill_bs4/process_time/real_time_median 26.4 ms 27.2 ms 3 items_per_second=37.9342/s
BM_prefill_bs4/process_time/real_time_stddev 0.075 ms 0.101 ms 3 items_per_second=0.107699/s
BM_prefill_bs4/process_time/real_time_cv 0.28 % 0.37 % 3 items_per_second=0.28%
+ RET_VALUE=0
+ '[' 0 -ne 0 ']'
87483506196da31623fc3c3316dcec8805bb61b0 is the first bad commit
commit 87483506196da31623fc3c3316dcec8805bb61b0
Author: Bangtian Liu <[email protected]>
Date: Tue Mar 11 15:18:06 2025 -0400
Integrates/llvm 20250310: Bump to llvm/llvm-project@967ab7e (#20207)
bump to
llvm@[967ab7e08e62a35cc65f34e21fbeb00abf3eb83f](https://github.com/iree-org/llvm-project/commit/967ab7e08e62a35cc65f34e21fbeb00abf3eb83f)
create one revert
commit@[c190c8d](https://github.com/iree-org/llvm-project/commit/c190c8df3a501b21f7f018ec3dd4293651736560)
to this PR https://github.com/llvm/llvm-project/pull/129850.
add the commit
[@d80a859](https://github.com/iree-org/llvm-project/commit/d80a85942af26ebfa883054136eb664d13f2714b)
to fix bazel test error.
PR #129850 attempted to use TableGen to define
OneShotBufferizePassOption for controlling the bufferization of all
operations. However, it appears that the TableGen implementation does
not yet fully support all the options available in the manually defined
OneShotBufferizationOptions struct.
For the usage of all the vector transformation options, follow Jakub's
suggestion below to pick up the default values instead of setting them
manually.
```c++
VectorTransformsOptions defaultOptions;
...
vector::populateVectorTransposeLoweringPatterns(
patterns, defaultOptions.vectorTransposeLowering);
```
---------
Signed-off-by: Bangtian Liu <[email protected]>
.../InputConversion/test/apply_pdl_patterns_tosa.mlir | 10 ++++++----
.../TOSA/InputConversion/test/convert_i48_to_i64.mlir | 4 ++--
.../input/TOSA/InputConversion/test/tosa.pdl.mlir | 6 +++++-
compiler/plugins/target/CUDA/CUDATarget.cpp | 2 +-
compiler/plugins/target/LLVMCPU/LLVMCPUTarget.cpp | 2 +-
.../src/iree/compiler/Codegen/LLVMCPU/ConvertToLLVM.cpp | 5 +++--
.../Codegen/LLVMCPU/LLVMCPUVectorTransposeLowering.cpp | 4 ++--
.../Codegen/LLVMCPU/LLVMCPUVirtualVectorLowering.cpp | 2 +-
.../LLVMCPU/test/select_x86_64_lowering_strategy.mlir | 2 +-
.../src/iree/compiler/Codegen/LLVMGPU/ConvertToNVVM.cpp | 9 +++++----
.../src/iree/compiler/Codegen/LLVMGPU/ConvertToROCDL.cpp | 9 +++++----
.../compiler/Codegen/LLVMGPU/LLVMGPUVectorLowering.cpp | 7 ++++---
.../compiler/Codegen/SPIRV/SPIRVFinalVectorLowering.cpp | 6 ++++--
.../Codegen/SPIRV/SPIRVInitialVectorLowering.cpp | 6 ++++--
.../iree/compiler/Dialect/HAL/Utils/LLVMLinkerUtils.cpp | 2 +-
samples/custom_dispatch/cpu/mlp_plugin/mlp_tosa.mlir | 10 ++++++----
.../cpu/mlp_plugin/mlp_tosa_spec.pdl.mlir | 6 +++++-
tests/e2e/tosa_ops/const.mlir | 2 +-
tests/e2e/tosa_ops/gather.mlir | 4 ++--
tests/e2e/tosa_ops/matmul.mlir | 8 ++++++--
tests/e2e/tosa_ops/mul.mlir | 4 ++--
tests/e2e/tosa_ops/mul_shift.mlir | 2 +-
tests/e2e/tosa_ops/pad.mlir | 15 +++++++++------
tests/e2e/tosa_ops/reshape.mlir | 6 +++---
tests/e2e/tosa_ops/while.mlir | 16 ++++++++--------
third_party/llvm-project | 2 +-
26 files changed, 89 insertions(+), 62 deletions(-)
bisect found first bad commit
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment