Skip to content

Instantly share code, notes, and snippets.

View relyt0925's full-sized avatar

Tyler Lisowski relyt0925

9 followers · 0 following
View GitHub Profile
@relyt0925
relyt0925 / gist:5182f8e3065d45d3a02249ed485d9e6a
Created July 27, 2024 17:54
new mtbench_branch output
[root@tyler-rhel-newimage instructlab]# /root/ilab model evaluate --benchmark mt_bench_branch --model /var/instructlabbigdisk/instructlab/skillscheckpoints/hf_format/samples_1056/ --judge-model /var/instructlabbigdisk/instructlab/models/prometheus-eval/prometheus-8x7b-v2.0/ --base-model /var/instructlabbigdisk/instructlab/models/ibm-granite/granite-7b-base/ --output-dir /var/instructlabbigdisk/instructlab/evaltracker/skillscheckpoints/samples_1056/ --gpus 8 --backend vllm --enable-serving-output --taxonomy-path /var/instructlabbigdisk/instructlab/.local/share/instructlab/taxonomy/ --base-branch HEAD --branch HEAD
INFO 2024-07-27 16:30:41,640 numexpr.utils:145: Note: detected 80 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
INFO 2024-07-27 16:30:41,641 numexpr.utils:148: Note: NumExpr detected 80 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
INFO 2024-07-27 16:30:41,641 numexpr.utils:161: NumExpr defaulting to 16 threads.
Generating
@relyt0925
relyt0925 / gist:ed5c0601e419028f6eedc12b4fb5fd28
Created July 27, 2024 04:44
newtraining output
[root@tyler-rhel-newimage instructlab]# /root/ilab model train --data-path /var/instructlabbigdisk/instructlab/generateddata/messages_Mixtral-8x7B-Instruct-v0_2024-07-27T04_27_23.jsonl --model-path /var/instructlabbigdisk/instructlab/models/ibm-granite/granite-7b-base/ --ckpt-output-dir /var/instructlabbigdisk/instructlab/knowledgecheckpoints/ --device cuda --gpus 8 --max-batch-len 1 --effective-batch-size 8 --save-samples 46
[2024-07-27 04:38:32,852] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[WARNING] async_io requires the dev libaio .so object and headers but these were not found.
[WARNING] async_io: please install the libaio-devel package with yum
[WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
[WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
[WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 b
@relyt0925
relyt0925 / gist:58852c7e2f8c1d8538db6b2bc361f0e5
Created July 27, 2024 04:29
sdgv018_output
[root@tyler-rhel-newimage instructlab]# /root/ilab data generate --taxonomy-path /var/instructlabbigdisk/instructlab/.local/share/instructlab/taxonomy/compositional_skills/writing/grounded/editing/spelling/qna.yaml --endpoint-url https://781d2e7c-us-east.lb.appdomain.cloud/v1 --model-family mixtral --sdg-scale-factor 100 --model /instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1 --output-dir /var/instructlabbigdisk/instructlab/generateddata/ --tls-insecure --rouge-threshold 1.0
INFO 2024-07-27 04:27:22,396 numexpr.utils:145: Note: detected 80 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
INFO 2024-07-27 04:27:22,396 numexpr.utils:148: Note: NumExpr detected 80 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
INFO 2024-07-27 04:27:22,397 numexpr.utils:161: NumExpr defaulting to 16 threads.
INFO 2024-07-27 04:27:22,785 datasets:58: PyTorch version 2.3.1 available.
Generating synthetic data using '/instructlab/models/mistralai/Mix
@relyt0925
relyt0925 / gist:f07f57fd09065a4019cbdcd20ec73ea6
Created July 11, 2024 23:48
sdg ouput challenge
{"system": "You are an AI language model developed by IBM Research. You are a cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive behavior.", "user": "Identify any Personal Identifying Information in the following input:.\nHere's an example of an input with Personal Identifying Information (PII) - {\"email\": \"[[email protected]](mailto:[email protected])\", \"phone\\_number\": \"+1 555-123-4567\", \"age\": 35}", "assistant": "The Personal Identifying Information in the given input are the email and phone number - [[email protected]](mailto:[email protected]) and +1 555-123-4567."}
{"system": "You are an AI language model developed by IBM Research. You are a cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive behavior.", "user": "Mask all Personal Identifying Information in the input with ***.\nHere's an example of an input with Personal
@relyt0925
relyt0925 / mt_bench_branch.log
Last active August 11, 2024 14:09
mt_bench_branch.log (ilab model evaluate --benchmark mt_bench_branch --model /instructlab/models/tuned-0701-1954/samples_4992 --judge-model /instructlab/models/prometheus-eval/prometheus-8x7b-v2.0 --taxonomy-path /instructlab/taxonomy --output-dir /instructlab/mtbench --base-model /instructlab/models/ibm/granite-7b-base --branch main --base-bran…
INFO 2024-07-05 19:34:55,630 utils.py:145: _init_num_threads Note: detected 80 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
INFO 2024-07-05 19:34:55,630 utils.py:148: _init_num_threads Note: NumExpr detected 80 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
INFO 2024-07-05 19:34:55,630 utils.py:161: _init_num_threads NumExpr defaulting to 16 threads.
INFO 2024-07-05 19:34:55,789 config.py:58: <module> PyTorch version 2.3.1 available.
Generating questions and reference answers from qna files for branch main...
INFO 2024-07-05 19:35:02,464 vllm.py:148: run_vllm vLLM starting up on pid 212 at http://127.0.0.1:48895/v1
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 148/148 [00:00<00:00, 169.53it/s]
generated 416 questions
100%|█████████████████████████████████████████████████████████████
@relyt0925
relyt0925 / mt_bench.log
Last active August 11, 2024 14:08
ilab mt_bench eval log (ilab model evaluate --benchmark mt_bench --model /instructlab/models/tuned-0701-1954/samples_4992 --judge-model /instructlab/models/prometheus-eval/prometheus-8x7b-v2.0 --taxonomy-path /instructlab/taxonomy/ --output-dir /instructlab/mtbench)
INFO 2024-07-05 18:53:02,883 utils.py:145: _init_num_threads Note: detected 80 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
INFO 2024-07-05 18:53:02,883 utils.py:148: _init_num_threads Note: NumExpr detected 80 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
INFO 2024-07-05 18:53:02,883 utils.py:161: _init_num_threads NumExpr defaulting to 16 threads.
INFO 2024-07-05 18:53:03,050 config.py:58: <module> PyTorch version 2.3.1 available.
Generating answers...
INFO 2024-07-05 18:53:12,971 vllm.py:148: run_vllm vLLM starting up on pid 212 at http://127.0.0.1:58173/v1
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80/80 [01:02<00:00, 1.27it/s]
Evaluating answers...
INFO 2024-07-05 18:56:08,297 vllm.py:148: run_vllm vLLM starting up on pid 255 at http://127.0.0.1:48517/v1
100%|██████████
@relyt0925
relyt0925 / mmlubranch.log
Created July 7, 2024 02:50
mmlu_branch execution log (ilab model evaluate --model models/tuned-0701-1954/samples_4992 --base-model models/ibm/granite-7b-base --benchmark mmlu_branch --sdg-path generated/lm_eval/tasks)
INFO 2024-07-06 18:00:23,647 utils.py:145: _init_num_threads Note: detected 80 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
INFO 2024-07-06 18:00:23,647 utils.py:148: _init_num_threads Note: NumExpr detected 80 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
INFO 2024-07-06 18:00:23,647 utils.py:161: _init_num_threads NumExpr defaulting to 16 threads.
INFO 2024-07-06 18:00:23,802 config.py:58: <module> PyTorch version 2.3.1 available.
INFO 2024-07-06 18:00:35,580 evaluator.py:152: simple_evaluate Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234
INFO 2024-07-06 18:00:35,580 evaluator.py:189: simple_evaluate Initializing hf model, with arguments: {'pretrained': 'models/tuned-0701-1954/samples_4992', 'dtype': 'bfloat16'}
/usr/local/lib64/python3.11/site-packages/torch/cuda/__init__.py:619: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
INFO 2024-07-06 18:00:35,598 hug
@relyt0925
relyt0925 / mmlu.log
Created July 7, 2024 02:48
example execution of mmlu (ilab model evaluate --model models/tuned-0701-1954/samples_4992 --benchmark mmlu)
INFO 2024-07-06 21:21:06,958 utils.py:145: _init_num_threads Note: detected 80 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
INFO 2024-07-06 21:21:06,958 utils.py:148: _init_num_threads Note: NumExpr detected 80 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
INFO 2024-07-06 21:21:06,958 utils.py:161: _init_num_threads NumExpr defaulting to 16 threads.
INFO 2024-07-06 21:21:07,119 config.py:58: <module> PyTorch version 2.3.1 available.
INFO 2024-07-06 21:21:13,649 evaluator.py:152: simple_evaluate Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234
INFO 2024-07-06 21:21:13,650 evaluator.py:189: simple_evaluate Initializing hf model, with arguments: {'pretrained': 'models/tuned-0701-1954/samples_4992', 'dtype': 'bfloat16'}
/usr/local/lib64/python3.11/site-packages/torch/cuda/__init__.py:619: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
INFO 2024-07-06 21:21:13,666 hug
@relyt0925
relyt0925 / sdggeneratefullrun.log
Last active July 8, 2024 01:42
log file from full run of SDG generate across public taxonomy/skills (look for ilab generate to see the individual SDG generate calls)
This file has been truncated, but you can view the full file.
knowledge/technical_manual/ibm_redbooks/qna.yaml
ilab generate --model-family mixtral --model /instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1 --taxonomy-path /instructlab/taxonomytar/taxonomy/knowledge/technical_manual/ibm_redbooks/qna.yaml --output-dir /instructlab/ibmgenerate/taxonomytar/taxonomy/knowledge/technical_manual/ibm_redbooks --num-instructions 100 --endpoint-url http://0.0.0.0:8080/v1 --model /instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1
time="2024-06-26T20:39:05Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
You are using an aliased command, this will be deprecated in a future release. Please consider using `ilab data generate` instead
Generating synthetic data using '/instructlab/models/mistralai/Mixtral-8x7B-Instruct-v0.1' model, taxonomy:'/instructlab/taxonomytar/taxonomy/knowledge/technical_manual/ibm_redbooks/qna.yaml' against http://0.0.0.0:8080/v1 server
Traceback (most recent call last):
File "/usr/loca
@relyt0925
relyt0925 / trainingrun.log
Created July 7, 2024 02:40
ilab train --num-epochs 10 log file
This file has been truncated, but you can view the full file.
nohup: ignoring input
Converting /instructlab/generated/train_combinedknowlegeskills.jsonl
Converting /instructlab/generated/test_combinedknowlegeskills.jsonl
[16:39:00] INFO !!!!!!!! tokenizer has add_bos_token or utils.py:192
add_eos_token
INFO eos: 32000, pad: 32001, system: 32002, user: utils.py:192
32003, assistant: 32004
Generating train split: 32962 examples [00:00, 295454.57 examples/s]
removing pretraining samples system msg
Map (num_proc=72): 100%|██████████| 32962/32962 [00:00<00:00, 71731.50 examples/s]