moondream_mm_eleuther.txt

(eleuther) [[email protected] ~/projects/lm-evaluation-harness (multimodal-prototyping)]$ lm_eval --model hf-multimodal --tasks mmmu --batch_size 8 --model_args pretrained=vikhyatk/moondream2 --trust_remote_code
2024-08-05:13:54:15,773 INFO     [__main__.py:272] Verbosity set to INFO
2024-08-05:13:54:15,965 INFO     [__init__.py:406] `group` and `group_alias` keys in tasks' configs will no longer be used in the next release of lm-eval. `tag` will be used to allow to call a collection of tasks just like `group`. `group` will be removed in order to not cause confusion with the new ConfigurableGroup which will be the offical way to create groups with addition of group-wide configuations.
2024-08-05:13:54:21,582 INFO     [__main__.py:357] Passed `--trust_remote_code`, setting environment variable `HF_DATASETS_TRUST_REMOTE_CODE=true`
2024-08-05:13:54:21,582 INFO     [__main__.py:369] Selected Tasks: ['mmmu']
2024-08-05:13:54:21,583 INFO     [evaluator.py:158] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234
2024-08-05:13:54:21,584 INFO     [evaluator.py:195] Initializing hf-multimodal model, with arguments: {'pretrained': 'vikhyatk/moondream2', 'trust_remote_code': True}
2024-08-05:13:54:23,282 INFO     [huggingface.py:169] Using device 'cuda'
configuration_moondream.py: 100%|████████████████████████████████████████████████████████████████████████████████| 3.34k/3.34k [00:00<00:00, 14.7MB/s]
tokenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████| 7.34k/7.34k [00:00<00:00, 27.9MB/s]
vocab.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 798k/798k [00:00<00:00, 8.85MB/s]
merges.txt: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 456k/456k [00:00<00:00, 61.5MB/s]
tokenizer.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 2.11M/2.11M [00:00<00:00, 11.6MB/s]
added_tokens.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 1.08k/1.08k [00:00<00:00, 6.00MB/s]
special_tokens_map.json: 100%|██████████████████████████████████████████████████████████████████████████████████████| 99.0/99.0 [00:00<00:00, 537kB/s]
Traceback (most recent call last):
  File "/home/jrcummings/.conda/envs/eleuther/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/home/jrcummings/projects/lm-evaluation-harness/lm_eval/__main__.py", line 375, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jrcummings/projects/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/jrcummings/projects/lm-evaluation-harness/lm_eval/evaluator.py", line 198, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jrcummings/projects/lm-evaluation-harness/lm_eval/api/model.py", line 148, in create_from_arg_string
    return cls(**args, **args2)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/jrcummings/projects/lm-evaluation-harness/lm_eval/models/huggingface.py", line 207, in __init__
    self._create_tokenizer(
  File "/home/jrcummings/projects/lm-evaluation-harness/lm_eval/models/hf_vlms.py", line 70, in _create_tokenizer
    self.tokenizer = self.processor.tokenizer
                     ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'CodeGenTokenizerFast' object has no attribute 'tokenizer'. Did you mean: '_tokenizer'?