Skip to content

Instantly share code, notes, and snippets.

@0xJchen
Created January 7, 2022 12:23
Show Gist options
  • Save 0xJchen/5f01987ce205f1814fe99062c0d71d4a to your computer and use it in GitHub Desktop.
Save 0xJchen/5f01987ce205f1814fe99062c0d71d4a to your computer and use it in GitHub Desktop.
(RolloutWorker pid=116297) visulization data save folder /home/game/wy/gym-fish/gym_fish/envs/data/
(RolloutWorker pid=116297) CUDA error at xxx.cpp:17 code=100(cudaErrorNoDevice) "cudaMalloc((void **)&data_gpu.nodeIdx, sizeof(int))"
Traceback (most recent call last):
File "ppo_custom.py", line 211, in <module>
trainer = ppo.PPOTrainer(config=ppo_config, env="myEnv")
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 103, in __init__
remote_checkpoint_dir, sync_function_tpl)
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 662, in __init__
sync_function_tpl)
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/tune/trainable.py", line 121, in __init__
self.setup(copy.deepcopy(self.config))
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 113, in setup
super().setup(config)
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 764, in setup
self._init(self.config, self.env_creator)
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 141, in _init
num_workers=self.config["num_workers"])
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 1733, in _make_workers
logdir=self.logdir)
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/evaluation/worker_set.py", line 90, in __init__
lambda p, pid: (pid, p.observation_space, p.action_space)))
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/worker.py", line 1715, in get
raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
2022-01-07 12:19:45,713 WARNING worker.py:1245 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID:
ffffffffffffffffab83b762813e1a504fbe64ce01000000 Worker ID: 758b6c36700c88bbb880e91b992c9aaff268f203b4033ab6f4026e0e Node ID: c93bb8f0edf565e13b29a1fdce556a602e5dabbda8be8be1d3e208c1 Worker IP address: 172.17.0.2 Worker port: 34983 Worker PID: 116297
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment