Created
January 7, 2022 12:23
-
-
Save 0xJchen/5f01987ce205f1814fe99062c0d71d4a to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(RolloutWorker pid=116297) visulization data save folder /home/game/wy/gym-fish/gym_fish/envs/data/ | |
(RolloutWorker pid=116297) CUDA error at xxx.cpp:17 code=100(cudaErrorNoDevice) "cudaMalloc((void **)&data_gpu.nodeIdx, sizeof(int))" | |
Traceback (most recent call last): | |
File "ppo_custom.py", line 211, in <module> | |
trainer = ppo.PPOTrainer(config=ppo_config, env="myEnv") | |
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 103, in __init__ | |
remote_checkpoint_dir, sync_function_tpl) | |
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 662, in __init__ | |
sync_function_tpl) | |
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/tune/trainable.py", line 121, in __init__ | |
self.setup(copy.deepcopy(self.config)) | |
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 113, in setup | |
super().setup(config) | |
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 764, in setup | |
self._init(self.config, self.env_creator) | |
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 141, in _init | |
num_workers=self.config["num_workers"]) | |
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 1733, in _make_workers | |
logdir=self.logdir) | |
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/evaluation/worker_set.py", line 90, in __init__ | |
lambda p, pid: (pid, p.observation_space, p.action_space))) | |
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper | |
return func(*args, **kwargs) | |
File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/worker.py", line 1715, in get | |
raise value | |
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task. | |
2022-01-07 12:19:45,713 WARNING worker.py:1245 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: | |
ffffffffffffffffab83b762813e1a504fbe64ce01000000 Worker ID: 758b6c36700c88bbb880e91b992c9aaff268f203b4033ab6f4026e0e Node ID: c93bb8f0edf565e13b29a1fdce556a602e5dabbda8be8be1d3e208c1 Worker IP address: 172.17.0.2 Worker port: 34983 Worker PID: 116297 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment