stack

(RolloutWorker pid=116297) visulization data save folder /home/game/wy/gym-fish/gym_fish/envs/data/
(RolloutWorker pid=116297) CUDA error at xxx.cpp:17 code=100(cudaErrorNoDevice) "cudaMalloc((void **)&data_gpu.nodeIdx, sizeof(int))"
Traceback (most recent call last):
  File "ppo_custom.py", line 211, in <module>
    trainer = ppo.PPOTrainer(config=ppo_config, env="myEnv")
  File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 103, in __init__
    remote_checkpoint_dir, sync_function_tpl)
  File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 662, in __init__
    sync_function_tpl)
  File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/tune/trainable.py", line 121, in __init__
    self.setup(copy.deepcopy(self.config))
  File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 113, in setup
    super().setup(config)
  File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 764, in setup
    self._init(self.config, self.env_creator)
  File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 141, in _init
    num_workers=self.config["num_workers"])
  File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 1733, in _make_workers
    logdir=self.logdir)
  File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/rllib/evaluation/worker_set.py", line 90, in __init__
    lambda p, pid: (pid, p.observation_space, p.action_space)))
  File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/envs/gym-fish/lib/python3.6/site-packages/ray/worker.py", line 1715, in get
    raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
2022-01-07 12:19:45,713 WARNING worker.py:1245 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID:
ffffffffffffffffab83b762813e1a504fbe64ce01000000 Worker ID: 758b6c36700c88bbb880e91b992c9aaff268f203b4033ab6f4026e0e Node ID: c93bb8f0edf565e13b29a1fdce556a602e5dabbda8be8be1d3e208c1 Worker IP address: 172.17.0.2 Worker port: 34983 Worker PID: 116297