Gym cuda error: out of memory:

问题描述：

[Error] [carb.gym.plugin] Gym cuda error: out of memory: ../../../source/plugins/carb/gym/impl/Gym/GymPhysXCuda.cu: 937

全部问题如下

(isaac) aq@dog:~/umi-on-legs/mani-centric-wbc$ python scripts/play.py --ckpt_path /home/aq/checkpoints/tossing/ours-real/model.pt --trajectory_file_path /home/aq/data/tossing.pkl --device cuda:0 --num_steps 1000 --num_envs 1 --visualize
Importing module 'gym_38' (/home/aq/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_38.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/aq/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
PyTorch version 2.1.0
Device count 1
/home/aq/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/aq/.cache/torch_extensions/py38_cu121 as PyTorch extensions root...
Emitting ninja build file /home/aq/.cache/torch_extensions/py38_cu121/gymtorch/build.ninja...
Building extension module gymtorch...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module gymtorch...
wandb: Tracking run with wandb version 0.15.12
wandb: W&B syncing is set to `offline` in this directory.  
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
Not connected to PVD
+++ Using GPU PhysX
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: enabled
[Error] [carb.gym.plugin] Gym cuda error: out of memory: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 1721
[Error] [carb.gym.plugin] Gym cuda error: invalid resource handle: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 5993
[Error] [carb.gym.plugin] Gym cuda error: out of memory: ../../../source/plugins/carb/gym/impl/Gym/GymPhysXCuda.cu: 937
[Error] [carb.gym.plugin] Failed to fill root state tensor
[Error] [carb.gym.plugin] Gym cuda error: invalid resource handle: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 6137
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysXCuda.cu: 991
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 4262
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 6428
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysXCuda.cu: 1028
[Error] [carb.gym.plugin] Failed to copy force sensor data
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 5754
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysXCuda.cu: 926
[Error] [carb.gym.plugin] Failed to fill rigid body state tensor
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 6428
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysXCuda.cu: 1028
[Error] [carb.gym.plugin] Failed to copy force sensor data
Traceback (most recent call last):
  File "/home/aq/miniconda3/envs/isaac/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 92, in _call_target
    return _target_(*args, **kwargs)
  File "/home/aq/umi-on-legs/mani-centric-wbc/legged_gym/env/isaacgym/env.py", line 158, in __init__
    self.state = EnvState.initialize(
  File "/home/aq/umi-on-legs/mani-centric-wbc/legged_gym/env/isaacgym/state.py", line 226, in initialize
    prev_dof_pos=dof_states[..., 0].clone(),
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "scripts/play.py", line 267, in <module>
    play()
  File "scripts/play.py", line 94, in play
    env: IsaacGymEnv = hydra.utils.instantiate(
  File "/home/aq/miniconda3/envs/isaac/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 226, in instantiate
    return instantiate_node(
  File "/home/aq/miniconda3/envs/isaac/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 347, in instantiate_node
    return _call_target(_target_, partial, args, kwargs, full_key)
  File "/home/aq/miniconda3/envs/isaac/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 97, in _call_target
    raise InstantiationException(msg) from e
hydra.errors.InstantiationException: Error in call to target 'legged_gym.env.isaacgym.env.IsaacGymEnv':
RuntimeError('CUDA error: an illegal memory access was encountered\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n')
wandb: Waiting for W&B process to finish... (failed 1).
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/aq/umi-on-legs/mani-centric-wbc/wandb/offline-run-20250313_214533-l596f0cl
wandb: Find logs at: ./wandb/offline-run-20250313_214533-l596f0cl/logs
(isaac) aq@dog:~/umi-on-legs/mani-centric-wbc$ 

问题解答：

查了一下显存大小只有8G，不足以支撑仿真。换了一台12G的显存可以跑起来