目的
在robotsuite 中，解除 visual 和 collision 的耦合，实现添加物体的可视化表达与实际操作分离；该方法对算法及系统设计无有效作用。
步骤
基于最简单案例Stack 的改进：
self.cubeC = BoxObject(
name="cube_target",
size_min=[0.04, 0.04, 0.02],
size_max=[0.04, 0.04, 0.02],
rgba=[0.5, 0.5, 0, 0.3],
obj_type='all',
# obj_type='collision',
# obj_type='visual',
)
Traceback (most recent call last):
File "collect_human_demonstrations.py", line 215, in <module>
env = suite.make(
File "/home/idm/Shares/imitation/robosuite/robosuite/environments/base.py", line 40, in make
return REGISTERED_ENVS[env_name](*args, **kwargs)
File "/home/idm/Shares/imitation/robosuite/robosuite/environments/manipulation/manipulation_env.py", line 162, in __init__
super().__init__(
File "/home/idm/Shares/imitation/robosuite/robosuite/environments/robot_env.py", line 214, in __init__
super().__init__(
File "/home/idm/Shares/imitation/robosuite/robosuite/environments/base.py", line 137, in __init__
self._initialize_sim()
File "/home/idm/Shares/imitation/robosuite/robosuite/environments/base.py", line 230, in _initialize_sim
self.sim = MjSim.from_xml_string(xml)
File "/home/idm/Shares/imitation/robosuite/robosuite/utils/binding_utils.py", line 1073, in from_xml_string
model = mujoco.MjModel.from_xml_string(xml)
ValueError: Error: mass and inertia of moving bodies must be larger than mjMINVAL
Object name = cube_target_main, id = 25, line = 275, column = -1
事实说明：使用 BoxObject 的方式是无法表示为 visual 模式的；
依旧是采用 xml 的方式实现 obj 的输入：
创建 xml 文件
<mujoco model="can-visual">
<asset>
<mesh file="meshes/can.msh" name="can_mesh"/>
</asset>
<worldbody>
<body>
<body name="object">
<!-- <geom pos="0 0 0" mesh="can_mesh" type="mesh" rgba="0.8 0.8 0.8 0.3" conaffinity="0" contype="0" group="1" condim="4"/> -->
<geom pos="-0.15 0 0.0" size="0.02 0.2 0.2" type="box" rgba="0.8 0.8 0.8 0.3" conaffinity="0" contype="0" group="1" condim="4" />
</body>
<site rgba="0 0 0 0" size="0.005" pos="0 0 -0.06" name="bottom_site"/>
<site rgba="0 0 0 0" size="0.005" pos="0 0 0.04" name="top_site"/>
<site rgba="0 0 0 0" size="0.005" pos="0.025 0.025 0" name="horizontal_radius_site"/>
</body>
</worldbody>
</mujoco>
创建物体
根据 xml 创建物体，指定部分属性
class CanVisualObject(MujocoXMLObject):
"""
Visual fiducial of coke can (used in PickPlace)
Fiducial objects are not involved in collision physics.
They provide a point of reference to indicate a position.
"""
def __init__(self, name):
super().__init__(
"./objects/cube.xml",
name=name,
joints=None,
obj_type="visual",
duplicate_collision_geoms=True,
)
env 中设置
添加物体：
self.cube_target = CanVisualObject(name="visual_cube_target")
objects = [self.needle, self.tripod, self.cube_target]
指定位置：
def _get_initial_placement_bounds(self):
cube_target=dict(
x=(0.05, 0.05),
y=(0.0, 0.),
z_rot=(np.pi / 3., np.pi / 2.),
reference=self.table_offset,
),
)
def _get_placement_initializer(self):
bounds = self._get_initial_placement_bounds()
self.placement_initializer = SequentialCompositeSampler(name="ObjectSampler")
# add new obj
self.placement_initializer.append_sampler(
sampler=UniformRandomSampler(
name="VisualCubeTargetSampler",
mujoco_objects=self.cube_target,
x_range=bounds["cube_target"]["x"],
y_range=bounds["cube_target"]["y"],
rotation=bounds["tripod"]["z_rot"],
rotation_axis='z',
ensure_object_boundary_in_range=False,
ensure_valid_placement=True,
reference_pos=bounds["tripod"]["reference"],
z_offset=0.001,
)
)
def _setup_references(self):
"""
Sets up references to important components. A reference is typically an
index or a list of indices that point to the corresponding elements
in a flatten array, which is how MuJoCo stores physical simulation data.
"""
super()._setup_references()
# Additional object references from this env
# import ipdb; ipdb.set_trace()
self.obj_body_id = dict(
needle=self.sim.model.body_name2id(self.needle.root_body),
tripod=self.sim.model.body_name2id(self.tripod.root_body),
cube_target=self.sim.model.body_name2id(self.cube_target.root_body)
)
def _reset_internal(self):
"""
Resets simulation internal configurations.
"""
super()._reset_internal()
# import ipdb; ipdb.set_trace()
# Reset all object positions using initializer sampler if we're not directly loading from an xml
if not self.deterministic_reset:
# Sample from the placement initializer for all objects
object_placements = self.placement_initializer.sample()
# Loop through all objects and reset their positions
for obj_pos, obj_quat, obj in object_placements.values():
if "visual" in obj.name.lower():
self.sim.model.body_pos[self.obj_body_id['cube_target']] = obj_pos
self.sim.model.body_quat[self.obj_body_id['cube_target']] = obj_quat
else:
self.sim.data.set_joint_qpos(obj.joints[0], np.concatenate([np.array(obj_pos), np.array(obj_quat)]))
可视化
修改后场景图如下，灰色的半透明盒子就是新添加的物体：
推理
果不其然，推理报错了，错误信息如下：
File "/home/idm/Shares/imitation/robomimic/robomimic/algo/algo.py", line 525, in __call__
ac = self.policy.get_action(obs_dict=ob, goal_dict=goal)
File "/home/idm/Shares/imitation/robomimic/robomimic/algo/bc.py", line 565, in get_action
action, self._rnn_hidden_state = self.nets["policy"].forward_step(
File "/home/idm/Shares/imitation/robomimic/robomimic/models/policy_nets.py", line 966, in forward_step
acts, state = self.forward(
File "/home/idm/Shares/imitation/robomimic/robomimic/models/policy_nets.py", line 909, in forward
out = self.forward_train(obs_dict=obs_dict, goal_dict=goal_dict, rnn_init_state=rnn_init_state, return_state=return_state)
File "/home/idm/Shares/imitation/robomimic/robomimic/models/policy_nets.py", line 853, in forward_train
outputs = RNN_MIMO_MLP.forward(
File "/home/idm/Shares/imitation/robomimic/robomimic/models/obs_nets.py", line 792, in forward
return self.nets["rnn"].forward(inputs=rnn_inputs, rnn_init_state=rnn_init_state, return_state=return_state)
File "/home/idm/Shares/imitation/robomimic/robomimic/models/base_nets.py", line 422, in forward
outputs, rnn_state = self.nets(inputs, rnn_init_state)
File "/home/idm/Shares/miniconda3/envs/imitation/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/idm/Shares/miniconda3/envs/imitation/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/idm/Shares/miniconda3/envs/imitation/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 875, in forward
self.check_forward_args(input, hx, batch_sizes)
File "/home/idm/Shares/miniconda3/envs/imitation/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 790, in check_forward_args
self.check_input(input, batch_sizes)
File "/home/idm/Shares/miniconda3/envs/imitation/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 236, in check_input
raise RuntimeError(
RuntimeError: input.size(-1) must be equal to input_size. Expected 37, got 51
其中，最关键的信息是：RuntimeError: input.size(-1) must be equal to input_size. Expected 37, got 51，这是因为新添加了一个物体，导致 RNN 输出维度不一致导致的。
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
