rearrangement-challenge-2022环境使用学习（一）

搭建了rearrangement-challenge-2022的环境：
https://github.com/facebookresearch/habitat-challenge/tree/rearrangement-challenge-2022
habitat最大的缺点是对不同的版本非常的敏感。本文只是针对rearrangement-challenge-2022的学习。

文档一开始会很不完善，会慢慢的补齐。

1、如何使用配置文件初始化环境

主要使用 habitat.Env类进行初始化。传一个config参数，该参数可以通过habitat.get_config()得到，该函数需要指定一个配置文件的路径，举例如下：

import habitat
env = habitat.Env(config=habitat.get_config("/home/yons/LK/habitat-challenge/habitat-lab/configs/datasets/pointnav/habitat_test.yaml")    

配置环境的过程会打印很多信息如下：

# 2024-03-04 21:17:09,546 Initializing dataset PointNav-v1
# 2024-03-04 21:17:09,708 initializing sim Sim-v0
# [21:17:09:715104]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : <Dataset>: Proposing JSON name : default.scene_dataset_config.json from original name : default| This file does not exist.
# [21:17:09:715197]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (capsule3DSolid:capsule3DSolid_hemiRings_4_cylRings_1_segments_12_halfLen_0.75_useTexCoords_false_useTangents_false) created and registered.
# [21:17:09:715224]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (capsule3DWireframe:capsule3DWireframe_hemiRings_8_cylRings_1_segments_16_halfLen_1) created and registered.
# [21:17:09:715242]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (coneSolid:coneSolid_segments_12_halfLen_1.25_rings_1_useTexCoords_false_useTangents_false_capEnd_true) created and registered.
# [21:17:09:715258]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (coneWireframe:coneWireframe_segments_32_halfLen_1.25) created and registered.
# [21:17:09:715270]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cubeSolid:cubeSolid) created and registered.
# [21:17:09:715282]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cubeWireframe:cubeWireframe) created and registered.
# [21:17:09:715299]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cylinderSolid:cylinderSolid_rings_1_segments_12_halfLen_1_useTexCoords_false_useTangents_false_capEnds_true) created and registered.
# [21:17:09:715314]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cylinderWireframe:cylinderWireframe_rings_1_segments_32_halfLen_1) created and registered.
# [21:17:09:715326]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (icosphereSolid:icosphereSolid_subdivs_1) created and registered.
# [21:17:09:715369]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (icosphereWireframe:icosphereWireframe_subdivs_1) created and registered.
# [21:17:09:715397]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (uvSphereSolid:uvSphereSolid_rings_8_segments_16_useTexCoords_false_useTangents_false) created and registered.
# [21:17:09:715429]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (uvSphereWireframe:uvSphereWireframe_rings_16_segments_32) created and registered.
# [21:17:09:715438]:[Metadata] AssetAttributesManager.cpp(110)::AssetAttributesManager : Built default primitive asset templates : 12
# [21:17:09:715778]:[Metadata] SceneDatasetAttributesManager.cpp(35)::createObject : File (default) not found, so new default dataset attributes created  and registered.
# [21:17:09:715787]:[Metadata] MetadataMediator.cpp(120)::createSceneDataset : Dataset default successfully created.
# [21:17:09:715798]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : <Physics Manager>: Proposing JSON name : ./data/default.physics_config.json from original name : ./data/default.physics_config.json| This file exists.
# [21:17:09:715881]:[Metadata] PhysicsAttributesManager.cpp(26)::createObject : JSON Configuration File (./data/default.physics_config.json) based physics manager attributes created and registered.
# [21:17:09:715890]:[Metadata] MetadataMediator.cpp(203)::setActiveSceneDatasetName : Previous active dataset  changed to default successfully.
# [21:17:09:715896]:[Metadata] MetadataMediator.cpp(175)::setCurrPhysicsAttributesHandle : Old physics manager attributes  changed to ./data/default.physics_config.json successfully.
# [21:17:09:715902]:[Metadata] MetadataMediator.cpp(66)::setSimulatorConfiguration : Set new simulator config for scene/stage : data/scene_datasets/habitat-test-scenes/skokloster-castle.glb and dataset : default which is currently active dataset.
# Renderer: NVIDIA GeForce RTX 2080 Ti/PCIe/SSE2 by NVIDIA Corporation
# OpenGL version: 4.6.0 NVIDIA 510.108.03
# Using optional features:
#     GL_ARB_vertex_array_object
#     GL_ARB_ES2_compatibility
#     GL_ARB_separate_shader_objects
#     GL_ARB_robustness
#     GL_ARB_texture_storage
#     GL_ARB_invalidate_subdata
#     GL_ARB_texture_storage_multisample
#     GL_ARB_multi_bind
#     GL_ARB_direct_state_access
#     GL_ARB_get_texture_sub_image
#     GL_ARB_texture_filter_anisotropic
#     GL_KHR_debug
# Using driver workarounds:
#     no-forward-compatible-core-context
#     no-layout-qualifiers-on-old-glsl
#     nv-zero-context-profile-mask
#     nv-implementation-color-read-format-dsa-broken
#     nv-cubemap-inconsistent-compressed-image-size
#     nv-cubemap-broken-full-compressed-image-query
#     nv-compressed-block-size-in-bits
# [21:17:09:817318]:[Metadata] MetadataMediator.cpp(297)::getSceneInstanceAttributesByName : Dataset : default has no preloaded SceneInstanceAttributes or StageAttributes named : data/scene_datasets/habitat-test-scenes/skokloster-castle.glb so loading/creating a new StageAttributes with this name, and then creating a SceneInstanceAttributes with the same name that references this stage.
# [21:17:09:817339]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : <Stage Template>: Proposing JSON name : data/scene_datasets/habitat-test-scenes/skokloster-castle.stage_config.json from original name : data/scene_datasets/habitat-test-scenes/skokloster-castle.glb| This file does not exist.
# [21:17:09:817384]:[Metadata] AbstractObjectAttributesManagerBase.h(184)::createObject : File (data/scene_datasets/habitat-test-scenes/skokloster-castle.glb) exists but is not a recognized config filename extension, so new default Stage Template attributes created and registered.
# [21:17:09:817412]:[Metadata] SceneDatasetAttributes.cpp(47)::addNewSceneInstanceToDataset : Dataset : 'default' : Stage Attributes 'data/scene_datasets/habitat-test-scenes/skokloster-castle.glb' specified in Scene Attributes exists in dataset library.
# [21:17:09:817418]:[Metadata] SceneDatasetAttributes.cpp(84)::addNewSceneInstanceToDataset : Dataset : 'default' : Lighting Layout Attributes 'no_lights' specified in Scene Attributes but does not exist in dataset, so creating.
# [21:17:09:817426]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : <Lighting Layout>: Proposing JSON name : no_lights.lighting_config.json from original name : no_lights| This file does not exist.
# [21:17:09:817437]:[Metadata] LightLayoutAttributesManager.cpp(36)::createObject : File (no_lights) not found, so new default light layout attributes created and registered.
# [21:17:09:817445]:[Sim] Simulator.cpp(237)::createSceneInstance : Navmesh file location in scene instance : data/scene_datasets/habitat-test-scenes/skokloster-castle.navmesh
# [21:17:09:817455]:[Sim] Simulator.cpp(242)::createSceneInstance : Loading navmesh from data/scene_datasets/habitat-test-scenes/skokloster-castle.navmesh
# [21:17:09:817468]:[Nav] PathFinder.cpp(852)::loadNavMesh : NavMeshSettings aren't present, guessing that they are the default
# [21:17:09:817532]:[Sim] Simulator.cpp(244)::createSceneInstance : Navmesh Loaded.
# [21:17:09:817545]:[Scene] SceneGraph.h(85)::createDrawableGroup : Created DrawableGroup: 
# [21:17:09:817552]:[Assets] ResourceManager.cpp(297)::loadSemanticSceneDescriptor : SceneInstance : data/scene_datasets/habitat-test-scenes/skokloster-castle.glb proposed Semantic Scene Descriptor filename : data/scene_datasets/habitat-test-scenes/skokloster-castle.scn
# [21:17:09:817560]:[Assets] ResourceManager.cpp(344)::loadSemanticSceneDescriptor : SSD File Naming Issue! Neither SceneInstanceAttributes-provided name : data/scene_datasets/habitat-test-scenes/skokloster-castle.scn  nor constructed filename : data/scene_datasets/habitat-test-scenes/info_semantic.json exist on disk.
# [21:17:09:817579]:[Assets] ResourceManager.cpp(1652)::ConfigureImporterManagerGLExtensions : Importing Basis files as BC7.
# [21:17:09:817696]:[Sim] Simulator.cpp(302)::createSceneInstance : Using scene instance-specified Light key : - no_lights -
# [21:17:09:817708]:[Metadata] MetadataMediator.cpp(66)::setSimulatorConfiguration : Set new simulator config for scene/stage : data/scene_datasets/habitat-test-scenes/skokloster-castle.glb and dataset : default which is currently active dataset.
# [21:17:09:817723]:[Sim] Simulator.cpp(412)::instanceStageForSceneAttributes : Start to load stage named : data/scene_datasets/habitat-test-scenes/skokloster-castle.glb with render asset : data/scene_datasets/habitat-test-scenes/skokloster-castle.glb and collision asset : data/scene_datasets/habitat-test-scenes/skokloster-castle.glb
# [21:17:09:817752]:[Assets] ResourceManager.cpp(691)::createStageAssetInfosFromAttributes : Frame :{"up":[0,0,1],"front":[0,1,0],"origin":[0,0,0]} for render mesh named : data/scene_datasets/habitat-test-scenes/skokloster-castle.glb|{"up":[0,0,1],"front":[0,1,0],"origin":[0,0,0]} for semantic mesh named :  with type specified as semantic|Semantic Txtrs : False
# [21:17:09:817760]:[Assets] ResourceManager.cpp(507)::loadStage : Start load render asset data/scene_datasets/habitat-test-scenes/skokloster-castle.glb .
# [21:17:09:817765]:[Assets] ResourceManager.cpp(973)::loadStageInternal : Attempting to load stage data/scene_datasets/habitat-test-scenes/skokloster-castle.glb 
# [21:17:09:817779]:[Assets] ResourceManager.cpp(855)::loadRenderAsset : Loading general asset named: data/scene_datasets/habitat-test-scenes/skokloster-castle.glb
# [21:17:09:817785]:[Assets] ResourceManager.cpp(1652)::ConfigureImporterManagerGLExtensions : Importing Basis files as BC7.
# [21:17:10:458713]:[Assets] ResourceManager.cpp(2089)::loadMaterials : Building 2 materials for asset named 'skokloster-castle.glb' : 
# [21:17:10:458752]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 00:Flat.
# [21:17:10:458766]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 01:Flat.
# [21:17:10:516568]:[Sim] Simulator.cpp(439)::instanceStageForSceneAttributes : Successfully loaded stage named : data/scene_datasets/habitat-test-scenes/skokloster-castle.glb
# [21:17:10:516599]:[Sim] Simulator.cpp(471)::instanceStageForSceneAttributes : 
# ---
# The active scene does not contain semantic annotations : activeSemanticSceneID_ = 0  
# ---
# [21:17:10:516615]:[Sim] Simulator.cpp(205)::reconfigure : CreateSceneInstance success == true for active scene name : data/scene_datasets/habitat-test-scenes/skokloster-castle.glb  with renderer.
# 2024-03-04 21:17:10,517 Initializing task Nav-v0    

这其中有很多子问题如下：

1.1 如何设置配置文件中要配置的参数

办法一：直接在yaml配置文件中修改

该方法简单直接不需要举例

办法二：在代码中修改

通过habitat.get_config()获取到config变量后，通过config.defrost()之后便可以在代码中设置一些变量，设置完后通过config.freeze()冻结配置参数

config=habitat.get_config("/home/yons/LK/habitat-challenge/habitat-lab/configs/datasets/pointnav/habitat_test.yaml"
config.defrost()
config.DATASET.SPLIT = "val"
config.freeze()

1.2 habitat.get_config在哪里定义

在…/habitat-challenge/habitat-lab/habitat/config/default.py中定义。该文件中定义了:
1） get_config()方法
2）所有配置变量的默认值

2 关于动作

…/habitat-lab/habitat/sims/habitat_simulator/actions.py中定义了默认的动作：

class _DefaultHabitatSimActions(Enum):
    STOP = 0
    MOVE_FORWARD = 1
    TURN_LEFT = 2
    TURN_RIGHT = 3
    LOOK_UP = 4
    LOOK_DOWN = 5

要执行动作需要初始化环境，然后通过HabitatSimActions获取动作，然后通过env.step执行动作

from habitat.sims.habitat_simulator.actions import HabitatSimActions
env = habitat.Env( config=habitat.get_config("/home/yons/LK/habitat-challenge/habitat-lab/configs/datasets/pointnav/habitat_test.yaml")
observations = env.reset()
action = HabitatSimActions.MOVE_FORWARD
observations = env.step(action)

一个通过键盘与环境交互的示例代码（quick_start3.py文件）如下：

import habitat
from habitat.sims.habitat_simulator.actions import HabitatSimActions
import cv2

FORWARD_KEY="w"
LEFT_KEY="a"
RIGHT_KEY="d"
FINISH="f"
####下面是向上看和向下看的动作，并不是所有的agent都可以向上看和向下看
###一个问题是为什么没有后退的动作？
# LOOKUP_KEY="u"
# LOOKDOWN_KEY="j"

def transform_rgb_bgr(image):
    return image[:, :, [2, 1, 0]]

def example():
    #选择一个配置文件初始化环境（配置文件需要有什么参数？）
    env = habitat.Env(
        config=habitat.get_config("/home/yons/LK/habitat-challenge/habitat-lab/configs/datasets/pointnav/habitat_test.yaml")
    )

    print("Environment creation successful")
    observations = env.reset()  #

    
    print('type(observations):',type(observations))    #type(observations): <class 'habitat.core.simulator.Observations'>
    print('observations["rgb"].shape:',observations["rgb"].shape)  #observations["rgb"].shape: (480, 640, 3)

    # cv2.imshow("RGB", transform_rgb_bgr(observations["rgb"]))
    cv2.imshow("RGB", observations["rgb"])  ####不知道为啥要转化rgb信号的顺序，经过测试，不转换的确看着不自然。

    print("Agent stepping around inside environment.")

    count_steps = 0
    while not env.episode_over:
        keystroke = cv2.waitKey(0)  #等待用户输入，操纵agent左右转和前进，与环境进行交互

        if keystroke == ord(FORWARD_KEY):  #如果是敲击的我们在最上面定义的某个键，做对应的事情，并打印
            action = HabitatSimActions.MOVE_FORWARD
            print("action: FORWARD")
        elif keystroke == ord(LEFT_KEY):
            action = HabitatSimActions.TURN_LEFT
            print("action: LEFT")
        elif keystroke == ord(RIGHT_KEY):
            action = HabitatSimActions.TURN_RIGHT
            print("action: RIGHT")
        elif keystroke == ord(FINISH):
            action = HabitatSimActions.STOP
            print("action: FINISH")
        else:
            print("INVALID KEY")
            continue

        observations = env.step(action)
        ####env.step(action)里面的action是HabitatSimActions中定义的动作
        count_steps += 1
        # cv2.imshow("RGB", transform_rgb_bgr(observations["rgb"]))
        cv2.imshow("RGB", observations["rgb"])
        


    print("Episode finished after {} steps.".format(count_steps))

    if (
        action == HabitatSimActions.STOP
        # and observations["pointgoal_with_gps_compass"][0] < 0.2
    ):
        print("you successfully navigated to destination point")
    else:
        print("your navigation was unsuccessful")


if __name__ == "__main__":
    example()

3获取habitat-lab路径

通过habitat.utils.gym_definitions.base_dir获取

import habitat.utils.gym_definitions as habitat_gym
print('habitat_gym.base_dir:',habitat_gym.base_dir)

下面是一个例子：

import habitat.utils.gym_definitions as habitat_gym
print('habitat_gym.base_dir:',habitat_gym.base_dir)
import os

import gym
import gym.spaces as spaces
import numpy as np

import habitat
import habitat.utils.gym_definitions as habitat_gym
from habitat.core.embodied_task import Measure
from habitat.core.registry import registry
from habitat.core.simulator import Sensor, SensorTypes
from habitat.tasks.rearrange.rearrange_sensors import RearrangeReward
from habitat.tasks.rearrange.rearrange_task import RearrangeTask
from habitat.utils.render_wrapper import overlay_frame
from habitat.utils.visualizations.utils import observations_to_image
from habitat_sim.utils import viz_utils as vut

# Quiet the Habitat simulator logging
os.environ["MAGNUM_LOG"] = "quiet"
os.environ["HABITAT_SIM_LOG"] = "quiet"


def insert_render_options(config):
    # Added settings to make rendering higher resolution for better visualization
    config.defrost()
    config.SIMULATOR.THIRD_RGB_SENSOR.WIDTH = 512
    config.SIMULATOR.THIRD_RGB_SENSOR.HEIGHT = 512
    config.SIMULATOR.CONCUR_RENDER = False
    config.SIMULATOR.AGENT_0.SENSORS.append("THIRD_RGB_SENSOR")
    config.freeze()
    return config


import importlib

# If the import block fails due to an error like "'PIL.TiffTags' has no attribute
# 'IFD'", then restart the Colab runtime instance and rerun this cell and the previous cell.
import PIL

importlib.reload(PIL.TiffTags)  # To potentially avoid PIL problem

print('habitat_gym.base_dir:',habitat_gym.base_dir)

##################用habitat自己的方式###########################
with habitat.Env(
    config=insert_render_options(
        habitat.get_config(
            os.path.join(
                habitat_gym.base_dir,
                "configs/tasks/rearrange/pick.yaml",
            )
        )
    )
) as env:
    observations = env.reset()  # noqa: F841

    print("Agent acting inside environment.")
    count_steps = 0
    # To save the video
    video_file_path = "data/example_interact.mp4"
    video_writer = vut.get_fast_video_writer(video_file_path, fps=30)

    while not env.episode_over:
        observations = env.step(env.action_space.sample())  # noqa: F841,随机动作
        info = env.get_metrics()

        render_obs = observations_to_image(observations, info)
        render_obs = overlay_frame(render_obs, info)

        video_writer.append_data(render_obs)

        count_steps += 1
    print("Episode finished after {} steps.".format(count_steps))

    video_writer.close()
    if vut.is_notebook():
        #此处并没有播放
        vut.display_video(video_file_path)