- 官方链接:http://ai2thor.allenai.org/tutorials/installation
-
安装:
- pip install ai2thor
- Before running the below code, make sure X server with OpenGL is running, and the OpenGL extensions have been installed for your graphics card.
-
import ai2thor.controller controller = ai2thor.controller.Controller() controller.start()
-
-
Concepts:
- 动作:离散的
- Object Visibility:定义了一个object距离摄像机的distance阈值。当agent和object的距离小于这个特定的阈值时,agent可以和object交互。
- Receptacle :可以包含一个object的object,比如水槽、冰箱、橱柜和桌面。
-
Scenes:
- version1.0有120个scenes,主要分为四个大类:厨房、起居室、卧室和浴室。每个房间有可进行操作的obeject。在初始化的时候object可以放在不同的容器,从而在一个相同的场景中有多种配置方式。
-
Kitchens (30 scenes, FloorPlan1 - FloorPlan30)
-
Living Rooms (30 scenes, FloorPlan201 - FloorPlan230)
-
Bedrooms (30 scenes, FloorPlan301 - FloorPlan330)
-
Bathrooms (30 scenes, FloorPlan401 - FloorPlan430)
-
Actions:
- action被定义在
unity/Assets/Scripts/DiscreteRemoteFPSAgentController.cs 当中。
- 下面折页定义了可操作的对象的几种状态 http://ai2thor.allenai.org/tutorials/object-types
- 在下面内容开始前,假设你已经运行了下述命令:
-
import ai2thor.controller controller = ai2thor.controller.Controller() controller.start() # can be any one of the scenes FloorPlan### controller.reset('FloorPlan28') controller.step(dict(action='Initialize', gridSize=0.25))
-
- 初始化:重置scene(例如定义gridSize)的时候必须调用“初始化”
-
controller.step(dict(action='Initialize', gridSize=0.25))
- 初始化的时候还可以定义一些其他的参数:比如
- gridSize(agent移动的网格尺寸) / renderDepthImage(是否使用深度图像) / renderClassImage(是否启用分类图像) / renderObjectImage(是否启用object segmentation image) / visibilityDistance(定义可视距离) / cameraY(相机高度) / fieldOfView(视椎)
- 详情见: http://ai2thor.allenai.org/tutorials/actions
-
- 随机化:用来打乱场景同事增加场景中的object。
-
controller.random_initialize( random_seed=None, randomize_open=False, unique_object_types=False, exclude_receptacle_object_pairs=[], max_num_repeats=1, remove_prob=0.5)
- 参数见链接 : http://ai2thor.allenai.org/tutorials/actions
-
- 前后左右移动(视角朝向不变) :
-
controller.step(dict(action='MoveAhead')) # 前后左右 'MoveAhead' 'MoveBack' 'MoveRight' 'MoveLeft'
-
- 向上看向下看:(30度)
-
controller.step(dict(action='LookUp')) # 'LookUp' 'LookDown'
-
- 右转&左转 (90度):
-
controller.step(dict(action='RotateRight')) # 'RotateRight' 'RotateLeft'
-
- 打开/关闭/捡起/放下 object:
-
# obejct必须在阈值范围内 controller.step(dict(action='OpenObject', objectId="Fridge|0.25|0.75|0.05")) # 'OpenObject' 'CloseObject'
-
# 捡起放入仓库 从仓库拿出放在可视范围内 仓库容量为1 controller.step(dict(action='PickupObject', objectId="Mug|0.25|-0.27|0.95")) # 'PickupObject' "PutObject"
-
- 移动agent:
-
controller.step(dict(action='Teleport', x=0.999, y=1.01, z=-0.3541)) # x y z ???
-
-
TeleportFull:
-
controller.step(dict(action='TeleportFull', x=0.999, y=1.01, z=-0.3541, rotation=90.0, horizon=30.0))
-
-
Failures:
-
action可能会失败。比如:agent移动过程中发生碰撞 / object的可见性带来的捡取放置操作的错误
-
- action被定义在
-
Event Metadata:
- 每次调用
controller.step() 都会返回一个event,
包含了环境以及环境中的object的信息。 -
Event :
-
# return object from controller.step() event = controller.step(dict(action=<SOME ACTION>))
-
event的具体属性: http://ai2thor.allenai.org/tutorials/event-metadata
-
-
Object:
-
obejct的具体属性: http://ai2thor.allenai.org/tutorials/event-metadata
-
- 每次调用
-
Object Types:
- 87种可交互的object,以及每个object的
-
Receptacle Object Types:
- 容器种类 & 每种容器可以包含的object的种类: http://ai2thor.allenai.org/tutorials/receptacle-object-types
-
Examples:
- 例子:把agent前移一步,同时返回对应的image和metadata。
-
import ai2thor.controller controller = ai2thor.controller.Controller() controller.start() # Kitchens: FloorPlan1 - FloorPlan30 # Living rooms: FloorPlan201 - FloorPlan230 # Bedrooms: FloorPlan301 - FloorPlan330 # Bathrooms: FloorPLan401 - FloorPlan430 controller.reset('FloorPlan28') controller.step(dict(action='Initialize', gridSize=0.25)) event = controller.step(dict(action='MoveAhead')) # Numpy Array - shape (width, height, channels), channels are in RGB order event.frame # Numpy Array in BGR order suitable for use with OpenCV event.cv2image() # current metadata dictionary that includes the state of the scene event.metadata
-
- 例子:调用复杂的动作——捡起杯子,打开微波炉,把杯子放进微波炉
- 先移动到能看见object的位置,然后执行动作。
-
import ai2thor.controller controller = ai2thor.controller.Controller() controller.start() # 选择要运行的scene controller.reset('FloorPlan28') # 初始化 同时配置场景参数 controller.step(dict(action='Initialize', gridSize=0.25)) # 让agent移动到特定坐标 controller.step(dict(action='Teleport', x=-1.25, y=1.00, z=-1.5)) controller.step(dict(action='LookDown')) # 向下看 event = controller.step(dict(action='Rotate', rotation=90)) # 有没有rotate这个动作? # In FloorPlan28, the agent should now be looking at a mug # 捡起杯子 for o in event.metadata['objects']: # 遍历视野中所有的object # 如果发现Mug if o['visible'] and o['pickupable'] and o['objectType'] == 'Mug': event = controller.step(dict(action='PickupObject', objectId=o['objectId']), raise_for_failure=True) # 执行捡起Mug mug_object_id = o['objectId'] # Mug的id break # the agent now has the Mug in its inventory # to put it into the Microwave, we need to open the microwave first # 打开微波炉 event = controller.step(dict(action='LookUp')) for o in event.metadata['objects']: # 找出 视野中 可见的 可打开的 微波炉 if o['visible'] and o['openable'] and o['objectType'] == 'Microwave': # 打开 微波炉 event = controller.step(dict(action='OpenObject', objectId=o['objectId']), raise_for_failure=True) # 同时提取微波炉的id receptacle_object_id = o['objectId'] break # agent右移 event = controller.step(dict(action='MoveRight'), raise_for_failure=True) # 执行“放置”动作 容器是微波炉 被放置的东西是杯子 event = controller.step(dict( action='PutObject', receptacleObjectId=receptacle_object_id, objectId=mug_object_id), raise_for_failure=True) # close the microwave # 关闭微波炉 event = controller.step(dict( action='CloseObject', objectId=receptacle_object_id), raise_for_failure=True)
- 例子:多线程:
- 在多线程中运行agent的多个instance:
-
import threading import time import ai2thor.controller thread_count = 8 def run(): env = ai2thor.controller.Controller() env.start() # 100 is an arbritary number for _ in range(100): t_start = time.time() env.reset('FloorPlan1') env.step({'action' : 'Initialize', 'gridSize' : 0.25}) print('init time', time.time() - t_start) t_start_total = time.time() for _ in range(10): env.step({'action' : 'MoveAhead'}) env.step({'action' : 'RotateRight'}) total_time = time.time() - t_start_total print('total time', total_time, 20 / total_time, 'fps') threads = [threading.Thread(target=run) for _ in range(thread_count)] for t in threads: t.daemon = True t.start() time.sleep(1) for t in threads: # calling join() in a loop/timeout to allow for Python 2.7 # to be interrupted with SIGINT while t.isAlive(): t.join(1) print('done')
- 例子:把agent前移一步,同时返回对应的image和metadata。