Open3D从RGB图与depth图产生RGB-D点云（采坑记录）

Open3D

Open3D是一个开源库，它支持处理3D数据的软件的快速开发。Open3D前端在C++和Python中公开了一组精心选择的数据结构和算法。后端经过高度优化，并设置为并行化。
前期也用了一段时间的PCL，但是C++真的啃不动，转而寻求python版的点云处理库，就找到了Open3D。Open3D国内的相关资源还是比较少的，因此我准备在学习过程中，在CSDN上记录遇到的问题以及解决方法，方便后续查找。

下面代码可以利用Open3D生成RGB-D数据

import open3d as o3d
import numpy as np
from matplotlib import pyplot as plt

color_raw = o3d.io.read_image("F:/Depth_image_C++/open_and_save/open_and_save/3Dprintcarrot/rgb/(1).png")
depth_raw = o3d.io.read_image("F:/Depth_image_C++/open_and_save/open_and_save/3Dprintcarrot/depth/(1).png")
rgbd_image = o3d.geometry.RGBDImage.create_from_color_and_depth(color_raw, depth_raw, convert_rgb_to_intensity=False)
print(rgbd_image)

plt.subplot(1, 2, 1)
plt.title('Redwood grayscale image')
plt.imshow(rgbd_image.color)
plt.subplot(1, 2, 2)
plt.title('Redwood depth image')
plt.imshow(rgbd_image.depth)
plt.show()

inter = o3d.camera.PinholeCameraIntrinsic()
inter.set_intrinsics(1280, 720, 599.795593, 599.633118, 645.792786, 372.238983)
pcd = o3d.geometry.PointCloud().create_from_rgbd_image(
    rgbd_image, inter)

# Flip it, otherwise the pointcloud will be upside down
pcd.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])
o3d.visualization.draw_geometries([pcd])

遇到的问题

因为我使用的是Azure kinect SDK获取的rgb图与深度图，所以相机的内参需要自己设定下。
内参获取我看相关资料可以在Azure-Kinect-Sensor-SDK/examples/calibration at develop · microsoft/Azure-Kinect-Sensor-SDK · 找到。只需在Visual Studio 2017中加载CmakeList.txt并进行构建，将生成Calibration_info.exe。连接并执行Azure Kinect后，输出如下。

C:\GitHub\Azure-Kinect-Sensor-SDK\build\Win-x64-Release-Ninja\bin>calibration_info.exe  > out.txt

下面是Azure kinect sdk的一些内参，供大家参考
图像尺寸为640×576的内参，格式为K4A_DEPTH_MODE_NFOV_UNBINNED

Found 1 connected devices:

===== Device 0: {DeviceID:XXXXXX} =====
resolution width: 640
resolution height: 576
principal point x: 309.301086
principal point y: 330.656738
focal length x: 505.119751
focal length y: 505.163605
radial distortion coefficients:
k1: 3.063722
k2: 1.988396
k3: 0.102371
k4: 3.397454
k5: 2.984188
k6: 0.544097
center of distortion in Z=1 plane, x: 0.000000
center of distortion in Z=1 plane, y: 0.000000
tangential distortion coefficient x: -0.000098
tangential distortion coefficient y: 0.000048
metric radius: 0.000000

也可以进行自动校准

auto calib = calibration.depth_camera_calibration

和

auto calib = calibration.color_camera_calibration

这是1080p的内参

Found 1 connected devices:

===== Device 0: {DeviceID:XXXXXX} =====
resolution width: 1920
resolution height: 1080
principal point x: 968.939209
principal point y: 558.608459
focal length x: 899.693420
focal length y: 899.449646
radial distortion coefficients:
k1: 0.678679
k2: -2.779486
k3: 1.569404
k4: 0.554819
k5: -2.610379
k6: 1.500811
center of distortion in Z=1 plane, x: 0.000000
center of distortion in Z=1 plane, y: 0.000000
tangential distortion coefficient x: 0.000476
tangential distortion coefficient y: 0.000104
metric radius: 0.000000

这是720p内参

Found 1 connected devices:

===== Device 0: {DeviceID:XXXXXX} =====
resolution width: 1280
resolution height: 720
principal point x: 645.792786
principal point y: 372.238983
focal length x: 599.795593
focal length y: 599.633118
radial distortion coefficients:
k1: 0.678679
k2: -2.779486
k3: 1.569404
k4: 0.554819
k5: -2.610379
k6: 1.500811
center of distortion in Z=1 plane, x: 0.000000
center of distortion in Z=1 plane, y: 0.000000
tangential distortion coefficient x: 0.000476
tangential distortion coefficient y: 0.000104
metric radius: 0.000000

有了内参，怎么用Open3D读取内参并用于点云生成中

有两种方法
第一种

inter = o3d.camera.PinholeCameraIntrinsic()
inter.set_intrinsics(1280, 720, 599.795593, 599.633118, 645.792786, 372.238983)

这里要注意的是inter = o3d.camera.PinholeCameraIntrinsic()，括号一定不能少，否则会报错

inter.set_intrinsics(1280, 720, 599.795593, 599.633118, 645.792786, 372.238983)
TypeError: set_intrinsics(): incompatible function arguments. The following argument types are supported:
    1. (self: open3d.cpu.pybind.camera.PinholeCameraIntrinsic, width: int, height: int, fx: float, fy: float, cx: float, cy: float) -> None

Invoked with: 1280, 720, 599.795593, 599.633118, 645.792786, 372.238983

第二种

inter= o3d.camera.PinholeCameraIntrinsic(1280, 720, 599.795593, 599.633118, 645.792786, 372.238983)

保存、读取内参

o3d.io.write_pinhole_camera_intrinsic('./file.json', inter)
pinholeCamera = o3d.io.read_pinhole_camera_intrinsic('./file.json')

最后放几张处理过程中的图片

最后的点云图有点失真，可能是因为我的彩色图是RGBA四通道图，转成RGB三通道图就可以正常显示了