kubernetes对接cinder创建pv报错：Failed to provision volume with StorageClass standard

1、环境信息

这个环境用于验证kubernetes使用cinder作为provider提供pv
kubernetes本身运行在openstack集群的虚拟机中，通过router打通和openstack管理面的链接

1.1、openstack 集群虚拟机列表

$ nova list
+--------------------------------------+-----------+--------+------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------+
| ID                                   | Name      | Status | Task State | Power State | Networks                                                                                                                          |
+--------------------------------------+-----------+--------+------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------+
| ccec3aee-15e5-4710-8e4c-5a4db6826022 | k8s-m1    | ACTIVE | -          | Running     | clu_mgt=180.0.0.3; clu_net2=172.0.2.139, 192.168.32.198                                                                           |
| 9eacafc1-8ce7-4073-a8c2-73270cffae31 | k8s-m2    | ACTIVE | -          | Running     | clu_mgt=180.0.0.206; clu_net2=172.0.2.146, 192.168.32.186                                                                         |
| 8128acd7-fd05-4367-985d-a36023f87de8 | k8s-m3    | ACTIVE | -          | Running     | clu_mgt=180.0.0.8; clu_net2=172.0.2.234, 192.168.32.159                                                                                                                           

1.2、kubernetes节点列表

# kubectl get nodes -o wide
NAME               STATUS     ROLES    AGE     VERSION                                INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION           CONTAINER-RUNTIME
k8s-m1             Ready      master   4d22h   v1.17.0                                172.0.2.139   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64   docker://19.3.6
k8s-m2             Ready      master   4d22h   v1.17.0                                172.0.2.146   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64   docker://19.3.6
k8s-m3             Ready      master   4d22h   v1.17.0                                172.0.2.234   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64   docker://19.3.6

1.3、cloud-config文件

其中：192.168.32.90为openstack internal api地址，k8s虚拟机通过float ip与该地址对接

# cat ../cloud-config
[Global]
username=admin
password=keystone
auth-url=http://192.168.32.90:5000/v3
tenant-id=d57f7ea9b3b94d9a87321807ec453ca8
domain-id=default

2、创建pv过程

创建过程同
k8s+virtual-kubelet部署带动态pv的pod到openstack-zun

3、创建pvc出现问题

3.1、pvc处于pending状态

# kubectl get pvc
NAME         STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
cinder-pvc   Pending    

3.2、查看pvc详细信息，显示和这个地址不通：172.255.0.113

# kubectl describe pvc cinder-pvc
Name:          cinder-pvc
Namespace:     default
StorageClass:  standard
Status:        Pending
Volume:
Labels:        <none>
Annotations:   kubectl.kubernetes.io/last-applied-configuration:
                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"cinder-pvc","namespace":"default"},"spec":{"accessM...
               volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/cinder
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Mounted By:    <none>
Events:
  Type     Reason              Age               From                         Message
  ----     ------              ----              ----                         -------
  Warning  ProvisioningFailed  8s (x2 over 16s)  persistentvolume-controller  Failed to provision volume with StorageClass "standard": failed to create a 1 GB volume: Post https://172.255.0.113:8776/v3/d57f7ea9b3b94d9a87321807ec453ca8/volumes: dial tcp 172.255.0.113:8776: connect: no route to host

3.3、172.255.0.113这个地址是openstack的public api地址

但是我们需要的是 internal api地址

$ openstack endpoint list | grep keystone
| 0fc44a1959dd4888a9a3125c9199dc2c | RegionOne | keystone      | identity        | True    | internal  | https://192.168.32.90:5000/v3                     |
| 3e39dce895e64d2794c022eb3d6cb85e | RegionOne | keystone      | identity        | True    | public    | https://172.255.0.113:5000/v3                     |
| 427000db07dd4feda52c04d121e24ca7 | RegionOne | keystone      | identity        | True    | admin     | https://192.168.166.113:35357/v3                    |

4、解决问题

4.1、初步分析

pvc详细信息中，https://172.255.0.113:8776/v3/d57f7ea9b3b94d9a87321807ec453ca8/volumes
是openstack cinder的public API endpoint，该endpoint是通过调用keystone endpoint查询接口获取

keystone endpoint查询接口有个参数，可以指定获取public、internal还是admin的endpoint

推测：kubernetes默认获取的是openstack public endpoint

4.2、分析kubernetes获取openstack endpoint代码

这里就略去代码分析过程直接贴最终代码了
代码路径：

gophercloud\openstack\client.go
gophercloud\endpoint_search.go

调用链

AuthenticatedClient->Authenticate->v3auth->NewIdentityV3->ApplyDefaults

ApplyDefaults代码

func (eo *EndpointOpts) ApplyDefaults(t string) {
	if eo.Type == "" {
		eo.Type = t
	}
	if eo.Availability == "" {
		eo.Availability = AvailabilityPublic
	}
}

可以看到，gophercloud在ApplyDefaults中将默认值设置为了public

4.3、为什么默认获取public api

从应用场景上来看，k8s要对接IAAS集群，一般情况下都是通过外部对接，所以默认设置获取public api也是合理的
而我的验证环境恰恰是将k8s部署在了openstack集群上，且需要对接openstack集群，走的是internal api，属于少数应用场景

4.4、解决问题

找到问题源头了，解决就比较简单了
可以直接修改代码如下：

func (eo *EndpointOpts) ApplyDefaults(t string) {
	if eo.Type == "" {
		eo.Type = t
	}
	if eo.Availability == "" {
		eo.Availability = AvailabilityInternal
	}
}

当然，更好的解决方法是将该变量作为可配参数引出来