《kubernetes-1.8.0》11-addon-Harbor

《kubernetes-1.8.0》11-addon-Harbor

《kubernetes 1.8.0 测试环境安装部署》

时间：2017-11-27

一、环境准备

docker版本：1.17
docker-compose版本：1.17

安装docker-compose

curl -L https://github.com/docker/compose/releases/download/1.17.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

从harbor 发布页面下载最新的harbor offline安装包（当前版本v1.3.0-rc1）：

wget  --continue http://harbor.orientsoft.cn/harbor-v1.3.0-rc1/harbor-offline-installer-v1.3.0-rc1.tgz

1.3.0开始建议去镜像站下载，嗷嗷快

解压tarball,并加载镜像：

$ tar -zxvf harbor-offline-installer-v1.3.0-rc1.tgz 
$ cd harbor
$ docker load -i harbor.v1.3.0-rc1.tar.gz

创建harbor所需证书：

签名请求：

$ cat > harbor-csr.json <<EOF
{
  "CN": "harbor",
  "hosts": [
    "127.0.0.1",
    "172.18.169.134"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

• "hosts"：本例中改字段使用ip，实际测试环境中使用的是域名，方便外网使用。

创建证书和秘钥：

由于是在node节点上部署，使用k8s根ca做签名，因此沿用原有k8s ca根证书，重新创建或者从master节点上拷贝k8s-gencert.json（配置文件） 以及ca证书和key（签完后删除）

 scp /etc/kubernetes/ssl/k8s-root-ca-key.pem root@172.18.169.134:/etc/kubernetes/ssl/

cat > /etc/kubernetes/ssl/k8s-gencert.json  << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}
EOF

创建证书秘钥：

$ cfssl gencert -ca=/etc/kubernetes/ssl/k8s-root-ca.pem \
  -ca-key=/etc/kubernetes/ssl/k8s-root-ca-key.pem \
  -config=/etc/kubernetes/ssl/k8s-gencert.json \
  -profile=kubernetes harbor-csr.json | cfssljson -bare harbor
$ ls harbor*
harbor.csr  harbor-csr.json  harbor-key.pem harbor.pem
$ sudo mkdir -p /etc/harbor/ssl
$ sudo mv harbor*.pem /etc/harbor/ssl
$ rm /etc/kubernetes/ssl/k8s-root-ca-key.pem 
$ rm harbor.csr  harbor-csr.json

二、部署harbor

修改harbor.cfg 文件

$ vi harbor.cfg

修改一下参数

hostname = 172.18.169.134 
ui_url_protocol = https 
ssl_cert = /etc/harbor/ssl/harbor.pem
ssl_cert_key = /etc/harbor/ssl/harbor-key.pem

加载和启动 harbor 镜像

[root@node-134 harbor]# vi harbor.cfg 
## Configuration file of Harbor

#The IP address or hostname to access admin UI and registry service.
#DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname = 172.18.169.134

#The protocol for accessing the UI and token/notification service, by default it is http.
#It can be set to https if ssl is enabled on nginx.
ui_url_protocol = https

#The password for the root user of mysql db, change this before any production use.
db_password = root123

#Maximum number of job workers in job service
max_job_workers = 3

#Determine whether or not to generate certificate for the registry's token.
#If the value is on, the prepare script creates new root cert and private key
#for generating token to access the registry. If the value is off the default key/cert will be used.
#This flag also controls the creation of the notary signer's cert.
customize_crt = on

#The path of cert and key files for nginx, they are applied only the protocol is set to https
ssl_cert = /etc/harbor/ssl/harbor.pem
ssl_cert_key = /etc/harbor/ssl/harbor-key.pem

#The path of secretkey storage
secretkey_path = /data

#Admiral's url, comment this attribute, or set its value to NA when Harbor is standalone
admiral_url = NA

#The password of the Clair's postgres database, only effective when Harbor is deployed with Clair.
#Please update it before deployment, subsequent update will cause Clair's API server and Harbor unable to access Clair's database.
clair_db_password = password

"harbor.cfg" 125L, 5021C written
[root@node-134 harbor]# ./install.sh 

[Step 0]: checking installation environment ...

Note: docker version: 17.09.0

Note: docker-compose version: 1.17.1

[Step 1]: loading Harbor images ...
Loaded image: vmware/harbor-log:v1.3.0-rc1
Loaded image: vmware/harbor-db:v1.3.0-rc1
Loaded image: vmware/harbor-jobservice:v1.3.0-rc1
Loaded image: vmware/notary-photon:server-0.5.1
Loaded image: vmware/notary-photon:signer-0.5.1
Loaded image: vmware/harbor-db-migrator:1.3
Loaded image: vmware/harbor-adminserver:v1.3.0-rc1
Loaded image: vmware/photon:1.0
Loaded image: vmware/mariadb-photon:10.2.10
Loaded image: vmware/harbor-ui:v1.3.0-rc1
Loaded image: vmware/nginx-photon:1.11.13
Loaded image: vmware/registry:2.6.2-photon
Loaded image: vmware/clair:v2.0.1-photon
Loaded image: vmware/postgresql:9.6.5-photon


[Step 2]: preparing environment ...
Generated and saved secret to file: /data/secretkey
Generated configuration file: ./common/config/nginx/nginx.conf
Generated configuration file: ./common/config/adminserver/env
Generated configuration file: ./common/config/ui/env
Generated configuration file: ./common/config/registry/config.yml
Generated configuration file: ./common/config/db/env
Generated configuration file: ./common/config/jobservice/env
Generated configuration file: ./common/config/log/logrotate.conf
Generated configuration file: ./common/config/jobservice/app.conf
Generated configuration file: ./common/config/ui/app.conf
Generated certificate, key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt
The configuration files are ready, please use docker-compose to start the service.


[Step 3]: checking existing instance of Harbor ...


[Step 4]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... 
Creating harbor-log ... done
Creating harbor-adminserver ... 
Creating registry ... 
Creating harbor-db ... 
Creating registry
Creating harbor-db
Creating registry ... done
Creating harbor-ui ... 
Creating harbor-ui ... done
Creating harbor-jobservice ... 
Creating nginx ... 
Creating harbor-jobservice
Creating nginx ... done

✔ ----Harbor has been installed and started successfully.----

Now you should be able to visit the admin portal at https://172.18.169.134. 
For more details, please visit https://github.com/vmware/harbor .

三、测试访问

浏览器访问 https://${NODE_IP}，示例的是 https://172.18.169.134

用账号 admin 和 harbor.cfg 配置文件中的默认密码 Harbor12345 登陆系统：

harbor 运行时产生的文件、目录：

$ # 日志目录
$ ls /var/log/harbor/
adminserver.log  jobservice.log  mysql.log  proxy.log  registry.log  ui.log
$ # 数据目录，包括数据库、镜像仓库
$ ls /data/
ca_download  config  database  job_logs registry  secretkey

docker 客户端访问：

将签署 harbor 证书的 CA 证书拷贝到 /etc/docker/certs.d/172.18.169.134 目录下

$ sudo mkdir -p /etc/docker/certs.d/172.18.169.134
$ sudo cp /etc/kubernetes/ssl/k8s-root-ca.pem /etc/docker/certs.d/172.18.169.134/ca.crt

测试登陆：

[root@node-134 harbor]# docker login 172.18.169.134
Username: admin
Password: 
Login Succeeded

认证信息自动保存到 ~/.docker/config.json 文件。

测试上传镜像：

web界面创建项目-elk：

查看ELK相关镜像：

[root@node-131 ~]# docker images
REPOSITORY                                                       TAG                 IMAGE ID            CREATED             SIZE
...
gcr.io/google-containers/elasticsearch                           v5.6.4              856b33b5decc        4 days ago          877MB
docker.elastic.co/kibana/kibana                                  5.6.4               ec9ec2836846        2 weeks ago         675MB
alpine                                                           3.6                 053cde6e8953        3 weeks ago         3.96MB
gcr.io/google-containers/fluentd-elasticsearch                   v2.0.2              38ec68ca7d24        3 weeks ago         135MB
...

给这些镜像贴上tag（格式：仓库地址/项目/镜像名:版本）：

docker tag gcr.io/google-containers/elasticsearch:v5.6.4 \
172.18.169.134/elk/elasticsearch:v5.6.4

查看一下：

[root@node-131 ~]# docker images
REPOSITORY                                                       TAG                 IMAGE ID            CREATED             SIZE
...
172.18.169.134/elk/elasticsearch                                 v5.6.4              856b33b5decc        4 days ago          877MB
...

上传镜像：

[root@node-131 ~]# docker push 172.18.169.134/elk/elasticsearch:v5.6.4
The push refers to a repository [172.18.169.134/elk/elasticsearch]
7fb277b4b1c2: Pushed 
c0fb0737c000: Pushed 
b5688bac09d0: Pushed 
e067b2830979: Pushed 
415c63c897f0: Pushed 
5efd536b792b: Pushed 
7bc10b7b826b: Pushed 
fb4c22ecaccd: Pushed 
bfda58dbf7bc: Pushed 
f40e9cb1a305: Pushed 
8b1feb2e5b7d: Pushed 
c177774cd13d: Pushed 
500b2871701f: Pushed 
v5.6.4: digest: sha256:e2386ddc70a6f02770e05967e39f9a9a0b0346875f91aefe7c04c79577f89d2b size: 3041

web界面查看：

另一台主机测试下载：

[root@node-132 ~]# docker login 172.18.169.134
Username: admin
Password: 
Login Succeeded
[root@node-132 ~]# docker pull 172.18.169.134/elk/elasticsearch:v5.6.4
v5.6.4: Pulling from elk/elasticsearch
Digest: sha256:e2386ddc70a6f02770e05967e39f9a9a0b0346875f91aefe7c04c79577f89d2b
Status: Downloaded newer image for 172.18.169.134/elk/elasticsearch:v5.6.4

[root@node-132 ~]# docker images
REPOSITORY                                                       TAG                 IMAGE ID            CREATED             SIZE
...
172.18.169.134/elk/elasticsearch                                 v5.6.4              856b33b5decc        4 days ago          877MB
...

至此harbor搭建完成

本系列其他内容：

• 01-环境准备

• 02-etcd群集搭建

• 03-kubectl管理工具

• 04-master搭建

• 05-node节点搭建

• 06-addon-calico

• 07-addon-kubedns

• 08-addon-dashboard

• 09-addon-kube-prometheus

• 10-addon-EFK

• 11-addon-Harbor

• 12-addon-ingress-nginx

• 13-addon-traefik

参考资料：

https://github.com/opsnull/follow-me-install-kubernetes-cluster/blob/master/13-部署harbor私有仓库.md

https://github.com/vmware/harbor