shell中获取Harbor中所有的镜像列表

福海鑫森

已于 2022-07-11 14:36:48 修改

阅读量1.5k

点赞数

分类专栏： devops 文章标签： harbor

于 2017-04-18 17:58:37 首次发布

原文链接：https://blog.51cto.com/lidabai/5217350

版权

devops 专栏收录该内容

21 篇文章 1 订阅

订阅专栏

我们要查询Harbor中某个项目下某个镜像是否存在，需要登录Harbor UI管理界面，然后进入到项目（Project）,再到搜索框输入镜像的名称来查找镜像在不在。

那么，如果只知道镜像的名称，不知道镜像在哪个项目（Project）呢？你是每个项目下都去搜索么？

那么！有什么方法可以简化这些操作呢？即：

不需要登录Harbor就可以看到Harbor里面所有的镜像；
仅提供镜像名称就可以看到镜像在哪个项目；
仅知道镜像名称就可以判断Harbor镜像仓库中是否存在这个镜像，从而判断上传的镜像是否上传成功。
我在工作中常遇到开发的同事的问题：

我上传一个镜像，帮忙看上传成功没？
我刚发布一个服务，构建的时候报错一直找不到镜像，这是什么原因？
我之前上传好几个版本的镜像，怎么知道都有哪些？

解决方案
刚才描述了我在维护Harbor中遇到的一些业务问题，那有什么方法来优化这些问题呢？来减轻我的维护难度。

Harbor提供了良好的API支持，我何不利用这一点，通过调用API来获取Harbor中所有项目下（Project）所有的镜像，以及该镜像下所有的版本，生成一个镜像清单，再将该清单放到一个文件中，查看的时候仅需要grep命令过滤下就根据镜像名称来查到我想要的信息呢？

具体实现
因为Harbor有两个版本的API，即1.0版本和2.0版本，不同的API调用方法存在一定的区别。

本处我就以shell脚本的方式来实现，Python和Go版本的可自行参考。

1版本APIshell脚本实现
适用于Harbor v2.0以下的版本。1）编写shell脚本
[root@Harbor_backup]# vim Harbor-image-list-100.sh

#!/bin/bash

#镜像清单文件，将获取到的镜像信息存到该文件中
File=harbor-images-`date '+%Y-%m-%d'`.txt

## 定义Harbor连接地址,这里需要改为你们自己的Harbor地址
Address=http://192.168.2.250:443

## 定义连接Harbor的用户名和密码（因为是获取全部的镜像，只有admin用户才有该权限）
Hamin=admin:Harbor12345

## 获取Harbor中有哪些项目（Project）
Project_List=$(curl -u "$Hamin"  -X GET  $Address/api/projects  -H "Content-Type: application/json"   | grep name | awk '/"name": /' | awk -F '"' '{print $4}')

for Project in $Project_List;do
  # 循环获取每个项目下所有的镜像
 Image_Names=$(curl -u "$Hamin"  -X GET $Address/api/search?q=$Project -H "Content-Type: application/json" | grep "repository_name" | awk -F "\"" '{print $4}')
    for Image in $Image_Names;do
    # 循环获取每个镜像所有的标签（版本）
    Image_Tags=$(curl -u "$Hamin"  -X GET  $Address/api/repositories/$Image/tags -H "Content-Type: application/json" | awk '/"name": /' | awk -F '"' '{print $4}')
        for Tag in $Image_Tags;do
        # 将获取到的镜像完整路径存档到镜像清单文件
        echo "$Address/$Image:$Tag" | grep -v Base | grep -v Image | grep -v CentOS >>  $File
        done
    done
done

2）执行脚本
[root@harbor-conso ~]# chmod +x Harbor-image-listk-100.sh
[root@harbor-conso ~]# sh Harbor-image-list-100.sh
...
先让子弹飞会儿，等一哈哈！

3）清单文件查看镜像
[root@harbor-conso ~]# cat harbor-images-2022-04-18.txt c| tail -10
http://192.168.2.250:443/lihonggang-25655/eeeee:rrrrr-20211207201732
http://192.168.2.250:443/lihonggang-25655/wwww:aaaa-20211207202110
http://192.168.2.250:443/lihonggang-25655/wwww:aaaa-20211207202855
http://192.168.2.250:443/gx-eop-es-30254/gx-eop-es:meng-v1.0.5-20220414154335
http://192.168.2.250:443/gx-eop-es-30254/gx-eop-es:meng-v1.0.5
http://192.168.2.250:443/gx-eop-es-30254/gx-eop-es:meng-v1.0.3-20220412182511
http://192.168.2.250:443/gx-eop-es-30254/gx-eop-es:meng-v1.0.3
http://192.168.2.250:443/gx-eop-es-30254/gx-eop-es:meng-v1.0.3-20220407184829
http://192.168.2.250:443/gx-eop-es-30254/gx-eop-es:meng-v1.0.5-20220412183221
http://192.168.2.250:443/wxgtest-25688/app-wxgtest:v1.0-wxg-20211207193353

API v2版本的Shell脚本实现
在shell脚本中通过调用Harbor API获取Harbor中所有项目下的所有镜像，然后导出到文本中。

本脚本按API V2版本的调用方式编写（2.0版本以下不适用），可通过以下命令查看你使用的Harbor API版本。[root@harbor-conso ~]# curl https://192.168.2.250:443/api/version -k
{"version":"v2.0"}
1.
2.
编写脚本
[root@harbor-conso ~]# Harbor-image-listk-v2.sh

#!/bin/bash
Harbor_Address=192.168.2.250:443       #Harbor主机地址
Harbor_User=admin                      #登录Harbor的用户
Harbor_Passwd=Harbor12345              #登录Harbor的用户密码
Images_File=harbor-images-`date '+%Y-%m-%d'`.txt   # 镜像清单文件                     #镜像清单文件
Tar_File=/backup/Harbor-backup/                                         #镜像tar包存放路径
set -x
# 获取Harbor中所有的项目（Projects）
Project_List=$(curl -u admin:Harbor12345  -H "Content-Type: application/json" -X GET  https://192.168.2.250:443/api/v2.0/projects  -k  | python -m json.tool | grep name | awk '/"name": /' | awk -F '"' '{print $4}')

for Project in $Project_List;do
   # 循环获取项目下所有的镜像
    Image_Names=$(curl -u admin:Harbor12345 -H "Content-Type: application/json" -X GET https://192.168.2.250:443/api/v2.0/projects/$Project/repositories -k | python -m json.tool | grep name | awk '/"name": /' | awk -F '"' '{print $4}')
    for Image in $Image_Names;do
        # 循环获取镜像的版本（tag)
        Image_Tags=$(curl -u admin:Harbor12345  -H "Content-Type: application/json"   -X GET  https://192.168.2.250:443/v2/$Image/tags/list  -k |  awk -F '"'  '{print $8,$10,$12}')
        for Tag in $Image_Tags;do
            # 格式化输出镜像信息
            echo "$Harbor_Address/$Image:$Tag"   >> harbor-images-`date '+%Y-%m-%d'`.txt
        done
    done
done

执行脚本
[root@harbor-conso ~]# chmod +x Harbor-image-listk-v2.sh
[root@harbor-conso ~]# sh Harbor-image-listk-v2.sh
1.
2.

从镜像清单查找镜像
然后打开脚本所在目录的harbor-images-`date '+%Y-%m-%d'`.txt文件即可看到镜像清单了。

复制
[root@harbor-conso ~]# cat harbor-images-2022-04-18.txt c| tail -10
http://192.168.2.250:443/lihonggang-25655/eeeee:rrrrr-20211207201732
http://192.168.2.250:443/lihonggang-25655/wwww:aaaa-20211207202110
http://192.168.2.250:443/lihonggang-25655/wwww:aaaa-20211207202855
http://192.168.2.250:443/gx-eop-es-30254/gx-eop-es:meng-v1.0.5-20220414154335
http://192.168.2.250:443/gx-eop-es-30254/gx-eop-es:meng-v1.0.5
http://192.168.2.250:443/gx-eop-es-30254/gx-eop-es:meng-v1.0.3-20220412182511
http://192.168.2.250:443/gx-eop-es-30254/gx-eop-es:meng-v1.0.3
http://192.168.2.250:443/gx-eop-es-30254/gx-eop-es:meng-v1.0.3-20220407184829
http://192.168.2.250:443/gx-eop-es-30254/gx-eop-es:meng-v1.0.5-20220412183221
http://192.168.2.250:443/wxgtest-25688/app-wxgtest:v1.0-wxg-20211207193353
-----------------------------------
©著作权归作者所有：来自51CTO博客作者键客李大白的原创作品，请联系作者获取转载授权，否则将追究法律责任
shell中获取Harbor中所有的镜像列表（超实用，建议收藏）
https://blog.51cto.com/lidabai/5217350

Harbor2.0 API 镜像仓库清理脚本--python

harbor_clearimage.py
# -*- coding:utf-8 -*-
import requests
from requests.auth import HTTPBasicAuth
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
import os
import time
import logging
from logging.handlers import RotatingFileHandler
 
class Harbor(object):
def __init__(self, api_url, api_user, api_passwd, tag_num, proj_exclude):
self.api_url = api_url
self.api_user = api_user
self.api_passwd = api_passwd
self.api_auth = HTTPBasicAuth(self.api_user, self.api_passwd)
self.tag_num = int(tag_num)
self.proj_exclude = proj_exclude
self.proj_url = self.api_url + "/projects"
self.repos_url = self.api_url + "/repositories"
self.header_dict = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36',
'Content-Type': 'application/x-www-form-urlencoded'
}
self.deldata = []
self.session = requests.Session()
self.session.auth = self.api_auth
retry = Retry(connect=3, backoff_factor=1)
adapter = HTTPAdapter(max_retries=retry)
self.session.mount('http://', adapter)
self.session.keep_alive = False
 
def soft_del_repos(self):
try:
projresp = self.session.get(self.proj_url, headers=self.header_dict)
if projresp.status_code == 200:
projdata = projresp.json()
for proj in projdata:
if proj['name'] not in self.proj_exclude # and proj['name'] == "gxjxhwebtest-28003" :
try:
reporesp = self.session.get(self.repos_url, params={"project_id": proj['project_id']} , headers=self.header_dict)
if reporesp.status_code == 200:
repodata = reporesp.json()
for repo in repodata:
if repo["tags_count"] > self.tag_num:
tag_url = self.repos_url + "/" + repo['name'] + "/tags"
tags = self.session.get(tag_url).json()
tagdata = sorted(tags, key=lambda a: a["created"])
del_tags = tagdata[0:len(tagdata) - self.tag_num]
for tag in del_tags:
del_repo_tag_url = tag_url + "/" + tag['name']
cmd = 'curl -v -X DELETE -u "' + self.api_user + ":" + self.api_passwd + '" -H "accept: application/json" ' + del_repo_tag_url
try:
#del_resp = self.session.delete(del_repo_tag_url,headers=self.header_dict)
ok = os.system(cmd)
if ok == 0 :
logging.info("httpdel:" + del_repo_tag_url )
_deldata = {"project_id":proj['project_id'],"project_name":proj['name'],"repo_name":repo['name'],"tag_name":tag['name']}
self.deldata.append(_deldata)
logging.info("httpdel project_id=" + str(proj['project_id']) + ",project_name=" + proj['name'] + ",repo_name=" + repo['name'] + ",tag_name=" + tag['name'])
else:
logging.error("exec_cmd fail:" + cmd )
except:
logging.error("exec_cmd fail:" + cmd )
except:
logging.error("httpget fail:" + self.repos_url)
else:
logging.error("httpget fail:" + self.proj_url )
except:
logging.error("apilogin fail:" + self.api_url )
return self.deldata
 
def hard_del_repo(self):
pwd_cmd = "cd /dcos/app/harbor/ " #进入到Harbor安装目录
stop_cmd = "docker-compose stop" #停止Harbor服务
del_cmd = "docker run -it --name gc --rm --volumes-from registry goharbor/registry-photon:v2.7.1-patch-2819-v1.8.6 garbage-collect /etc/registryctl/config.yml"
start_cmd = "docker-compose start" #启动Harbor服务
os.system(pwd_cmd)
ok1 = os.system(stop_cmd)
if ok1 == 0 :
time.sleep(10)
ok2 = os.system(del_cmd)
ok3 = os.system(start_cmd)
if ok3 == 0 :
logging.info("hard_del_repo ok:")
else:
logging.error("hard_del_repo fail:")
 
if __name__ == "__main__":
Rthandler = RotatingFileHandler('harbor_repo_clear.log', maxBytes=10*1024*1024,backupCount=5)
logging.basicConfig(level=logging.INFO)
formatter = logging.Formatter('%(levelname)s %(asctime)s %(process)d %(thread)d %(pathname)s %(filename)s %(funcName)s[line:%(lineno)d] %(message)s')
Rthandler.setFormatter(formatter)
logging.getLogger('').addHandler(Rthandler)
 
api_url = "http://192.168，2，66:443/api" #Harbor服务的API URL
api_user = "admin" #超级管理员
api_passwd = "Harbor12345" #超级管理员的用户密码
tag_num = 20 #保留的tag数量
proj_exclude = ["library"]
harborClient = Harbor(api_url,api_user,api_passwd,tag_num,proj_exclude)
data = harborClient.soft_del_repos()
if len(data) > 0 :
#harborClient.hard_del_repos()
logging.info("hard_del_repo:")

执行Python脚本

python harbor_clearimage.py

Harbor2.0 API 镜像仓库清理脚本--shell

执行前须安装jq（json解析工具）

wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum install jq -y

shell脚本如下,需要放置harbor机器执行：

#!/bin/bash
set -e
HARBOR_URL=xxxx.xxxx.xxxx
HARBOR_USER=user
HARBOR_PASSWD=password
#最大保留镜像个数。超过后的删除。
OLD_VERSION_NUM=20

#获取仓库项目列表
function get_projects_list(){
  projects_list=$(curl -s -k -u ${HARBOR_USER}:${HARBOR_PASSWD} "https://${HARBOR_URL}/api/v2.0/projects?page=1&page_size=100")
  mkdir -p $PWD/projectsList
  echo "${projects_list}" | jq '.[]' | jq -r '.name' > $PWD/projectsList/projectsList.txt
}

#获取每个项目下仓库列表
function get_repos_list(){
  mkdir -p $PWD/reposList
  for project in $(cat $PWD/projectsList/projectsList.txt);do
    repos_list=$(curl -s -k -u ${HARBOR_USER}:${HARBOR_PASSWD} "https://${HARBOR_URL}/api/v2.0/projects/${project}/repositories?page=1&page_size=100")
    echo "${repos_list}" | jq '.[]' | jq -r '.name'| awk -F "/" '{print $2}' > $PWD/reposList/${project}.txt
  done
}

#获取每个仓库下镜像digest列表
function get_tag_list(){

  for project in $(cat $PWD/projectsList/projectsList.txt)
  do
    for repo in `cat $PWD/reposList/${project}.txt`
      do
        mkdir -p $PWD/tagsList/$project/$repo
        curl -s -k -u ${HARBOR_USER}:${HARBOR_PASSWD} "https://${HARBOR_URL}/api/v2.0/projects/${project}/repositories/${repo}/artifacts?page=1&page_size=100&with_tag=true&wi
th_label=false&with_scan_overview=false&with_signature=false&with_immutable_status=false" | jq '.[]' | jq -r '.digest' > $PWD/tagsList/$project/$repo/tags.txt
     done
  done
}

#遍历每个项目下每个仓库执行镜像删除
function delete_images(){
  for project in $(cat $PWD/projectsList/projectsList.txt)
  do
    for repo in `cat $PWD/reposList/${project}.txt`
      do
        TOTAL_NUN=`cat $PWD/tagsList/$project/$repo/tags.txt |wc -l `
        if [ $TOTAL_NUN -gt $OLD_VERSION_NUM ];then
            DELETE_NUN=`echo $TOTAL_NUN - $OLD_VERSION_NUM | bc `
            tail -n  $DELETE_NUN $PWD/tagsList/$project/$repo/tags.txt > $PWD/tagsList/$project/$repo/tags-deleted.txt
            echo "################## ${project}/${repo} have $DELETE_NUN images need to delete ###############################################"
            echo "#######################begine delete ${project}/${repo} images###################################"
            for i in `cat $PWD/tagsList/$project/$repo/tags-deleted.txt`
              do
                IMAGE_NAME=`curl -s -k -u ${HARBOR_USER}:${HARBOR_PASSWD} "https://${HARBOR_URL}/api/v2.0/projects/${project}/repositories/${repo}/artifacts/${i}?page=1&page_
size=10&with_tag=true&with_label=false&with_scan_overview=false&with_signature=false&with_immutable_status=false" | jq -r '.tags' | jq -r '.[]' | jq -r '.name'`
                echo "######################### delete image $HARBOR_URL/$project/$repo:$IMAGE_NAME #################"
                curl -X 'DELETE' -u ${HARBOR_USER}:${HARBOR_PASSWD} "https://${HARBOR_URL}/api/v2.0/projects/${project}/repositories/${repo}/artifacts/${i}"
                echo "######################## delete image $HARBOR_URL/$project/$repo:$IMAGE_NAME success #########"
             done
             echo "####################### ${project}/${repo} images have deleted success###################################"
             echo "*********************************************************************************************************"
          fi
     done
  done

}

#启动容器，执行harbor gc 回收
function clean_registry(){
#  image_name=$(docker ps | grep registry | grep photon | awk -F " " '{print $2}')
#  docker run -it --name gc --rm --volumes-from registry ${image_name} garbage-collect  /etc/registry/config.yml
   docker run -it --name gc --rm --volumes-from registry vmware/registry:2.6.2-photon garbage-collect  /etc/registry/config.yml

}

function entrance(){
 get_projects_list
 get_repos_list
 get_tag_list
 delete_images
 clean_registry
}

entrance

定时清理添加定时任务：

crontab -e
00 3 * * * /opt/docker-clean/harbor-clean.sh > /opt/docker-clean/harbor-clean.log 2>&1