BlackDuck升级和备份

最新推荐文章于 2024-11-10 22:22:46 发布

sssssssssnake

最新推荐文章于 2024-11-10 22:22:46 发布

阅读量244

点赞数 10

文章标签： docker

本文链接：https://blog.csdn.net/sssssssssnake/article/details/140567787

版权

内网BD服务的 KB服务更新
整个流程说明

厂家邮寄更新KB数据盘到机房
联系IT协助将数据盘插到BD服务器的USB3.0的接口上面
将数据盘挂载到系统上面
将数据盘的数据rsync到/blackduck 目录下面
解压复制过来的数据盘
检查解压后的数据盘大小
联系商家协助升级
升级之前先备份hub的数据库（如果hub更新失败，需要重新安装hub服务，需要恢复之前的hub数据）
关停hub服务
关停kb服务
重新更新KB服务（更新就是重新安装）
启动更新后的KB服务
更新HUB服务
修改HUB配置参数
启动HUB服务
验证服务是否完成
完成升级

步骤参考
挂载数据盘到操作系统

# 以下命令用 root 身份运行
# 查看数据盘
fdisk -l

# 外挂的数据盘，前2次都是 /dev/sdc,可以根据大小判断
# /KB 目录是专门创建用于挂载移动硬盘的目录
mount /dev/sdc2 /mnt/

创建新的KB数据目录
KB每个月都会更新数据，所以数据目录可以根据月份来进行区分
以下以2月份的升级操作做参考，创建的目录是KB_2024_02

# 创建KB 数据目录
mkdir /blackduck/KB_2024_02

复制/mnt/ 目录数据到/blackduck/KB_2024_02目录

创建新session
screen -S cpy1
cp -r  /mnt/* /blackduck/KB_2024_02
退出
ctrl+a+d
screen -ls
重新进入
screen -r cpy1

解压KB_2024_02的数据

screen -S cpy1
bash /blackduck/KB_2024_02/decompression.sh

校验大小
数据解压之后可以根据目录中的文件提示校验大小

KBon-prem 2024.1


Software:
Total: 22G         no need for extraction before installation

Data:
ikb_match: 4.9T                           after extraction: 17T     var/lib/postgresql/basebackup/ikb_match/
snippet_db_2024.02_2.tar.gz: 121G         after extraction: 338G    var/lib/postgresql/basebackup/snippet_db/
kb_spider_api_2024.02_2.tar.gz: 381G      after extraction: 1.4T    var/lib/postgresql/basebackup/kb_spider_api/
kb_search_2024.02_2.tar.gz: 5.3G          after extraction: 6.8G    data/
fingerprints: 800G                        after extraction: 2.2T
total: 6.2T                               total after extraction: 26T


Note:

Information above is just for the reference and may have some deviation.

Directory ikb_match can be extracted using the extract.sh in this folder. Please see the file header of extract.sh for the prerequisite for running this script.

Please see the Data/ikb_match/checksum_sha256 for the sha256 checksum of the tar.gz files in Data/ikb_match directory.

The total size and fingerprints size after extraction also include the corresponding tar file.

For Resource Guidance & Container Scalability, Please refer to the Black Duck Documentation:
https://community.synopsys.com/s/document-item?bundleId=bd-hub&topicId=Install_Common/HardwareRequirements_2.html&_LANG=enus

Hub 数据库备份
查看备份相关内容即可，其他步骤无需操作

离线KB数据每月发布一次更新，KB为Black Duck的后端知识库，无用户数据，无需备份，按需选择月份进行升级即可。Hub中的数据存储主要在数据库中，数据库备份即还原方法如下：
 
数据库 audit_event大小查看：
docker ps -a
找到postgre 数据库的容器，进入容器
docker exec -it <container_ID> psql bds_hub
执行:
SELECT pg_size_pretty( pg_total_relation_size('st.audit_event') );
\q 
需满足磁盘空间2.5倍以上数据库大小，同时数据库所在磁盘可用空间60%以上
 
备份hub数据
在hub安装目录下执行：
docker-swarm/bin/hub_create_data_dump.sh <backup directory>
# docker-swarm/bin/hub_create_data_dump.sh /blackduck/hub_backup_0625 
停掉原有hub
docker stack rm hub
查看容器状态，直到hub所有容器均停掉为止
watch docker ps -a
还原数据
1> 查找postgre的volume
  docker volume ls
2> 删除postgre的volume
  docker volume rm hub_postgres96-data-volume
3>进入<hub_installation directory>/docker-swarm目录， 执行如下命令
docker stack deploy -c docker-compose.dbmigrate.yml hub --with-registry-auth
4> 查看容器状态
watch docker ps -a
待所有容器up且healthy之后，执行：
./bin/hub_db_migrate.sh <backup directory>
migrate脚本执行完成后，运行VACUUM命令，清理数据库空间，优化数据库性能  VACUUM命令要求两倍以上的audit_event表所需硬盘空间，参考步骤1查看空间大小
找到postgre 数据的容器，进入容器
docker exec -it <container_ID> psql bds_hub
执行:
VACUUM FULL ANALYZE st.audit_event;
\q
完成上述步骤后，重新启动hub即可。

升级

# Check the versions
# Usually latest versions
docker -v
docker-compose -v

# Check the docker root directory. Should be the largest directory. Take /data/docker as a example.
docker info
export DOCKER=/blackduck/docker

# check docker container, image,  volume
docker container ls -a
docker images
docker volume ls -a
# In terms of an upgrade, stop and remove all the KB containers, images and the volumes.
#docker stop $(docker ps -a -q); docker container prune -f
#docker image prune -a -f
rm -f $DOCKER/volumes/kb_pg_api/_data
rm -f $DOCKER/volumes/kb_pg_match/_data
rm -f $DOCKER/volumes/kb_pg_snippet/_data
rm -f $DOCKER/volumes/kb_search/_data
docker volume ls|grep kb_ | awk -F "local     " '{print $2}'|xargs -L 1 docker volume rm


# Extract the Orchestration tarball files. 
export CGA=/blackduck/KB_2024_02/Software
cd $CGA/Orchestration
kb-onprem-2024.1.0.tar

# Extract the KB tarball files. 
tar -xf kb-onprem-2024.1.1.tar

# Load docker images
cd $CGA/KBServices
for i in *.tar; do docker load -i $i; done;
# The above step can be achieved using $CGA/KBServices/load_images.sh
# Check if all the images have the proper name and version.
# Otherwise, use the tag command such as`docker image tag 3548556fb637 blackducksoftware/kb_auth:3.0.1`
# You must make sure that the image version exists.
#./load_images.sh




# Create volumes
$CGA/Orchestration/kb-orch/setup/create-volumes.sh


#check the size of each directory to see if the packs has been extracted successfully

# Create symbolic links
# Export docker root directory
export DOCKER=/blackduck/docker
cd $DOCKER/volumes/kb_pg_api
rm -rf _data
ln -s /blackduck/KB_2024_02/Data/var/lib/postgresql/basebackup/kb_spider_api _data
ll _data

cd $DOCKER/volumes/kb_pg_match
rm -rf _data
ln -s /blackduck/KB_2024_02/Data/var/lib/postgresql/basebackup/ikb_match/iscan_v7 _data
ll _data

cd $DOCKER/volumes/kb_pg_snippet
rm -rf _data
ln -s /blackduck/KB_2024_02/Data/var/lib/postgresql/basebackup/snippet_db _data
ll _data

cd $DOCKER/volumes/kb_search
rm -rf _data
ln -s /blackduck/KB_2024_02/Data/data _data
ll _data

# run the followings to check the links
cd ../
root@bdadmin-PowerEdge-T640:/data/docker/volumes# tree .
.
├── backingFsBlockDev
├── kb_pg_api
│   └── _data -> /data/KBon-prem01/Data/var/lib/postgresql/basebackup/kb_spider_api
├── kb_pg_match
│   └── _data -> /data/KBon-prem01/Data/var/lib/postgresql/basebackup/ikb_match/iscan_v7
├── kb_pg_snippet
│   └── _data -> /data/KBon-prem01/Data/var/lib/postgresql/basebackup/snippet_db
├── kb_search
│   └── _data -> /data/KBon-prem01/Data/data
└── metadata.db







# note down where you extract the fingerprints directory for further use
/blackduck/KB_2024_02/fingerprints/fingerprints_20220916



# Modify the setup scripts and execute
cd $CGA/Orchestration/kb-orch/setup
# Manually delete all the commands except those with `docker run` in the beginning.
# e.g. cat setup-api-data.sh | grep "docker run"
#./setup-api-data.sh; ./setup-match-non_baseline.sh; ./setup-search-data.sh; ./setup-snippet-data.sh



# Check the value of ISCAN_VERSION in the kb.env file. # ISCAN_VERSION=iscan_v7    baseline: iscan_v7_20210525

cd $CGA/Orchestration/kb-orch
cat kb.env

# Modify the docker-compose.onprem.yml
# services->kbapisnippetlookup->volumes-> and replace with
##- /data/KBon-prem01/fingerprints/:/mnt/data/fingerprints
# Modify the shm_size if needed.
vim $CGA/Orchestration/kb-orch/docker-compose.onprem.yml

#Modify the Security settings in $CGA/Orchestration/kb-orch/config/overrides.properties
bds.security.hostHeaderAllowlistCsv=localhost,127.0.0.1,10.132.63.66


# Modify the docker-compose.onprem.yml to make it as follows:
    kbapidetail:

        image: sigsynopsys/kbapi-detail:4.17.2

        depends_on:

            - kbapiauth

            - kbapidb

        tmpfs:

            - /mnt/logs

        volumes:

            - ./config/.pgpass:/home/kbapi_user/.pgpass
            - ./config/.jwt-secret:/home/kbapi_user/.jwt-secret
            - ./config/overrides.properties:/mnt/config/overrides.properties
			#- ./config/saved-feedback.txt:/mnt/logs/saved-feedback.txt

        restart: always

    kbapifeedback:

        image: sigsynopsys/kbapi-feedback:4.17.2

        depends_on:

            - kbapidetail

            - kbapiauth

        tmpfs:
            - /mnt/logs

        volumes:

            - ./config/.pgpass:/home/kbapi_user/.pgpass
            - ./config/.jwt-secret:/home/kbapi_user/.jwt-secret
            - ./config/overrides.properties:/mnt/config/overrides.properties

        restart: always




# Use docker-compose to start the KB server.
docker-compose -f docker-compose.onprem.yml -p hub up -d

# If you need to restart the server, you can stop the server.
docker-compose -f docker-compose.onprem.yml -p hub down

# All the containers should be healthy.

# Check if the KB is health. Note that the unhealthy kb_search and kb-meta-api are normal.

curl http://<ip of kb server>:8148/api/authentication-health



#For Hub installation

# Modify the hub connection to KB server
cd /blackduck/KB_2024_02/Software/Orchestration/hub-2023.10.1/docker-swarm
mv blackduck-config.env blackduck-config.env.bak
mv blackduck-config.externalHub.env blackduck-config.env
#nano blackduck-config.env
# TODO: In nano editor you can press `ctrl + \` to search <Ip Address of On Prem KB> and replace with the KB address, e.g. 10.132.63.66

#start hub as normal
#e.g. docker stack deploy -c docker-compose.yml -c docker-compose.local-overrides.yml hub

# Insert the license

#stop hub services
docker stack rm hub
#insert the license
rm /data/docker/volumes/hub_config-volume/_data/suite_v1.xml
#cp $LICENSE_PATH/suite_v1.xml /blackduck_Hub/docker/volumes/hub_config-volume/_data/ 
cp /data/suite_v1.xml /data/docker/volumes/hub_config-volume/_data
#after restart hub, check the license status:
#suite_v1.xml permission should be 644 owner should be systemd-network 
#owner 100:root

# Visit https://ip-address
# username: sysadmin
# password: blackduck
# Successful login, no more registration needed.

备份hub数据库

cd /blackduck/KB/Current_GA/Orchestration/hub-2021.10.3
docker-swarm/bin/hub_create_data_dump.sh /blackduck/hub_backup_0625

先停hub
docker stack rm hub

观察
watch docker ps -a
确保hub服务全部停止

在停kb
docker-compose -f docker-compose.onprem.yml -p hub down


清理image
docker image prune -a -f

清理kb volume
docker volume ls

cd /blackduck/docker/volumes

kb_pg_api
kb_pg_match
kb_pg_snippet
kb_search

依次进入目录，删除软连接
rm -f _data

然后删除4个kb的volume
docker volume rm kb_pg_api
docker volume rm kb_pg_match
docker volume rm kb_pg_snippet
docker volume rm kb_search
docker volume ls


# load KB image
cd /blackduck/KB_2024_02/Current_GA/KBServices
for i in *.tar; do docker load -i $i; done;

# load hub image
cd /blackduck/KB_2024_02/Current_GA/BlackDuck
for i in *.tar; do docker load -i $i; done;


 docker images |wc -l
 35个
 
 
解压hub和kb
cd /blackduck/KB_2024_02/Current_GA/Orchestration
tar -xf kb-onprem-2022.5.0.tar -C ./

安装kb
cd /blackduck/KB_2024_02/Current_GA/Orchestration/kb-orch
cd /blackduck/KB_2024_02/Current_GA/Orchestration/kb-orch/setup
创建kb volumes
bash create-volumes.sh

创建软链接

cd /blackduck/docker/volumes/kb_pg_api
rm -rf _data
ll -d /blackduck/KB_2024_02/kbspider/var/lib/postgresql/basebackup/kb_spider_api
ln -s /blackduck/KB_2024_02/kbspider/var/lib/postgresql/basebackup/kb_spider_api _data

cd /blackduck/docker/volumes/kb_pg_match
rm -rf _data
ll -d /blackduck/KB_2024_02/non-baseline/var/lib/postgresql/basebackup/ikb_match/iscan_v7
ln -s /blackduck/KB_2024_02/non-baseline/var/lib/postgresql/basebackup/ikb_match/iscan_v7 _data

cd /blackduck/docker/volumes/kb_pg_snippet
rm -rf _data
ll -d /blackduck/KB_2024_02/snippet/var/lib/postgresql/basebackup/snippet_db
ln -s /blackduck/KB_2024_02/snippet/var/lib/postgresql/basebackup/snippet_db _data

cd /blackduck/docker/volumes/kb_search
rm -rf _data
ll -d /blackduck/KB_2024_02/solr/data
ln -s /blackduck/KB_2024_02/solr/data _data

执行权限设置命令
cd /blackduck/KB_2024_02/Current_GA/Orchestration/kb-orch/setup
grep -r "docker run" ./

docker run --rm -v kb_pg_api:/data -u root:root --entrypoint="" blackducksoftware/kb_api_db:4.0.2 sh -c 'chown -R postgres:postgres /data'
docker run --rm -v kb_pg_match:/data -u root:root --entrypoint="" blackducksoftware/kb_match_db:4.0.0 sh -c 'chown -R postgres:postgres /data'
docker run --rm -v kb_pg_snippet:/data -u root:root --entrypoint="" blackducksoftware/kb_snippet_db:3.0.1 sh -c 'chown -R postgres:postgres /data'


/blackduck/KB_2024_02/Current_GA/Orchestration/kb-orch
vim docker-compose.onprem.yml
fingerprints 目录替换成/blackduck/KB_2024_02/fingerprints/
shm_size 调整为5120mb


启动kb
docker-compose -f docker-compose.onprem.yml -p hub up -d

测试kb
curl http://10.10.13.249:8148/api/authentication-health
{"isRegistrationHealthy":true,"isRegistrationCacheHealthy":true,"lastCheckedDate":"2022-06-25T06:52:07.979Z","_meta":{"href":"/api/authentication-health","links":[]},"isHealthy":true}r

cd /blackduck/KB_2024_02/Current_GA/Orchestration/hub-2022.4.0/docker-swarm
cp 旧版本的docker-compose.local-overrides.yml文件过来使用

#仅修改docker-compose.local-overrides.yml这个配置里面的https证书配置，资源配置使用sizes-gen03模板 
#启动hub，以每小时120次扫描的资源配置启动sizes-gen03/120sph.yaml
#docker stack deploy -c docker-compose.yml -c docker-compose.local-overrides.yml -c sizes-gen03/120sph.yaml hub
/blackduck/KB_2024_02/Current_GA/Orchestration/hub-2022.4.0/docker-swarm/blackduck-config.env
cp blackduck-config.externalHub.env blackduck-config.env
最下面增加这4行
JOBRUNNER_OPTS=-XX:ActiveProcessorCount=4
WEBAPP_SERVICE_OPTS=-XX:ActiveProcessorCount=4
SCAN_SERVICE_OPTS=-XX:ActiveProcessorCount=4
REGISTRATION_SERVICE_OPTS=-XX:ActiveProcessorCount=4
BOMENGINE_SERVICE_OPTS=-XX:ActiveProcessorCount=4
AUTHENTICATION_SERVICE_OPTS=-XX:ActiveProcessorCount=4

将里面的<IP ADDRESS OF ONPREM SERVER> 修改为KB的IP地址 10.10.13.249
:%s/<IP ADDRESS OF ONPREM SERVER>/10.10.13.249

启动hub
docker stack deploy -c docker-compose.yml -c docker-compose.local-overrides.yml -c docker-compose.bdba.yml hub

内网BD服务数据定时备份
crontab任务
00 18 * * * bash -x /blackduck/backup/backup.sh > /blackduck/backup.log 2>&1
/blackduck/backup/backup.sh
每次更新需要修改脚本

#!/bin/bash

date

#docker stack rm hub
#cd /blackduck/KB_2023_08/Current_GA/Orchestration/hub-2023.4.1/docker-swarm
#docker stack deploy -c docker-compose.dbmigrate.yml hub
#cd -

DT=date '+%Y_%m_%d'

backupdir=/blackduck/backup/data_2024_02

current_dir=/blackduck/backup/current

rm -rf ${current_dir}

mkdir -p ${current_dir}

find ${backupdir} -type f -name "*.tar" -mtime +6 |xargs rm -f

cd /blackduck/KB_2024_02/Software/Orchestration/hub-2023.10.1

ls docker-swarm/bin/hub_create_data_dump.sh

#docker ps -a

docker-swarm/bin/hub_create_data_dump.sh --live-system ${current_dir}

cd -

#cd /blackduck/KB_2023_08/Current_GA/Orchestration/hub-2023.4.1/docker-swarm
#docker stack deploy -c docker-compose.yml -c docker-compose.bdba.yml -c docker-compose.local-overrides.yml hub
#cd -

tar -cf ${backupdir}/bd_${DT}.tar ${current_dir}

rsync -avHrlptDP --delete --rsh=ssh  ${backupdir}/  root@10.34.5.116:${backupdir}/

date