发现gitlab服务停止（四）

参考

Gitlab快速部署及日常维护（社区版RPM包方式安装）_51CTO博客_gitlab社区版使用教程

本文主要记录2023/11/9在Centos7上排查Gitlab服务停止的操作过程，以及过程中问题的处理。

连接服务器发现内存只用了半G，难道退出终端后gitlab服务就自动关闭了吗

查看状态，服务已经停止

[root@gitlab gitlab]# gitlab-ctl status
fail: alertmanager: runsv not running
fail: gitaly: runsv not running
fail: gitlab-exporter: runsv not running
fail: gitlab-kas: runsv not running
fail: gitlab-workhorse: runsv not running
fail: logrotate: runsv not running
fail: nginx: runsv not running
fail: node-exporter: runsv not running
fail: postgres-exporter: runsv not running
fail: postgresql: runsv not running
fail: prometheus: runsv not running
fail: puma: runsv not running
fail: redis: runsv not running
fail: redis-exporter: runsv not running
fail: sidekiq: runsv not running
[root@gitlab gitlab]#

先去看看日志吧

gitlab常用的默认安装目录

gitlab组件日志路径：/var/log/gitlab
 
gitlab配置路径：/etc/gitlab/  路径下有gitlab.rb配置文件
 
应用代码和组件依赖程序：/opt/gitlab
 
各个组件存储路径： /var/opt/gitlab/
 
仓库默认存储路径   /var/opt/gitlab/git-data/repositories
 
版本文件备份路径：/var/opt/gitlab/backups/
 
nginx安装路径：/var/opt/gitlab/nginx/
 
redis安装路径：/var/opt/gitlab/redis

[root@gitlab etc]# cd /var/log/gitlab/
[root@gitlab gitlab]# ll
总用量 4
drwx------. 2 gitlab-prometheus root         47 11月  8 16:50 alertmanager
drwx------. 2 git               root         71 11月  8 17:14 gitaly
drwx------. 2 git               root         47 11月  8 16:50 gitlab-exporter
drwx------. 2 git               root         47 11月  8 16:48 gitlab-kas
drwx------. 2 git               root       4096 11月  8 18:59 gitlab-rails
drwx------. 2 git               root          6 11月  8 16:45 gitlab-shell
drwx------. 2 git               root         47 11月  8 16:50 gitlab-workhorse
drwx------. 2 root              root         47 11月  8 16:47 logrotate
drwxr-x---. 2 root              gitlab-www  131 11月  8 16:50 nginx
drwx------. 2 gitlab-prometheus root         47 11月  8 16:50 node-exporter
drwx------. 2 gitlab-psql       root         47 11月  8 16:51 postgres-exporter
drwx------. 2 gitlab-psql       root         47 11月  8 16:48 postgresql
drwx------. 2 gitlab-prometheus root         47 11月  8 16:50 prometheus
drwx------. 2 git               root         93 11月  8 16:50 puma
drwxr-xr-x. 2 root              root         50 11月  8 16:47 reconfigure
drwx------. 2 gitlab-redis      root         47 11月  8 16:48 redis
drwx------. 2 gitlab-redis      root         47 11月  8 16:50 redis-exporter
drwx------. 2 git               root         47 11月  8 16:50 sidekiq
[root@gitlab gitlab]# sudo cat ./gitlab-rails/production.log
Raven 3.1.2 configured not to capture errors: DSN not set
Raven 3.1.2 configured not to capture errors: DSN not set
Raven 3.1.2 configured not to capture errors: DSN not set
Raven 3.1.2 configured not to capture errors: DSN not set
  Rendered layout layouts/mailer/devise.html.haml (Duration: 58.6ms | Allocations: 26286)
Delivered mail 654b4dade1773_62aa6dfd83454@gitlab.mail (359.6ms)

执行命令查看所有的logs：sudo gitlab-ctl tail

判断大概在退出终端90分钟后，gitlab服务停止

==> /var/log/gitlab/gitlab-exporter/current <==
2023-11-08_10:58:44.53934 - -> /ruby
2023-11-08_10:58:52.13681 ::1 - - [08/Nov/2023:18:58:52 CST] "GET /sidekiq HTTP/1.1" 200 119348
2023-11-08_10:58:52.13684 - -> /sidekiq
2023-11-08_10:58:59.21303 ::1 - - [08/Nov/2023:18:58:59 CST] "GET /database HTTP/1.1" 200 1968
2023-11-08_10:58:59.21307 - -> /database
2023-11-08_10:58:59.53605 ::1 - - [08/Nov/2023:18:58:59 CST] "GET /ruby HTTP/1.1" 200 1073
2023-11-08_10:58:59.53607 - -> /ruby
2023-11-08_10:59:05.53635 == Sinatra has ended his set (crowd applauds)
2023-11-08_10:59:05.61880 [2023-11-08 18:59:05] INFO  going to shutdown ...
2023-11-08_10:59:05.61918 [2023-11-08 18:59:05] INFO  WEBrick::HTTPServer#start done.

因为现在还在学习阶段，不求甚解，以跑通gitlab的CI/CD为先。先启动gitlab服务。

[root@gitlab gitlab]# systemctl enable gitlab-runsvdir.service
[root@gitlab gitlab]# gitlab-ctl restart
fail: alertmanager: runsv not running
fail: gitaly: runsv not running
fail: gitlab-exporter: runsv not running
fail: gitlab-kas: runsv not running
fail: gitlab-workhorse: runsv not running
fail: logrotate: runsv not running
fail: nginx: runsv not running
fail: node-exporter: runsv not running
fail: postgres-exporter: runsv not running
fail: postgresql: runsv not running
fail: prometheus: runsv not running
fail: puma: runsv not running
fail: redis: runsv not running
fail: redis-exporter: runsv not running
fail: sidekiq: runsv not running

启动失败，看到第一个信息fail: alertmanager: runsv not running，需要启动gitlab运行服务目录(runsvdir)

[root@gitlab gitlab]# systemctl start gitlab-runsvdir
[root@gitlab gitlab]# gitlab-ctl start
ok: run: alertmanager: (pid 5352) 7s
ok: run: gitaly: (pid 5338) 7s
ok: run: gitlab-exporter: (pid 5356) 7s
ok: run: gitlab-kas: (pid 5350) 7s
ok: run: gitlab-workhorse: (pid 5343) 7s
ok: run: logrotate: (pid 5336) 7s
ok: run: nginx: (pid 5339) 7s
ok: run: node-exporter: (pid 5335) 7s
ok: run: postgres-exporter: (pid 5353) 7s
ok: run: postgresql: (pid 5351) 7s
ok: run: prometheus: (pid 5329) 7s
ok: run: puma: (pid 5341) 7s
ok: run: redis: (pid 5337) 7s
ok: run: redis-exporter: (pid 5328) 7s
ok: run: sidekiq: (pid 5345) 7s

查看状态，又能了

[root@gitlab gitlab]# gitlab-ctl status
run: alertmanager: (pid 5352) 53s; run: log: (pid 5347) 53s
run: gitaly: (pid 5338) 53s; run: log: (pid 5332) 53s
run: gitlab-exporter: (pid 5356) 53s; run: log: (pid 5348) 53s
run: gitlab-kas: (pid 5350) 53s; run: log: (pid 5331) 53s
run: gitlab-workhorse: (pid 5343) 53s; run: log: (pid 5342) 53s
run: logrotate: (pid 5336) 53s; run: log: (pid 5325) 53s
run: nginx: (pid 5339) 53s; run: log: (pid 5333) 53s
run: node-exporter: (pid 5335) 53s; run: log: (pid 5334) 53s
run: postgres-exporter: (pid 5353) 53s; run: log: (pid 5349) 53s
run: postgresql: (pid 5351) 53s; run: log: (pid 5346) 53s
run: prometheus: (pid 5329) 53s; run: log: (pid 5327) 53s
run: puma: (pid 5341) 53s; run: log: (pid 5340) 53s
run: redis: (pid 5337) 53s; run: log: (pid 5330) 53s
run: redis-exporter: (pid 5328) 53s; run: log: (pid 5326) 53s
run: sidekiq: (pid 5345) 53s; run: log: (pid 5344) 53s

埋下个疑问：为啥gitlab-ctl restart命令不能，而systemctl start gitlab-runsvdir命令可以启动

Gitlab服务的启停管理

启动服务： gitlab-ctl start

停止服务： gitlab-ctl stop

重启服务： gitlab-ctl restart

查看状态： gitlab-ctl status

Gitlab的supervisor方式启动服务

服务启动命令：   systemctl start gitlab-runsvdir.service

服务停止命令：   systemctl stop gitlab-runsvdir.service

服务重启命令：   systemctl restart gitlab-runsvdir.service

服务开机启动命令：   systemctl enable gitlab-runsvdir.service

取消开机启动命令：   systemctl disable gitlab-runsvdir.service

服务查看命令：   systemctl list-unit-files

Gitlab服务日志查看：/usr/bin/gitlab-ctl tail         #可以查看到gitlab所有插件的日志情况

----END LINE