k8s中修复mongodb启动失败

已于 2024-04-15 13:02:45 修改
阅读量517 收藏 5
点赞数 5
分类专栏： Database Cloud Native 文章标签： kubernetes mongodb 容器
于 2024-04-15 11:45:59 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_43652666/article/details/137772612
版权

背景

同事反馈 dev环境的yapi不能登录，看了一下是同事两年前用helm搭建的。单副本使用。
排查发现是后端数据库mongodb数据库挂掉。

root@dev-k8s-master03:~# kubectl get svc 
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
mongodb       ClusterIP   10.101.87.7     <none>        27017/TCP        2y39d
yapi-svc      NodePort    10.108.180.80   <none>        3000:31781/TCP   2y39d

故障现象

 2024-04-12T09:52:26.257+0000 I CONTROL  [initandlisten] 4096 MB of memory available to the process out of 64321 MB total system memory

 2024-04-12T09:52:26.257+0000 I CONTROL  [initandlisten] options: { config: "/opt/bitnami/mongodb/conf/mongodb.conf", net: { bindIpAll: true, ipv6: false, port: 27017, unixDomainSocket: { enabled: true, pathPrefix: "/opt/bitnami/mongodb/tmp" } }, processManagement: { fork: false, pidFilePath: "/opt/bitnami/mongodb/tmp/mongodb.pid" }, security: { authorization: "enabled" }, setParameter: { enableLocalhostAuthBypass: "false" }, storage: { dbPath: "/bitnami/mongodb/data/db", directoryPerDB: false, journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/opt/bitnami/mongodb/logs/mongodb.log", quiet: false, verbosity: 0 } }

 2024-04-12T09:52:26.258+0000 W STORAGE  [initandlisten] Detected unclean shutdown - /bitnami/mongodb/data/db/mongod.lock is not empty.

 2024-04-12T09:52:26.259+0000 I STORAGE  [initandlisten] Detected data files in /bitnami/mongodb/data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.

 2024-04-12T09:52:26.259+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.

 2024-04-12T09:52:26.259+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1536M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),

 2024-04-12T09:52:26.365+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1712915546:365352][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:365352][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error

 2024-04-12T09:52:26.365+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:365392][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:365392][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata

 2024-04-12T09:52:26.365+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:365398][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:365398][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk

 2024-04-12T09:52:26.365+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:365403][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:365403][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options

 2024-04-12T09:52:26.376+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1712915546:376001][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:376001][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error

 2024-04-12T09:52:26.376+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:376056][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:376056][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata

 2024-04-12T09:52:26.376+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:376067][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:376067][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk

 2024-04-12T09:52:26.376+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:376077][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:376077][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options

 2024-04-12T09:52:26.386+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1712915546:386587][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:386587][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error

 2024-04-12T09:52:26.386+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:386638][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:386638][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata

 2024-04-12T09:52:26.386+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:386645][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:386645][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk

 2024-04-12T09:52:26.386+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:386652][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:386652][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options

 2024-04-12T09:52:26.390+0000 W STORAGE  [initandlisten] Failed to start up WiredTiger under any compatibility version.

 2024-04-12T09:52:26.390+0000 W STORAGE  [initandlisten] WiredTiger metadata corruption detected

 2024-04-12T09:52:26.390+0000 F STORAGE  [initandlisten] Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair

 2024-04-12T09:52:26.390+0000 F -        [initandlisten] Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 701

 2024-04-12T09:52:26.390+0000 F -        [initandlisten]

2024-04-12T09:52:26.257+0000 I CONTROL [initandlisten] 4096 MB of memory available to the process out of 64321 MB total system memory

2024-04-12T09:52:26.257+0000 I CONTROL [initandlisten] options: { config: “/opt/bitnami/mongodb/conf/mongodb.conf”, net: { bindIpAll: true, ipv6: false, port: 27017, unixDomainSocket: { enabled: true, pathPrefix: “/opt/bitnami/mongodb/tmp” } }, processManagement: { fork: false, pidFilePath: “/opt/bitnami/mongodb/tmp/mongodb.pid” }, security: { authorization: “enabled” }, setParameter: { enableLocalhostAuthBypass: “false” }, storage: { dbPath: “/bitnami/mongodb/data/db”, directoryPerDB: false, journal: { enabled: true } }, systemLog: { destination: “file”, logAppend: true, logRotate: “reopen”, path: “/opt/bitnami/mongodb/logs/mongodb.log”, quiet: false, verbosity: 0 } }

2024-04-12T09:52:26.258+0000 W STORAGE [initandlisten] Detected unclean shutdown - /bitnami/mongodb/data/db/mongod.lock is not empty.

2024-04-12T09:52:26.259+0000 I STORAGE [initandlisten] Detected data files in /bitnami/mongodb/data/db created by the ‘wiredTiger’ storage engine, so setting the active storage engine to ‘wiredTiger’.

2024-04-12T09:52:26.259+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.

2024-04-12T09:52:26.259+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1536M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),

2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1712915546:365352][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:365352][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error

2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:365392][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:365392][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata

2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:365398][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:365398][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk

2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:365403][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:365403][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options

2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1712915546:376001][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:376001][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error

2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:376056][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:376056][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata

2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:376067][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:376067][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk

2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:376077][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:376077][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options

2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1712915546:386587][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:386587][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error

2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:386638][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:386638][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata

2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:386645][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:386645][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk

2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:386652][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:386652][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options

2024-04-12T09:52:26.390+0000 W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version.

2024-04-12T09:52:26.390+0000 W STORAGE [initandlisten] WiredTiger metadata corruption detected

2024-04-12T09:52:26.390+0000 F STORAGE [initandlisten] Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair

2024-04-12T09:52:26.390+0000 F - [initandlisten] Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 701

2024-04-12T09:52:26.390+0000 F - [initandlisten]

分析&思路

看前面的E级别的log信息，存储引擎读取root page失败。unable to read root page from file:WiredTiger.wt

找到这个mongo挂载的nfs存储，先将数据库文件做一个备份。

单副本的mongo并且一直不断重启，网上搜到的使用mogod --repair执行修复 根本不能进入到容器中执行的。

在同一个pod中使用跟mongo同样的镜像做一个二次封装，命名为mongodebug 把这个镜像放在mongo的pod中，使得这个pod存在两个容器，然后挂载同一个共享存储，进入到mongodebug执行mongod命令进行修复

实施

必要条件

挂载持久卷存储的容器路径/bitnami/mongodb/data/db

制作mongodebug容器镜像

目的：需要一个可执行的mongd命令

拉镜像，运行，找到容器启动脚本，覆盖掉。
防止读取同一份配置文件不能启动

root@master01:~# docker run -it bitnami/mongodb:4.0.14-debian-9-r24 /bin/bash 
 10:11:48.80 
 10:11:48.80 Welcome to the Bitnami mongodb container
 10:11:48.81 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mongodb
 10:11:48.81 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mongodb/issues
 10:11:48.81 Send us your feedback at containers@bitnami.com
 10:11:48.81 

I have no name!@7b9781de0eed:/$ 
I have no name!@7b9781de0eed:/$ 
I have no name!@7b9781de0eed:/$ ls
bin  bitnami  boot  dev  entrypoint.sh	etc  home  lib	lib64  libbitnami.sh  libcomponent.sh  libfile.sh  libfs.sh  liblog.sh	libmongodb.sh  libnet.sh  libos.sh  libservice.sh  libvalidations.sh  libversion.sh  licenses  media  mnt  opt	postunpack.sh  proc  root  run	run.sh	sbin  setup.sh	srv  sys  tmp  usr  var

I have no name!@7b9781de0eed:/$ cat entrypoint.sh 
#!/bin/bash
# shellcheck disable=SC1091

set -o errexit
set -o nounset
set -o pipefail


# Load libraries
. /libbitnami.sh
. /libmongodb.sh

# Load MongoDB env. variables
eval "$(mongodb_env)"

print_welcome_page

if [[ "$*" = "/run.sh" ]]; then
    info "** Starting MongoDB setup **"
    /setup.sh
    info "** MongoDB setup finished! **"
fi

echo ""
exec "$@"

I have no name!@7b9781de0eed:/$ cat run.sh 
#!/bin/bash
# shellcheck disable=SC1091

set -o errexit
set -o nounset
set -o pipefail

. /libmongodb.sh
. /libos.sh

# Load MongoDB env. variables
eval "$(mongodb_env)"

cmd=$(command -v mongod)

flags=("--config=$MONGODB_CONF_FILE")

if [[ -n "${MONGODB_EXTRA_FLAGS:-}" ]]; then
    read -r -a extra_flags <<< "$MONGODB_EXTRA_FLAGS"
    flags+=("${extra_flags[@]}")
fi

info "** Starting MongoDB **"
if am_i_root; then
    exec gosu "$MONGODB_DAEMON_USER" "${cmd}" "${flags[@]}"
else
    exec "${cmd}" "${flags[@]}"
fi
I have no name!@7b9781de0eed:/$ cmd=$(command -v mongod)
I have no name!@7b9781de0eed:/$ echo $cmd
/opt/bitnami/mongodb/bin/mongod
I have no name!@7b9781de0eed:/$ echo $MONGODB_CONF_FILE
/opt/bitnami/mongodb/conf/mongodb.conf
I have no name!@7b9781de0eed:/$ ls
bin  bitnami  boot  dev  entrypoint.sh	etc  home  lib	lib64  libbitnami.sh  libcomponent.sh  libfile.sh  libfs.sh  liblog.sh	libmongodb.sh  libnet.sh  libos.sh  libservice.sh  libvalidations.sh  libversion.sh  licenses  media  mnt  opt	postunpack.sh  proc  root  run	run.sh	sbin  setup.sh	srv  sys  tmp  usr  var
I have no name!@7b9781de0eed:/$ ps -ef 
UID          PID    PPID  C STIME TTY          TIME CMD
1001           1       0  0 10:11 pts/0    00:00:00 /bin/bash
1001          25       1  0 10:25 pts/0    00:00:00 ps -ef
I have no name!@7b9781de0eed:/$ exit
exit

dockerfile文件

root@master01:/data/mongodb# cat Dockerfile 
FROM bitnami/mongodb:4.0.14-debian-9-r24
USER root
RUN echo "tail -f /etc/hosts" > /setup.sh
RUN echo "tail -f /etc/hosts" > /run.sh

构建镜像&推送到一个具有公共读的镜像仓库中

root@master01:/data/mongodb# docker build -t xxx/xxx/mongodb-debug:v1 .

测试

root@master01:/data/mongodb# docker run -it xxx/xxx/mongodb-debug:v1 /bin/bash 
 11:15:17.58 
 11:15:17.58 Welcome to the Bitnami mongodb container
 11:15:17.58 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mongodb
 11:15:17.58 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mongodb/issues
 11:15:17.58 Send us your feedback at containers@bitnami.com
 11:15:17.58 

root@8af93d798c63:/# ls /bitnami/
mongodb
root@8af93d798c63:/# cd bitnami/
root@8af93d798c63:/bitnami# ls
mongodb

编辑部署文件

编辑mongodb的deployment文件，将自定义的mongodb镜像做为第二个容器镜像，并配置pvc

kubectl edit deployment mongodb

      containers:
        - name: mongodbdebug
          image: xxx/istio_test/mongodb-debug:v1
          volumeMounts:
          - name: data
            mountPath: /bitnami/mongodb

        - name: mongodb
          image: xxx/xxx/mongodb:4.0.14-debian-9-r24

进入自定义的容器内部执行修复

root@dev-k8s-master03:/data# kubectl exec -it mongodb-7f97f799d9-qz82v -c mongodbdebug -- bash 
root@mongodb-7f97f799d9-qz82v:/# ls bitnami/mongodb/data/db/
WiredTiger			      collection-11--5099670552419067160.wt  collection-4-5162049135609455324.wt    index-1--8902867768542019650.wt   index-2--5099670552419067160.wt	index-30--5099670552419067160.wt  index-44--5099670552419067160.wt  index-56--5099670552419067160.wt  index-7-5162049135609455324.w
.........

root@mongodb-7f97f799d9-qz82v:/bitnami/mongodb/data/db# mongod --dbpath /bitnami/mongodb/data/db --repair

2024-04-12T11:23:29.167+0000 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] MongoDB starting : pid=40 port=27017 dbpath=/bitnami/mongodb/data/db 64-bit host=mongodb-7f97f799d9-qz82v
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] db version v4.0.14
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] git version: 1622021384533dade8b3c89ed3ecd80e1142c132
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] modules: none
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] build environment:
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten]     distmod: debian92
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten]     distarch: x86_64
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/bitnami/mongodb/data/db" } }
2024-04-12T11:23:29.175+0000 I STORAGE  [initandlisten] Detected data files in /bitnami/mongodb/data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2024-04-12T11:23:29.175+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=31648M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2024-04-12T11:23:29.284+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1712921009:284845][40:0x7f7ccdc7d080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712921009:284845][40:0x7f7ccdc7d080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
省略日志.......
2024-04-12T11:23:34.236+0000 W STORAGE  [initandlisten] Modifications made by repair:
2024-04-12T11:23:34.236+0000 W STORAGE  [initandlisten]   WiredTiger metadata salvaged
2024-04-12T11:23:34.236+0000 I STORAGE  [initandlisten] finished checking dbs
2024-04-12T11:23:34.236+0000 I STORAGE  [initandlisten] WiredTigerKVEngine shutting down
2024-04-12T11:23:34.238+0000 I STORAGE  [initandlisten] Shutting down session sweeper thread
2024-04-12T11:23:34.238+0000 I STORAGE  [initandlisten] Finished shutting down session sweeper thread
2024-04-12T11:23:34.378+0000 I STORAGE  [initandlisten] shutdown: removing fs lock...
2024-04-12T11:23:34.380+0000 I CONTROL  [initandlisten] now exiting
2024-04-12T11:23:34.380+0000 I CONTROL  [initandlisten] shutting down with code:0

新的报错

mongodb提示读取数据没权限，看了一下是由于构建镜像时候用了root用户执行命令，导致修改后的文件的所有者是root，chown xxx mongodb数据库文件即可

验证

root@dev-k8s-master03:/data# kubectl get pod 
NAME                           READY   STATUS             RESTARTS   AGE
mongodb-68d99d8cf9-wlcn2       1/1     Running            0          2d16h
yapi-pod-6f889c4b68-6xgqc      1/1     Running            0          2d18h

yapi也能正常登录了。

refer

mongodb社区
https://www.mongodb.com/community/forums/t/my-mongodb-not-working-properly-wt-error-non-specific-wiredtiger-error/145434/4

官网修复意外断电导致的mongo报错
https://www.mongodb.com/zh-cn/docs/manual/tutorial/recover-data-following-unexpected-shutdown/

mongodb镜像层
https://hub.docker.com/layers/bitnami/mongodb/4.0.14-debian-9-r24/images/sha256-fd839bcba360b2f470d003706f21ad4b8b1306be0b74e3ab936ff69b72f2e5d3?context=explore

知本知至
关注
  • 5
    点赞
  • 踩
  • 5
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
docker环境下mongo副本集的部署及异常修复
baidu_39340547的博客
01-17 1288
最近更换了办公地点。部署在本地docker环境里的mongo数据库不能使用了。原因是本地的ip地址变更，以前的mongo副本集的配置需要更新。处理完后，索性重新记录一下mongo副本集在docker中的部署流程。
彻底修复Mongodb容器wt文件数据损坏
weixin_39245293的博客
09-05 907
[initandlisten] An incomplete repair has been detected! This is likely because a repair operation unexpectedly failed before completing. MongoDB will not start up again without --repair. [initandlisten] Fatal Assertion 50922 at src/mongo/db/storage/storag
MongoDB --repair WT_ERROR: non-specific WiredTiger error；potential hardware corruption, read checksu
最新发布
weixin_43098896的博客
07-09 631
只要wt文件还在，就有办法恢复，不要慌，不要二次破坏，不要二次破坏，不要二次破坏。
MongoDB 数据文件损坏修复救命repair与致命危险
www_xue_xi的博客
12-31 7156
最近，一客户单实例mongodb数据库，没有备份的情况下遇到了断电导致的数据文件损坏，由于客户业务需要 及数据的不敏感性，要求尽快恢复业务，使用了Mongdb的自动修复repair命令进行修复。可喜的是，帮助用户尽 快恢复了服务，可悲的是在客户可接受情况下相关数据文件内的数据丢失。这里，对这一过程做个总结，同时说明 repair后为什么数据丢失。 正常的mongodb数据查询 > show dbs; admin 0.000GB config 0.00...
Centos 7 LVM xfs文件系统修复
qq_39354297的博客
12-18 1122
情况1： [sda] Assuming drive cache: write through Internal error xfs XFS_WANT_CORRUPTED_GOTO at line 1662 of file fs/xfs/libxfs/xfs_alloc.c Caller xfs_free_extent+0x130 [xfs] Internal error xfs_tran...
记一次mongodb节点一直处于recovering状态问题修复
卢说
06-29 1509
刚开始启动的时候，该节点状态会处于STARTUP2的状态，这表名它正在从主节点复制数据，如果现在去查看节点监控，会发现其入口带宽占用比较大，相对于primary节点出口带宽也比较大。比如我这个节点是shard1，则三处shard1数据目录下的所有数据。意思就是说这个节点上的数据过于陈旧，无法实现主从同步。解决办法是先备份集群数据，然后再重做这个节点。该节点mongod服务日志一直在刷如下日志。mongoDB版本：5.0.4。那就是数据版本落后太多了。等数据同步完就正常了。
MongoDB异常修复
鳄鱼儿
01-28 2182
????前言 服务器异常断电，在服务器重启恢复后，发现mongodb启动异常，怀疑是由于断电mongodb异常退出造成的。 ????尝试方案 删除mongod.lock文件 在原数据路径下进行恢复，mongod --dbpath /pathToMongodb/db --repair 等待恢复完毕，重新启动mongodb，mongod -f /pathToMongodb/mongodb.conf 依照此步骤操作，mongodb成功恢复。 ????mongodb启动与关闭脚本 启动脚本 #!/bin/bash
K8s 部署 MongoDB（副本集）
02-03
MongoDB中的副本集(Replica Set)是一组维护相同数据集的mongod服务。 副本集可提供冗余和高可用性，是所有生产部署的基础。也可以说，副本集类似于有自动故障恢复功能的主从集群。通俗的讲就是用多台机器进行同一...
k8s 安装 mongodb 分片（Sharding）+ 副本集（Replica Set）
04-14
k8s 安装 MongoDB 分片（Sharding）+ 副本集（Replica Set） k8s 安装 MongoDB 分片（Sharding）+ 副本集（Replica Set）是结合 Kubernetes（k8s）和 MongoDB 实现高可用性和高性能的解决方案。本解决方案通过使用 ...
MongoDB-in-Kubernetes:在K8s集群中部署的MongoDB Sharded集群
05-02
在K8s集群中部署的MongoDB Sharded集群 用于Shard和Config服务器的Pod的部署： 此处，分片副本集存储实际数据或块（文档集合），而配置服务器副本集存储mongodb集群的元数据和配置信息。 在k8s集群中，我们创建了...
Mongodb启动命令参数中文说明
09-10
在管理 MongoDB 实例时，正确理解和使用启动命令参数至关重要。以下是对 MongoDB 启动命令参数的详细中文说明，包括基本配置、主/从参数以及 Sharding 分片选项。 **基本配置** 1. **--quiet**: 使 MongoDB 输出更...
Linux安装MongoDB启动及常见问题解决
01-09
MongoDB安装过程以及问题记录 一、MongoDB安装 下载MongoDB 官方地址：https://www.mongodb.com/download-center/community 可能需要注册一下我这里是Centos7系统所以选择的是RHEL7.0的TGZ,其他系统自行选择 如果不...
安装MongoDB4.0以上版本出现 Verify that you have sufficient privileges to start system services暴力处理法
hrainning的博客
10-18 9784
从第一次安装mongdb时的青涩，到现在它已经变成了4.0.2的版本了，于是我自然要来体验一番 可谁知道 它居然变了 我熟练的和当年一样，点点点 和当初一样，我才不要跟着它走，我要走出自己的一条路，(自定义安装路线) ？？？？ 什么情况 查阅各种资料后依然不行，是安装在英文名字的文件后呀，怎么会错？？ 后来 生气我只好把它放在根目录之下，这样总不会被名字困扰吧 然后...
mongodb WT_ERROR: non-specific WiredTiger error
wangjinyu124419的博客
11-11 3357
mac mongo版本降级后无法启动，日志错误如下： 2020-11-11T23:09:39.744+0800 E STORAGE [initandlisten] WiredTiger (-31802) [1605107379:744746][7046:0x11559ddc0], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTi
ocp 1Z0-042 1-60题解析
weixin_34362875的博客
02-27 519
1. Because of a power outage,instance failure has occurred. From what point in the redo log does recovery begin and where does it end?A. Current redo log and inactive redo logB. Checkpoint position to...
Oracle OCP ­042练习题
babymouse1212的博客
05-02 926
Exam  : Oracle 1Z0­042  Title  : Oracle Database 10g Administration I      1. In your Oracle 10g database , you have scheduled a job to update the optimizer statistics at 05:00 pm  every Friday. T
修改服务器时间mongodb崩溃,Mongodb 崩溃报错 Too many open files的问题解析
weixin_39995774的博客
08-05 572
在项目实际使用过程中，客户反馈能打开网页但无法登陆，第一时间感觉到应该是数据库服务器挂了，于是查看Mongodb数据库服务器日志，果不其然挂了。报错信息如下：2020-12-28T13:21:21.731+0800 E STORAGE [conn2624] WiredTiger error (24) [1609132881:731422][23581:0x7fe157189700], WT_SE...
修复mongodb启动失败问题
热门推荐
地址ch3nye.top
10-09 2万+
启动mongodb的时候遇到了以下几个问题，我在此处给出我的一些解决方法。 注意我的方法中的大部分命令在root权限下才能生效，所以建议修复数据库时使用root用户。 service mongodb start 失败 status显示failed root@machine ~ # service mongod start root@machine ~ # service mongod s...
k8s中部署MongoDB分片与副本集的详细教程
总结来说，要在k8s中安装MongoDB并实现分片和副本集，你需要设置NFS共享，创建PV，配置副本集的StatefulSet或Deployment，然后根据需要配置分片。这个过程涉及到多个组件的协同工作，包括存储、网络、服务发现和...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值