gerrit Git仓库数据损坏修复

一、起因

研发同事克隆项目到本地开发后提交PR到gerrit代码仓库报错，提示：error: remote unpack failed: error Short read of block.；

root@local:/demo$ git push origin HEAD:refs/for/master
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 303 bytes | 303.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2)
error: remote unpack failed: error Short read of block.
fatal: Unpack error, check server log
To ssh://192.168.0.1:29418/demo
 ! [remote rejected]     HEAD -> refs/for/master (n/a (unpacker error))
error: failed to push some refs to 'ssh://admin@192.168.0.1:29418/demo'

根据提示判断缺失了某些东西，登录gerrit服务器查看error.log错误日志如下：

[2023-11-23 11:51:43,459] [SSH git-receive-pack '/demo' (admin)] WARN  org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file /usr/local/gerrit/git/demo.git/objects/pack/pack-eb9877a005b673cd5e9392327c552651fcbf18fd.pack was deleted, removing it from pack list[2023-11-23 11:51:43,931] [SSH git-receive-pack '/demo' (admin)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user admin account 1) during git-receive-pack '/demo'
com.google.gerrit.sshd.BaseCommand$Failure: fatal: Unpack error, check server log
        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:159)
        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)
        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:442)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:417)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException: Unpack error on project "demo":
  AdvertiseRefsHook: org.eclipse.jgit.transport.AdvertiseRefsHookChain@d0badc2class org.eclipse.jgit.transport.AdvertiseRefsHookChain
        
        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:158)
        ... 12 more
Caused by: org.eclipse.jgit.errors.UnpackException: Exception while parsing pack stream
        at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:307)
        at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:206)
        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:97)
        ... 12 more
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown cf1ce10a08b7c5fb3e0cc24561f51292bcb9d1f9
        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:158)
        at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:227)
        at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:859)
        at org.eclipse.jgit.transport.BaseReceivePack.checkConnectivity(BaseReceivePack.java:1354)
        at org.eclipse.jgit.transport.BaseReceivePack.receivePackAndCheckConnectivity(BaseReceivePack.java:1047)
        at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:250)
        ... 14 more

二、排查思路

1、本地客户端.git是否有损坏的文件，创建新的空项目，修改远程仓库地址，将现有项目推送到新的远程仓库

# 克隆有问题的项目到本地
root@local:/demo$ git clone ssh:admin@192.168.0.1/demo
# 修改远程地址到新项目
root@local:/demo$ git remote set-url origin ssh://admin@192.168.0.1/demo2
# 推送现有项目到
root@local:/demo$ git push -u origin --all

结果：克隆到本地项目代码能推送到新项目仓库且能做修改重新提交PR，排除.git有损坏的文件

2、gerrit服务端git仓库是否有损坏的文件，登录gerrit进入git仓库项目下，执行git fsck --full --no-dangling检查

# cd到gerrit git仓库中demo有问题的仓库
root@gerrit:/demo.git$ cd /usr/local/gerrit/git/demo.git/
# 执行git自带的检查命令
root@gerrit:/demo.git$ git fsck --full --no-dangling
error: object file ./objects/2a/cdc20d19d8cae08ed8adb741511139bb316b86 is empty
error: unable to mmap ./objects/2a/cdc20d19d8cae08ed8adb741511139bb316b86: No such file or directory
error: 2acdc20d19d8cae08ed8adb741511139bb316b86: object corrupt or missing: ./objects/2a/cdc20d19d8cae08ed8adb741511139bb316b86
error: object file ./objects/cf/1ce10a08b7c5fb3e0cc24561f51292bcb9d1f9 is empty
error: unable to mmap ./objects/cf/1ce10a08b7c5fb3e0cc24561f51292bcb9d1f9: No such file or directory
error: cf1ce10a08b7c5fb3e0cc24561f51292bcb9d1f9: object corrupt or missing: ./objects/cf/1ce10a08b7c5fb3e0cc24561f51292bcb9d1f9
error: object file ./objects/de/08318ad845bd960aedf0ab1ce85fc6e26f608f is empty
error: unable to mmap ./objects/de/08318ad845bd960aedf0ab1ce85fc6e26f608f: No such file or directory
error: de08318ad845bd960aedf0ab1ce85fc6e26f608f: object corrupt or missing: ./objects/de/08318ad845bd960aedf0ab1ce85fc6e26f608f
error: object file ./objects/df/8aded019cb1b23af4a4f3c5171472e76461a56 is empty
error: unable to mmap ./objects/df/8aded019cb1b23af4a4f3c5171472e76461a56: No such file or directory
error: df8aded019cb1b23af4a4f3c5171472e76461a56: object corrupt or missing: ./objects/df/8aded019cb1b23af4a4f3c5171472e76461a56
Checking object directories: 100% (256/256), done.
Checking objects: 100% (389135/389135), done.
error: object file ./objects/cf/1ce10a08b7c5fb3e0cc24561f51292bcb9d1f9 is empty
error: object file ./objects/cf/1ce10a08b7c5fb3e0cc24561f51292bcb9d1f9 is empty
fatal: loose object cf1ce10a08b7c5fb3e0cc24561f51292bcb9d1f9 (stored in ./objects/cf/1ce10a08b7c5fb3e0cc24561f51292bcb9d1f9) is corrupt
# 进一步确认git cat-file可以用来实现所有Git对象的读取，包括数据对象、树对象、提交对象的查看。
root@gerrit:/demo.git$ git cat-file -p cf1ce10a08b7c5fb3e0cc24561f51292bcb9d1f9
error: object file ./objects/cf/1ce10a08b7c5fb3e0cc24561f51292bcb9d1f9 is empty
fatal: Not a valid object name cf1ce10a08b7c5fb3e0cc24561f51292bcb9d1f9

结果：通过git fsck检查发现gerrit git仓库中Objects有损坏的哈希值文本

三、修复损坏对象文件

1、安装修复工具git-repair

git-repair (branchable.com)

工具怎么工作：

        Git-repair首先删除所有损坏的对象，并从远程存储库检索所有丢失的对象。

root@gerrit:/demo.git$ apt-get install -y git-repair

2、检查修复

# 修复之前先备份
root@gerrit:/demo.git$ cp -r demo.git demo_bak.git
# cd到损坏的项目
root@gerrit:/demo.git$ cd /usr/local/gerrit/git/demo.git/
# 再次检查确认有损坏文件
root@gerrit:/demo.git$ git fsck --full --no-dangling
# 使用git-repair修复工具
root@gerrit:/demo.git$ git-repair
....
Initialized empty Git repository in /tmp/tmprepoiYz5H6/.git/
1 missing objects could not be recovered!
If you have a clone of this bare repository, you should add it as a remote of this repository, and retry.
If there are no clones of this repository, you can instead retry with the --force parameter to force recovery to a possibly usable state.

• 这时切换到本地客户端尝试执行git push推送查看报错信息

root@local:/demo$ git push origin HEAD:refs/for/master
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 303 bytes | 303.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2)
error: remote unpack failed: error Missing unknown cf1ce10a08b7c5fb3e0cc24561f51292bcb9d1f9
fatal: Unpack error, check server log
To ssh://192.168.0.1:29418/demo
 ! [remote rejected]     HEAD -> refs/for/master (n/a (unpacker error))
error: failed to push some refs to 'ssh://admin@192.168.0.1:29418/demo'

报错提示变了，因为gerrit git仓库执行修复工具命令后会删除损坏的对象，本地.git和gerrit git仓库对不上，忽略继续进一步修复

• 切换到gerrit git仓库执行git prune修剪命令将从仓库中删除无效的对象文件

root@gerrit:/demo.git$ git prune

• 继续执行清理Git存储，此命令会清理没有链接的对象文件和垃圾文件

root@gerrit:/demo.git$ git gc
Counting objects: 191602, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (184020/184020), done.
Writing objects: 100% (191602/191602), done.
Total 191602 (delta 71568), reused 1 (delta 0)
Removing duplicate objects: 100% (256/256), done.

执行以上操作后修复工作结束，开始验证修复结果

3、验证修复

• 切换到本地客户端提交验证修复是否成功

root@local:/demo$ touch 1.txt
root@local:/demo$ git add .
root@local:/demo$ git commit -m "add 1.txt"
[master 9a6591be6] add 1.txt
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 1.txt
root@local:/demo$ git push origin HEAD:refs/for/master
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 303 bytes | 303.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2)
remote: Processing changes: refs: 1, done    
remote: ERROR: [cc2bf52] missing Change-Id in commit message footer
remote: 
remote: Hint: To automatically insert Change-Id, install the hook:
remote:   gitdir=$(git rev-parse --git-dir); scp -p -P 29418 admin@192.168.0.1:hooks/commit-msg ${gitdir}/hooks/
remote: And then amend the commit:
remote:   git commit --amend
remote: 
To ssh://192.168.0.1:29418/demo
 ! [remote rejected]     HEAD -> refs/for/master ([cc2bf52] missing Change-Id in commit message footer)
error: failed to push some refs to 'ssh://admin@192.168.0.1:29418/demo

原因：项目仓库.git/hooks目录下，commit-msg文件缺失。

如上所示，依次在项目路径下输入如下命令，即可解决

root@local:/demo$ gitdir=$(git rev-parse --git-dir); scp -p -P 29418 admin@192.168.0.1:hooks/commit-msg ${gitdir}/hooks/
root@local:/demo$ git commit --amend
# 再次推送PR成功解决问题
root@local:/demo$ git push origin HEAD:refs/for/master
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Delta compression using up to 8 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 296 bytes | 296.00 KiB/s, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1)
remote: Processing changes: new: 1, refs: 1, done    
remote: 
remote: New Changes:
remote:   http://192.16.0.1/1 1
remote: 
To ssh://192.16.0.1:29418/demo
 * [new branch]          HEAD -> refs/for/master