https://ceph.com/planet/%E5%85%B3%E4%BA%8Escrub%E7%9A%84%E8%AF%A6%E7%BB%86%E5%88%86%E6%9E%90%E5%92%8C%E5%BB%BA%E8%AE%AE/
AUGUST 19, 2017
关于scrub的详细分析和建议
![scrub](https://i0.wp.com/7xweck.com1.z0.glb.clouddn.com/scrub.png)
前言
关于scrub这块一直想写一篇文章的,这个在很久前,就做过一次测试,当时是看这个scrub到底有多大的影响,当时看到的是磁盘读占很高,启动deep-scrub后会有大量的读,前端可能会出现 slow request,这个是当时测试看到的现象,一个比较简单的处理办法就是直接给scrub关掉了,当然关掉了就无法检测底层到底有没有对象不一致的问题
关于这个scrub生产上是否开启,仁者见仁,智者见智,就是选择的问题了,这里不做讨论,个人觉得开和关都有各自的道理,本篇是讲述的如果想开启的情况下如何把scrub给控制住
最近在ceph群里看到一段大致这样的讨论:
scrub是个坑
小文件多的场景一定要把scrub关掉
单pg的文件量达到一定规模,scrub一开就会有slow request
这个问题解决不了
上面的说法有没有问题呢?在一般情况下来看,确实如此,但是我们是否能尝试去解决下这个问题,或者缓解下呢?那么我们就来尝试下
scrub的一些追踪
下面的一些追踪并不涉及代码,仅仅从配置和日志的观测来看看scrub到底干了什么
环境准备
我的环境为了便于观测,配置的是一个pg的存储池,然后往这个pg里面put了100个对象,然后对这个pg做deep-scrub,deep-scrub比scrub对磁盘的压力要大些,所以本篇主要是去观测的deep-scrub
开启对pg目录的访问的监控
使用的是inotifywait,我想看下deep-scrub的时候,pg里面的对象到底接收了哪些请求
inotifywait -m 1.0_head
1.0_head/ OPEN,ISDIR 1.0_head/ ACCESS,ISDIR 1.0_head/ ACCESS,ISDIR 1.0_head/ CLOSE_NOWRITE,CLOSE,ISDIR 1.0_head/ OPEN,ISDIR 1.0_head/ ACCESS,ISDIR 1.0_head/ ACCESS,ISDIR 1.0_head/ CLOSE_NOWRITE,CLOSE,ISDIR 1.0_head/ OPEN a16__head_8FA46F40__1 1.0_head/ ACCESS a16__head_8FA46F40__1 1.0_head/ OPEN a39__head_621FD720__1 1.0_head/ ACCESS a39__head_621FD720__1 1.0_head/ OPEN a30__head_655287E0__1 1.0_head/ ACCESS a30__head_655287E0__1 1.0_head/ OPEN a91__head_B02EE3D0__1 1.0_head/ ACCESS a91__head_B02EE3D0__1 1.0_head/ OPEN a33__head_9E9E3E30__1 1.0_head/ ACCESS a33__head_9E9E3E30__1 1.0_head/ OPEN a92__head_6AFC6B30__1 1.0_head/ ACCESS a92__head_6AFC6B30__1 1.0_head/ OPEN a22__head_AC48AAB0__1 1.0_head/ ACCESS a22__head_AC48AAB0__1 1.0_head/ OPEN a42__head_76B90AC8__1 1.0_head/ ACCESS a42__head_76B90AC8__1 1.0_head/ OPEN a5__head_E5A1A728__1 1.0_head/ ACCESS a5__head_E5A1A728__1 1.0_head/ OPEN a34__head_4D9ABA68__1 1.0_head/ ACCESS a34__head_4D9ABA68__1 1.0_head/ OPEN a69__head_7AF2B6E8__1 1.0_head/ ACCESS a69__head_7AF2B6E8__1 1.0_head/ OPEN a95__head_BD3695B8__1 1.0_head/ ACCESS a95__head_BD3695B8__1 1.0_head/ OPEN a67__head_6BCD37B8__1 1.0_head/ ACCESS a67__head_6BCD37B8__1 1.0_head/ OPEN a10__head_F0F08AF8__1 1.0_head/ ACCESS a10__head_F0F08AF8__1 1.0_head/ OPEN a3__head_88EF0BF8__1 1.0_head/ ACCESS a3__head_88EF0BF8__1 1.0_head/ OPEN a82__head_721BC094__1 1.0_head/ ACCESS a82__head_721BC094__1 1.0_head/ OPEN a48__head_27A729D4__1 1.0_head/ ACCESS a48__head_27A729D4__1 1.0_head/ OPEN a36__head_F63E6AF4__1 1.0_head/ ACCESS a36__head_F63E6AF4__1 1.0_head/ OPEN a29__head_F06D540C__1 1.0_head/ ACCESS a29__head_F06D540C__1 1.0_head/ OPEN a31__head_AC83164C__1 1.0_head/ ACCESS a31__head_AC83164C__1 1.0_head/ OPEN a59__head_884F9B6C__1 1.0_head/ ACCESS a59__head_884F9B6C__1 1.0_head/ OPEN a58__head_06954F6C__1 1.0_head/ ACCESS a58__head_06954F6C__1 1.0_head/ OPEN a55__head_2A42E61C__1 1.0_head/ ACCESS a55__head_2A42E61C__1 1.0_head/ OPEN a90__head_1B88FEDC__1 1.0_head/ ACCESS a90__head_1B88FEDC__1 1.0_head/ OPEN,ISDIR 1.0_head/ ACCESS,ISDIR 1.0_head/ ACCESS,ISDIR 1.0_head/ CLOSE_NOWRITE,CLOSE,ISDIR 1.0_head/ OPEN,ISDIR 1.0_head/ ACCESS,ISDIR 1.0_head/ ACCESS,ISDIR 1.0_head/ CLOSE_NOWRITE,CLOSE,ISDIR 1.0_head/ OPEN a100__head_C29E0C42__1 1.0_head/ ACCESS a100__head_C29E0C42__1 1.0_head/ OPEN a15__head_87123BE2__1 1.0_head/ ACCESS a15__head_87123BE2__1 1.0_head/ OPEN a23__head_AABFFB92__1 1.0_head/ ACCESS a23__head_AABFFB92__1 1.0_head/ OPEN a41__head_4EA9A5D2__1 1.0_head/ ACCESS a41__head_4EA9A5D2__1 1.0_head/ OPEN a85__head_83760D72__1 1.0_head/ ACCESS a85__head_83760D72__1 1.0_head/ OPEN a72__head_8A105D72__1 1.0_head/ ACCESS a72__head_8A105D72__1 1.0_head/ OPEN a60__head_5536480A__1 1.0_head/ ACCESS a60__head_5536480A__1 1.0_head/ OPEN a73__head_F1819D0A__1 1.0_head/ ACCESS a73__head_F1819D0A__1 1.0_head/ OPEN a78__head_6929D12A__1 1.0_head/ ACCESS a78__head_6929D12A__1 1.0_head/ OPEN a57__head_2C43153A__1 1.0_head/ ACCESS a57__head_2C43153A__1 1.0_head/ OPEN a1__head_51903B7A__1 1.0_head/ ACCESS a1__head_51903B7A__1 1.0_head/ OPEN a12__head_14D7ABC6__1 1.0_head/ ACCESS a12__head_14D7ABC6__1 1.0_head/ OPEN a63__head_9490B166__1 1.0_head/ ACCESS a63__head_9490B166__1 1.0_head/ OPEN a53__head_DF95B716__1 1.0_head/ ACCESS a53__head_DF95B716__1 1.0_head/ OPEN a13__head_E09E0896__1 1.0_head/ ACCESS a13__head_E09E0896__1 1.0_head/ OPEN a27__head_7ED31896__1 1.0_head/ ACCESS a27__head_7ED31896__1 1.0_head/ OPEN a43__head_7052A656__1 1.0_head/ ACCESS a43__head_7052A656__1 1.0_head/ OPEN a28__head_E6257CD6__1 1.0_head/ ACCESS a28__head_E6257CD6__1 1.0_head/ OPEN a35__head_ACABD736__1 1.0_head/ ACCESS a35__head_ACABD736__1 1.0_head/ OPEN a54__head_B9482876__1 1.0_head/ CLOSE_WRITE,CLOSE a12__head_14D7ABC6__1 1.0_head/ ACCESS a54__head_B9482876__1 1.0_head/ OPEN a4__head_F12ACA76__1 1.0_head/ CLOSE_WRITE,CLOSE a63__head_9490B166__1 1.0_head/ ACCESS a4__head_F12ACA76__1 1.0_head/ OPEN a84__head_B033038E__1 1.0_head/ ACCESS a84__head_B033038E__1 1.0_head/ OPEN a19__head_D6A64F9E__1 1.0_head/ ACCESS a19__head_D6A64F9E__1 1.0_head/ OPEN a93__head_F54E757E__1 1.0_head/ ACCESS a93__head_F54E757E__1 1.0_head/ OPEN a7__head_1F08F77E__1 1.0_head/ ACCESS a7__head_1F08F77E__1 1.0_head/ OPEN,ISDIR 1.0_head/ ACCESS,ISDIR 1.0_head/ ACCESS,ISDIR 1.0_head/ CLOSE_NOWRITE,CLOSE,ISDIR
|