前言
服务出现 OOM 导致异常退出，对于8G规格实例，在不高 QPS 的场景下偶发出现爆内存的情况，进而整体走查下服务的内存使用情况。期间用到很多工具，记录下使用过程。
net/http/pprof
go 自 1.11 后自带 Debug 工具，能够抓取 Runtime 时的资源开销表现,net/http/pprof 提供了开启一个 http 服务来监测服务Runtime 开销，具体参见： pprof。项目工程已经打开debug 端口，可以直接访问本地页面：http://localhost:8080/debug/pprof/
每一个资源链接，对应统计的是何种使用资源，在 Profile Description 中有详细说明。
注意区分下 allocs 和 heap 。前者是服务过去分配的内存总和，后者是当前存活对象堆内存占用。
查看 allocs 页面内容：
heap profile: 27: 3474048 [1962: 163599632] @ heap/1048576
1: 1474560 [1: 1474560] @ 0x49e2750 0x49f7757 0x404a16a 0x404a137 0x404a137 0x403cde5 0x4073741
# 0x49e274f golang.org/x/net/webdav.(*memFile).Write+0x28f /.../go/pkg/mod/golang.org/x/net@v0.0.0-20210226172049-e18ecbb05110/webdav/file.go:600
# 0x49f7756 github.com/swaggo/files.init.8+0xd6 /.../go/pkg/mod/github.com/swaggo/files@v0.0.0-20190704085106-630677cd5c14/b0xfile__swagger-ui-standalone-preset.js.map.go:21
# 0x404a169 runtime.doInit+0x89 /usr/local/go/src/runtime/proc.go:5652
# 0x404a136 runtime.doInit+0x56 /usr/local/go/src/runtime/proc.go:5647
# 0x404a136 runtime.doInit+0x56 /usr/local/go/src/runtime/proc.go:5647
# 0x403cde4 runtime.main+0x1c4
......
# runtime.MemStats
# Alloc = 25980872
# TotalAlloc = 1169832720
# Sys = 147410952
# Lookups = 0
# Mallocs = 22502239
# Frees = 22387686
# HeapAlloc = 25980872
# HeapSys = 132448256
# HeapIdle = 100294656
# HeapInuse = 32153600
# HeapReleased = 90972160
# HeapObjects = 114553
# Stack = 1769472 / 1769472
# MSpan = 351696 / 573440
# MCache = 20832 / 32768
# BuckHashSys = 1562838
# GCSys = 8410504
# OtherSys = 2613674
# NextGC = 48638368
# LastGC = 1634028715094099000
# PauseNs = [72905 55586 55783 253229 198362 79655 88079 87972 92039 95024 181327 110309 104561 86640 69906 311533 100288 80743 92016 95645 157123 137269 71777 110670 94478 62426 116399 94126 303704 138511 128189 97607 90788 75073 177091 124844 84018 76317 217243 150808 159977 297082 160933 107545 75450 91570 62496 5678518 224132 109622 102157 134959 80755 89098 133107 136055 75341 102839 122052 161542 131019 195743 959214 122872 107377 138324 70146 167725 125591 107987 115333 237980 213628 122926 200681 194209 123568 142651 130335 146937 141924 92132 111528 162683 102550 64273 78349 132196 138162 126515 123264 79881 233103 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
# PauseEnd = [1634018078703895000 1634018078711267000 1634018078727372000 1634018079689849000 1634018079855146000 1634018079903099000 1634018079945718000 1634018079982788000 1634018080020130000 1634018136566915000 1634018256570900000 1634018376570507000 1634018491810610000 1634018522267616000 1634018522979170000 1634018522988692000 1634018523849314000 1634018525312684000 1634018525344248000 1634018645347500000 1634018765350802000 1634018885358357000 1634019005369751000 1634019125380340000 1634019245391095000 1634019365407786000 1634019485449801000 1634019605459575000 1634019725472042000 1634019845482585000 1634019965489782000 1634020085502764000 1634020205509918000 1634020325548461000 1634020445563112000 1634020565574554000 1634020685583551000 1634020805597814000 1634020925610417000 1634021045620936000 1634021165539576000 1634021285548151000 1634021405555310000 1634021525567319000 1634021645576323000 1634021765585616000 1634021885591717000 1634022032221439000 1634022152233114000 1634022272237286000 1634022392242686000 1634022512246817000 1634022632257575000 1634022752269648000 1634022872279633000 1634022992292542000 1634023112301074000 1634023232335305000 1634023352339784000 1634023472349032000 1634023592357829000 1634024280656911000 1634025114794129000 1634025234791336000 1634025354795624000 1634025474801160000 1634025594810318000 1634025714820341000 1634025834831343000 1634025954840401000 1634026074849385000 1634026194852159000 1634026314855789000 1634026434870330000 1634026554878401000 1634026674909917000 1634026794921428000 1634026914932317000 1634027034937975000 1634027154947598000 1634027274960399000 1634027394970137000 1634027514980497000 1634027634995161000 1634027755005894000 1634027875007687000 1634027995016117000 1634028115026150000 1634028235039196000 1634028355047826000 1634028475055432000 1634028595086564000 1634028715094099000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
# NumGC = 93
# NumForcedGC = 0
# GCCPUFraction = 8.631619696555182e-06
# DebugGC = false
# MaxRSS = 144908288
第一行的统计信息来自 mprof.go 中的内存 profiling 统计
heap profile: 27(inused object number): 3474048(inused bytes) [1962(alloc objects number): 163599632(alloc bytes)] @ heap/1048576
这段内容就是 mprof.go 中 MemProfileRecord 内容
var MemProfileRate int = 512 * 1024
// A MemProfileRecord describes the live objects allocated
// by a particular call sequence (stack trace).
type MemProfileRecord struct {
AllocBytes, FreeBytes int64 // number of bytes allocated, freed
AllocObjects, FreeObjects int64 // number of objects allocated, freed
Stack0 [32]uintptr // stack trace for this record; ends at first 0 entry
}
@ heap/1048576 中的 1048576 = 2*MemProfileRate
底部的runtime.MemStats 可以查阅 mstats.go
// A MemStats records statistics about the memory allocator.
type MemStats struct {
// General statistics.
Alloc uint64 // bytes allocated and not yet freed
TotalAlloc uint64 // bytes allocated (even if freed)
Sys uint64 // bytes obtained from system (sum of XxxSys below)
Lookups uint64 // number of pointer lookups
Mallocs uint64 // number of mallocs
Frees uint64 // number of frees
// Main allocation heap statistics.
HeapAlloc uint64 // bytes allocated and not yet freed (same as Alloc above)
HeapSys uint64 // bytes obtained from system
HeapIdle uint64 // bytes in idle spans
HeapInuse uint64 // bytes in non-idle span
HeapReleased uint64 // bytes released to the OS
HeapObjects uint64 // total number of allocated objects
// Low-level fixed-size structure allocator statistics.
// Inuse is bytes used now.
// Sys is bytes obtained from system.
StackInuse uint64 // bytes used by stack allocator
StackSys uint64
MSpanInuse uint64 // mspan structures
MSpanSys uint64
MCacheInuse uint64 // mcache structures
MCacheSys uint64
BuckHashSys uint64 // profiling bucket hash table
GCSys uint64 // GC metadata
OtherSys uint64 // other system allocations
// Garbage collector statistics.
NextGC uint64 // next collection will happen when HeapAlloc ≥ this amount
LastGC uint64 // end time of last collection (nanoseconds since 1970)
PauseTotalNs uint64
PauseNs [256]uint64 // circular buffer of recent GC pause durations, most recent at [(NumGC+255)%256]
PauseEnd [256]uint64 // circular buffer of recent GC pause end times
NumGC uint32
GCCPUFraction float64 // fraction of CPU time used by GC
EnableGC bool
DebugGC bool
// Per-size allocation statistics.
// 61 is NumSizeClasses in the C code.
BySize [61]struct {
Size uint32
Mallocs uint64
Frees uint64
}
}
go tool pprof
go tool 中包含很多命令，可以参考官方介绍，也可以参见命令梳理。
承接上文中开启的服务 debug ，可以利用 go tool pprof 进一步查看，建议查看一下 go tool pprof，可以有两种使用姿势，
- 直接默认连接查看的debug http 服务进程，以
interactive shell的形式查看
- 另一种则是生成图片
go tool pprof -alloc_space -cum -svg http://ip:port/debug/pprof/heap > heap.svg
本地调试过程中，发现服务单个数据接口请求能够达到 800M 的内存占用：
这其中面对一些稍微大些的数据获取统计时的主要内存开销，可见并不是用在服务的统计上。
go tool 生成图片时要求安装 graphviz，MacOS 可以直接
brew install graphviz
在 go tool pprof 中提供了一些保留便捷 option 选项
Legacy convenience options:
-inuse_space Same as -sample_index=inuse_space
-inuse_objects Same as -sample_index=inuse_objects
-alloc_space Same as -sample_index=alloc_space
-alloc_objects Same as -sample_index=alloc_objects
-total_delay Same as -sample_index=delay
-contentions Same as -sample_index=contentions
-mean_delay Same as -mean -sample_index=delay
通常情况下：
- 用–inuse_space来分析程序常驻内存的占用情况
- 用–alloc_objects来分析内存的临时分配情况，可以提高程序的运行速度
go-torch
上面调用关系图如果觉得不够直观，还可以利用 go-torch 进行火焰图绘制
安装
go get github.com/uber/go-torch
cd $GOPATH/src/github.com/uber/go-torch
# go-torch 需要 FlameGraph
git clone https://github.com/brendangregg/FlameGraph.git
cp ./FlameGraph/flamegraph.pl /usr/local/bin
使用方式
go-torch -t 10 -alloc_space http://localhost:port/debug/pprof/heap --colors=mem
默认 30s 采样，实际效果：
内存优化
由 pprof 的统计图可以看出服务内存开销主要集中在
- 本地 http client 中的 response dump
- []byte 转化，实质是很多字符串log信息
对应优化：
- http client 的 response dump后的信息，绝大多数都是无效信息，并且开销大，hold内存多，调整client中的信息记录，在http status code 未非正常时 dump response
- 调整日志，关闭无效日志，减少了字符串拼接转化
- 服务本地大变量优化，在一些统计时会有几万个待统计对象的 slice，此部分需要根据 slice 中元素判定元素内容做一些逻辑处理，先前的实现重复 2次 copy 了这个大slice内容，转而用下标 切片，全局维持一份 slice
优化后的内存表现
766
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
