dell4528s linux_linux – 使用RHEL6的12G戴尔服务器上的“功率限制通知”

服务器：Poweredge r620

操作系统：RHEL 6.4

内核：2.6.32-358.18.1.el6.x86_64

我在生产环境中遇到应用程序警报.关键的CPU饥饿进程正在缺乏资源并导致处理积压.问题出现在最近部署的集群中的所有第12代戴尔服务器(r620)上.据我所知,这种情况的实例与CPU峰值的使用率相匹配,并伴随着dmesg中的大量“功率限制通知”垃圾邮件.其中一个事件的摘录：

Nov 7 10:15:15 someserver [.crit] CPU12: Core power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU0: Core power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU6: Core power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU14: Core power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU18: Core power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU2: Core power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU4: Core power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU16: Core power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU0: Package power limit notification (total events = 11)

Nov 7 10:15:15 someserver [.crit] CPU6: Package power limit notification (total events = 13)

Nov 7 10:15:15 someserver [.crit] CPU14: Package power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU18: Package power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU20: Core power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU8: Core power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU2: Package power limit notification (total events = 12)

Nov 7 10:15:15 someserver [.crit] CPU10: Core power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU22: Core power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU4: Package power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU16: Package power limit notification (total events = 13)

Nov 7 10:15:15 someserver [.crit] CPU20: Package power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU8: Package power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU10: Package power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU22: Package power limit notification (total events = 14)

Nov 7 10:15:15 someserver [.crit] CPU15: Core power limit notification (total events = 369)

Nov 7 10:15:15 someserver [.crit] CPU3: Core power limit notification (total events = 369)

Nov 7 10:15:15 someserver [.crit] CPU1: Core power limit notification (total events = 369)

Nov 7 10:15:15 someserver [.crit] CPU5: Core power limit notification (total events = 369)

Nov 7 10:15:15 someserver [.crit] CPU17: Core power limit notification (total events = 369)

Nov 7 10:15:15 someserver [.crit] CPU13: Core power limit notification (total events = 369)

Nov 7 10:15:15 someserver [.crit] CPU15: Package power limit notification (total events = 375)

Nov 7 10:15:15 someserver [.crit] CPU3: Package power limit notification (total events = 374)

Nov 7 10:15:15 someserver [.crit] CPU1: Package power limit notification (total events = 376)

Nov 7 10:15:15 someserver [.crit] CPU5: Package power limit notification (total events = 376)

Nov 7 10:15:15 someserver [.crit] CPU7: Core power limit notification (total events = 369)

Nov 7 10:15:15 someserver [.crit] CPU19: Core power limit notification (total events = 369)

Nov 7 10:15:15 someserver [.crit] CPU17: Package power limit notification (total events = 377)

Nov 7 10:15:15 someserver [.crit] CPU9: Core power limit notification (total events = 369)

Nov 7 10:15:15 someserver [.crit] CPU21: Core power limit notification (total events = 369)

Nov 7 10:15:15 someserver [.crit] CPU23: Core power limit notification (total events = 369)

Nov 7 10:15:15 someserver [.crit] CPU11: Core power limit notification (total events = 369)

Nov 7 10:15:15 someserver [.crit] CPU13: Package power limit notification (total events = 376)

Nov 7 10:15:15 someserver [.crit] CPU7: Package power limit notification (total events = 375)

Nov 7 10:15:15 someserver [.crit] CPU19: Package power limit notification (total events = 375)

Nov 7 10:15:15 someserver [.crit] CPU9: Package power limit notification (total events = 374)

Nov 7 10:15:15 someserver [.crit] CPU21: Package power limit notification (total events = 375)

Nov 7 10:15:15 someserver [.crit] CPU23: Package power limit notification (total events = 374)

一个小小的谷福显示,这通常与CPU运行热,或电压调节有关.我不认为这是发生了什么.群集中所有服务器的温度传感器运行正常,iDRAC中禁用了功率上限策略,并且我的系统配置文件在所有这些服务器上都设置为“性能”：

# omreport chassis biossetup | grep -A10 'System Profile'

System Profile Settings

------------------------------------------

System Profile : Performance

CPU Power Management : Maximum Performance

Memory Frequency : Maximum Performance

Turbo Boost : Enabled

C1E : Disabled

C States : Disabled

Monitor/Mwait : Enabled

Memory Patrol Scrub : Standard

Memory Refresh Rate : 1x

Memory Operating Voltage : Auto

Collaborative CPU Performance Control : Disabled

> A Dell mailing list post几乎完美地描述了症状.戴尔建议作者尝试使用性能配置文件,但这没有帮助.他最终在Dell’s guide for configuring a server for low latency environments中应用了一些设置,其中一个设置(或其组合)似乎解决了问题.

> Kernel.org bug #36182注意到默认情况下启用了功率限制中断调试,这会导致CPU电压调节进入的情况下性能下降.

> A RHN KB article(需要RHN登录)提到影响PE r620和r720服务器未运行性能配置文件的问题,并建议更新两周前发布的内核. …除了我们正在运行性能配置文件…

我在网上找到的所有东西都在这里让我圈起来.到底发生了什么事？