线上问题排查 too many open files

最新推荐文章于 2024-08-08 02:00:59 发布
额仑白狼
最新推荐文章于 2024-08-08 02:00:59 发布
阅读量1.9k
点赞数
分类专栏：程序员自我修养 java 文章标签： java 线上问题排查
本文链接：https://blog.csdn.net/a544879146/article/details/109060394
版权
java 同时被 2 个专栏收录
8 篇文章 0 订阅
订阅专栏
程序员自我修养
4 篇文章 0 订阅
订阅专栏
场景描述
单体应用改造成分布式（未知原因，埋下暗雷），旧应用每日定时重启，所以实际不会存在too many open files。分布式应用基本每周发布一次，要扛过一周，逐渐运行会报 too many open files 异常。改造的新系统因为办公室政治原因必须有突破，不能也每天重启。
分析：高并发文件读，单个文件读写有上限。
常规处理思路
1、操作系统局部最大文件打开数
-个进程能打开的文件数(操作系统中-一个网络连接，也算是文件打开数)
ulimit-n查看限制
修改限制: /etc/security/limits.conf
2、操作系统全局最大文件打开数
查看限制: cat /proc/sys/fs/file-nr
修改限制: echo 1200000 > /proc/sys/fs/file-max
如图，前一天部署后，第二天8点多开始，业务繁忙应用打开文件数逐渐爆炸
实际处理思路：查询当前应用进程打开文件（句柄）数量，具体分析后再处理
一、查询
通过命令ulimit -a可以查看当前系统设置的最大句柄数是多少：
lsof -p 进程id | wc -l可以统计进程打开了多少文件
strace -p
ulimit -n 512
cat /proc/pid/limits
参数说明 https://www.cnblogs.com/bsjl/p/8302468.html
COMMAND   PID   USER   FD      TYPE             DEVICE  SIZE/OFF     NODE NAME
java    12690 mwuser  cwd       DIR              253,0      8192     2394 /home/mwuser
java    12690 mwuser  rtd       DIR              253,0       238       64 /
java    12690 mwuser  txt       REG              253,0      7734 50911369 /mwbase/jdk/jdk1.8.0_171/bin/java
java    12690 mwuser  mem       REG              253,0     88776       84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
java    12690 mwuser  mem       REG              253,0    257982 51104085 /mwbase/jdk/jdk1.8.0_171/jre/lib/amd64/libsunec.so
java    12690 mwuser  mem       REG              253,0    105824     1575 /usr/lib64/libresolv-2.17.so
java    12690 mwuser  mem       REG              253,0     31408     1531 /usr/lib64/libnss_dns-2.17.so
java    12690 mwuser  mem       REG              253,0 106070960 16783980 /usr/lib/locale/locale-archive
java    12690 mwuser  mem       REG              253,0     93308 51106339 /mwbase/jdk/jdk1.8.0_171/jre/lib/amd64/libnio.so
java    12690 mwuser  mem       REG              253,0    115485 51106336 /mwbase/jdk/jdk1.8.0_171/jre/lib/amd64/libnet.so
java    12690 mwuser  mem       REG              253,0     50353 51106344 /mwbase/jdk/jdk1.8.0_171/jre/lib/amd64/libmanagement.so
java    12690 mwuser  mem       REG              253,0  18006420 51668271 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/skywalking-agent.jar
java    12690 mwuser  mem       REG              253,0    277834 17928343 /mwbase/jdk/jdk1.8.0_171/jre/lib/ext/sunjce_provider.jar
java    12690 mwuser  mem       REG              253,0   1178928 17928342 /mwbase/jdk/jdk1.8.0_171/jre/lib/ext/localedata.jar
java    12690 mwuser  mem       REG              253,0  18480356 17928336 /mwbase/jdk/jdk1.8.0_171/jre/lib/ext/jfxrt.jar
java    12690 mwuser  mem       REG              253,0     41672 17928333 /mwbase/jdk/jdk1.8.0_171/jre/lib/ext/sunec.jar
java    12690 mwuser  mem       REG              253,0    115651 17931797 /mwbase/jdk/jdk1.8.0_171/jre/lib/jce.jar
java    12690 mwuser  mem       REG              253,0   3860502 17928335 /mwbase/jdk/jdk1.8.0_171/jre/lib/ext/cldrdata.jar
java    12690 mwuser  mem       REG              253,0   3135615 17928311 /mwbase/jdk/jdk1.8.0_171/jre/lib/charsets.jar
java    12690 mwuser  mem       REG              253,0   2023884 17928338 /mwbase/jdk/jdk1.8.0_171/jre/lib/ext/nashorn.jar
java    12690 mwuser  mem       REG              253,0    660559 17928331 /mwbase/jdk/jdk1.8.0_171/jre/lib/jsse.jar
java    12690 mwuser  mem       REG              253,0   3505295 17928322 /mwbase/jdk/jdk1.8.0_171/jre/lib/resources.jar
java    12690 mwuser  mem       REG              253,0     35650 51668270 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/activations/apm-toolkit-trace-activation-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     35575 51668269 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/activations/apm-toolkit-opentracing-activation-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     26096 51668268 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/activations/apm-toolkit-logback-1.x-activation-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     21843 51668267 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/activations/apm-toolkit-log4j-2.x-activation-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     16759 51668266 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/activations/apm-toolkit-log4j-1.x-activation-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     17473 51668264 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-armeria-0.8.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     22564 51668263 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-netty-socketio-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     26953 51668262 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-pulsar-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     24199 51668261 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-ehcache-2.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     13391 51668260 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-light4j-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     24645 51668259 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-cassandra-java-driver-3.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     20430 51668258 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-solrj-7.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     15883 51668257 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/resteasy-server-3.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     46564 51668256 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-vertx-core-3.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     17195 51668255 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/dubbo-2.7.x-conflict-patch-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     13413 51668254 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-dubbo-2.7.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     18875 51668253 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-canal-1.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     16922 51668252 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/dubbo-conflict-patch-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     21120 51668251 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-rabbitmq-5.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     34383 51668250 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-undertow-2.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     37170 51668249 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-elasticsearch-6.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     34406 51668248 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-elasticsearch-5.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     21202 51668247 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-activemq-5.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     17440 51668246 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/sofa-rpc-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     30118 51668245 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-hystrix-1.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     17807 51668244 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-servicecomb-java-chassis-1.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     17543 51668243 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-servicecomb-java-chassis-0.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     39591 51668242 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-kafka-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     28035 51668241 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-httpasyncclient-4.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     34259 51668240 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-mongodb-2.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     12357 51668239 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-elastic-job-2.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     31285 34536999 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-rocketmq-4.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     31465 34422729 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-rocketmq-3.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     47162 44550847 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-postgresql-8.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     22185 44550846 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-h2-1.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     27583 44550845 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-mysql-8.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     32546 44550844 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-mysql-6.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     30937 44550843 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-mysql-5.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     20364 44550842 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-mysql-commons-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     34608 44550841 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-grpc-1.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     21597 44550840 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-xmemcached-2.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     22941 44550839 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-sharding-sphere-4.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     22593 44550838 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-sharding-sphere-3.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     18833 44550837 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-sharding-jdbc-1.5.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     15286 44550836 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-spymemcached-2.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     13380 44550835 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-jetty-client-9.0-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0  66327620 17928313 /mwbase/jdk/jdk1.8.0_171/jre/lib/rt.jar
java    12690 mwuser  mem       REG              253,0    128794 51106332 /mwbase/jdk/jdk1.8.0_171/jre/lib/amd64/libzip.so
java    12690 mwuser  mem       REG              253,0     61624     1533 /usr/lib64/libnss_files-2.17.so
java    12690 mwuser  mem       REG              253,0     49341 51106341 /mwbase/jdk/jdk1.8.0_171/jre/lib/amd64/libinstrument.so
java    12690 mwuser  mem       REG              253,0    226512 51106326 /mwbase/jdk/jdk1.8.0_171/jre/lib/amd64/libjava.so
java    12690 mwuser  mem       REG              253,0     66472 51106331 /mwbase/jdk/jdk1.8.0_171/jre/lib/amd64/libverify.so
java    12690 mwuser  mem       REG              253,0     43776     1577 /usr/lib64/librt-2.17.so
java    12690 mwuser  mem       REG              253,0   1137024     1523 /usr/lib64/libm-2.17.so
java    12690 mwuser  mem       REG              253,0  17045796 17931792 /mwbase/jdk/jdk1.8.0_171/jre/lib/amd64/server/libjvm.so
java    12690 mwuser  mem       REG              253,0   2151832     1515 /usr/lib64/libc-2.17.so
java    12690 mwuser  mem       REG              253,0     19288     1521 /usr/lib64/libdl-2.17.so
java    12690 mwuser  mem       REG              253,0    104225 51097614 /mwbase/jdk/jdk1.8.0_171/lib/amd64/jli/libjli.so
java    12690 mwuser  mem       REG              253,0    142184     1573 /usr/lib64/libpthread-2.17.so
java    12690 mwuser  mem       REG              253,0    163336     1508 /usr/lib64/ld-2.17.so
java    12690 mwuser  mem       REG              253,0     18403 44550834 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-jetty-server-9.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     13352 44550833 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-jetty-client-9.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     15930 44550832 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-nutz-mvc-annotation-1.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     17007 44550831 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-nutz-http-1.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     12598 44550830 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-struts2-2.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     19542 44550829 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-springmvc-annotation-5.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     28024 44550828 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-spring-core-patch-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     21003 44550827 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-springmvc-annotation-3.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     11734 44550826 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-spring-cloud-feign-1.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     25499 44550825 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-springmvc-annotation-4.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     26121 44550824 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-springmvc-annotation-commons-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     34057 44550823 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-resttemplate-4.3.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     27971 44550822 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-spring-concurrent-util-4.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0      8700 44550821 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/spring-commons-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     13226 44550820 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-spring-async-annotation-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     27971 44550819 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-okhttp-3.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     18054 44550818 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-feign-default-http-9.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     30913 44550817 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-mongodb-3.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     17800 44550816 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/motan-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     22729 44550814 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-redisson-3.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     28116 44550813 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-jedis-2.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     89002 44550811 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-jdbc-commons-6.6.0.jar
java    12690 mwuser  mem       REG              253,0 124865798 34995412 /mwbase/appsystem/chinacri-pcarcore/chinacri-pcarcore-service-service/chinacri-pcarcore-service-service.jar
java    12690 mwuser  mem       REG              253,0     32768 34536985 /tmp/hsperfdata_mwuser/12690
java    12690 mwuser  mem       REG              253,0     20123 44550815 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/tomcat-7.x-8.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     20035 44550812 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-httpClient-4.x-plugin-6.6.0.jar
java    12690 mwuser  mem       REG              253,0     13149 44550810 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-dubbo-plugin-6.6.0.jar
java    12690 mwuser    0r      CHR                1,3       0t0     1028 /dev/null
java    12690 mwuser    1w      CHR                1,3       0t0     1028 /dev/null
java    12690 mwuser    2w      CHR                1,3       0t0     1028 /dev/null
java    12690 mwuser    3r      REG              253,0  66327620 17928313 /mwbase/jdk/jdk1.8.0_171/jre/lib/rt.jar
java    12690 mwuser    4r      REG              253,0 124865798 34995412 /mwbase/appsystem/chinacri-pcarcore/chinacri-pcarcore-service-service/chinacri-pcarcore-service-service.jar
java    12690 mwuser    5r      REG              253,0  18006420 51668271 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/skywalking-agent.jar
java    12690 mwuser    6r      REG              253,0   2023884 17928338 /mwbase/jdk/jdk1.8.0_171/jre/lib/ext/nashorn.jar
java    12690 mwuser    7r      REG              253,0     13149 44550810 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-dubbo-plugin-6.6.0.jar
java    12690 mwuser    8r      REG              253,0     89002 44550811 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-jdbc-commons-6.6.0.jar
java    12690 mwuser    9r      REG              253,0     20035 44550812 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-httpClient-4.x-plugin-6.6.0.jar
java    12690 mwuser   10r      REG              253,0     28116 44550813 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-jedis-2.x-plugin-6.6.0.jar
java    12690 mwuser   11r      REG              253,0     22729 44550814 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-redisson-3.x-plugin-6.6.0.jar
java    12690 mwuser   12r      REG              253,0     20123 44550815 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/tomcat-7.x-8.x-plugin-6.6.0.jar
java    12690 mwuser   13r      REG              253,0     17800 44550816 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/motan-plugin-6.6.0.jar
java    12690 mwuser   14r      REG              253,0     30913 44550817 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-mongodb-3.x-plugin-6.6.0.jar
java    12690 mwuser   15r      REG              253,0     18054 44550818 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-feign-default-http-9.x-plugin-6.6.0.jar
java    12690 mwuser   16r      REG              253,0     27971 44550819 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-okhttp-3.x-plugin-6.6.0.jar
java    12690 mwuser   17r      REG              253,0     13226 44550820 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-spring-async-annotation-plugin-6.6.0.jar
java    12690 mwuser   18r      REG              253,0      8700 44550821 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/spring-commons-6.6.0.jar
java    12690 mwuser   19r      REG              253,0     27971 44550822 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-spring-concurrent-util-4.x-plugin-6.6.0.jar
java    12690 mwuser   20r      REG              253,0     34057 44550823 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-resttemplate-4.3.x-plugin-6.6.0.jar
java    12690 mwuser   21r      REG              253,0     26121 44550824 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-springmvc-annotation-commons-6.6.0.jar
java    12690 mwuser   22r      REG              253,0     25499 44550825 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-springmvc-annotation-4.x-plugin-6.6.0.jar
java    12690 mwuser   23r      REG              253,0     11734 44550826 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-spring-cloud-feign-1.x-plugin-6.6.0.jar
java    12690 mwuser   24r      REG              253,0     21003 44550827 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-springmvc-annotation-3.x-plugin-6.6.0.jar
java    12690 mwuser   25r      REG              253,0     28024 44550828 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-spring-core-patch-6.6.0.jar
java    12690 mwuser   26r      REG              253,0     19542 44550829 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-springmvc-annotation-5.x-plugin-6.6.0.jar
java    12690 mwuser   27r      REG              253,0     12598 44550830 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-struts2-2.x-plugin-6.6.0.jar
java    12690 mwuser   28r      REG              253,0     17007 44550831 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-nutz-http-1.x-plugin-6.6.0.jar
java    12690 mwuser   29r      REG              253,0     15930 44550832 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-nutz-mvc-annotation-1.x-plugin-6.6.0.jar
java    12690 mwuser   30r      REG              253,0     13352 44550833 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-jetty-client-9.x-plugin-6.6.0.jar
java    12690 mwuser   31r      REG              253,0     18403 44550834 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-jetty-server-9.x-plugin-6.6.0.jar
java    12690 mwuser   32r      REG              253,0     13380 44550835 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-jetty-client-9.0-plugin-6.6.0.jar
java    12690 mwuser   33r      REG              253,0     15286 44550836 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-spymemcached-2.x-plugin-6.6.0.jar
java    12690 mwuser   34r      REG              253,0     18833 44550837 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-sharding-jdbc-1.5.x-plugin-6.6.0.jar
java    12690 mwuser   35r      REG              253,0     22593 44550838 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-sharding-sphere-3.x-plugin-6.6.0.jar
java    12690 mwuser   36r      REG              253,0     22941 44550839 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-sharding-sphere-4.x-plugin-6.6.0.jar
java    12690 mwuser   37r      REG              253,0     21597 44550840 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-xmemcached-2.x-plugin-6.6.0.jar
java    12690 mwuser   38r      REG              253,0     34608 44550841 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-grpc-1.x-plugin-6.6.0.jar
java    12690 mwuser   39r      REG              253,0     20364 44550842 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-mysql-commons-6.6.0.jar
java    12690 mwuser   40r      REG              253,0     30937 44550843 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-mysql-5.x-plugin-6.6.0.jar
java    12690 mwuser   41r      REG              253,0     32546 44550844 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-mysql-6.x-plugin-6.6.0.jar
java    12690 mwuser   42r      REG              253,0     27583 44550845 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-mysql-8.x-plugin-6.6.0.jar
java    12690 mwuser   43r      REG              253,0     22185 44550846 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-h2-1.x-plugin-6.6.0.jar
java    12690 mwuser   44r      REG              253,0     47162 44550847 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-postgresql-8.x-plugin-6.6.0.jar
java    12690 mwuser   45r      REG              253,0     31465 34422729 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-rocketmq-3.x-plugin-6.6.0.jar
java    12690 mwuser   46r      REG              253,0     31285 34536999 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-rocketmq-4.x-plugin-6.6.0.jar
java    12690 mwuser   47r      REG              253,0     12357 51668239 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-elastic-job-2.x-plugin-6.6.0.jar
java    12690 mwuser   48r      REG              253,0     34259 51668240 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-mongodb-2.x-plugin-6.6.0.jar
java    12690 mwuser   49r      REG              253,0     28035 51668241 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-httpasyncclient-4.x-plugin-6.6.0.jar
java    12690 mwuser   50r      REG              253,0     39591 51668242 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-kafka-plugin-6.6.0.jar
java    12690 mwuser   51r      REG              253,0     17543 51668243 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-servicecomb-java-chassis-0.x-plugin-6.6.0.jar
java    12690 mwuser   52r      REG              253,0     17807 51668244 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-servicecomb-java-chassis-1.x-plugin-6.6.0.jar
java    12690 mwuser   53r      REG              253,0     30118 51668245 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-hystrix-1.x-plugin-6.6.0.jar
java    12690 mwuser   54r      REG              253,0     17440 51668246 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/sofa-rpc-plugin-6.6.0.jar
java    12690 mwuser   55r      REG              253,0     21202 51668247 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-activemq-5.x-plugin-6.6.0.jar
java    12690 mwuser   56r      REG              253,0     34406 51668248 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-elasticsearch-5.x-plugin-6.6.0.jar
java    12690 mwuser   57r      REG              253,0     37170 51668249 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-elasticsearch-6.x-plugin-6.6.0.jar
java    12690 mwuser   58r      REG              253,0     34383 51668250 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-undertow-2.x-plugin-6.6.0.jar
java    12690 mwuser   59r      REG              253,0     21120 51668251 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-rabbitmq-5.x-plugin-6.6.0.jar
java    12690 mwuser   60r      REG              253,0     16922 51668252 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/dubbo-conflict-patch-6.6.0.jar
java    12690 mwuser   61r      REG              253,0     18875 51668253 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-canal-1.x-plugin-6.6.0.jar
java    12690 mwuser   62r      REG              253,0     13413 51668254 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-dubbo-2.7.x-plugin-6.6.0.jar
java    12690 mwuser   63r      REG              253,0     17195 51668255 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/dubbo-2.7.x-conflict-patch-6.6.0.jar
java    12690 mwuser   64r      REG              253,0     46564 51668256 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-vertx-core-3.x-plugin-6.6.0.jar
java    12690 mwuser   65r      REG              253,0     15883 51668257 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/resteasy-server-3.x-plugin-6.6.0.jar
java    12690 mwuser   66r      REG              253,0     20430 51668258 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-solrj-7.x-plugin-6.6.0.jar
java    12690 mwuser   67r      REG              253,0     24645 51668259 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-cassandra-java-driver-3.x-plugin-6.6.0.jar
java    12690 mwuser   68r      REG              253,0     13391 51668260 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-light4j-plugin-6.6.0.jar
java    12690 mwuser   69r      REG              253,0     24199 51668261 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-ehcache-2.x-plugin-6.6.0.jar
java    12690 mwuser   70r      REG              253,0     26953 51668262 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-pulsar-plugin-6.6.0.jar
java    12690 mwuser   71r      REG              253,0     22564 51668263 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-netty-socketio-plugin-6.6.0.jar
java    12690 mwuser   72r      REG              253,0     17473 51668264 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/plugins/apm-armeria-0.8.x-plugin-6.6.0.jar
java    12690 mwuser   73r      REG              253,0     16759 51668266 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/activations/apm-toolkit-log4j-1.x-activation-6.6.0.jar
java    12690 mwuser   74r      REG              253,0     21843 51668267 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/activations/apm-toolkit-log4j-2.x-activation-6.6.0.jar
java    12690 mwuser   75r      REG              253,0     26096 51668268 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/activations/apm-toolkit-logback-1.x-activation-6.6.0.jar
java    12690 mwuser   76r      REG              253,0     35575 51668269 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/activations/apm-toolkit-opentracing-activation-6.6.0.jar
java    12690 mwuser   77r      REG              253,0     35650 51668270 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/activations/apm-toolkit-trace-activation-6.6.0.jar
java    12690 mwuser   78w      REG              253,0 241399857 17943269 /mwbase/appsystem/skywalking/apache-skywalking-apm-bin-es7/agent/logs/skywalking-api.log
java    12690 mwuser   79r      REG              253,0   3505295 17928322 /mwbase/jdk/jdk1.8.0_171/jre/lib/resources.jar
java    12690 mwuser   80r      REG              253,0    660559 17928331 /mwbase/jdk/jdk1.8.0_171/jre/lib/jsse.jar
java    12690 mwuser   81r      CHR                1,8       0t0     1032 /dev/random
java    12690 mwuser   82r      CHR                1,9       0t0     1033 /dev/urandom
java    12690 mwuser   83r      CHR                1,8       0t0     1032 /dev/random
java    12690 mwuser   84r      CHR                1,8       0t0     1032 /dev/random
java    12690 mwuser   85r      CHR                1,9       0t0     1033 /dev/urandom
java    12690 mwuser   86r      CHR                1,9       0t0     1033 /dev/urandom
java    12690 mwuser   87r     FIFO                0,9       0t0 26472910 pipe
java    12690 mwuser   88w     FIFO                0,9       0t0 26472910 pipe
java    12690 mwuser   89u  a_inode               0,10         0     7150 [eventpoll]
java    12690 mwuser   90r     FIFO                0,9       0t0 26472911 pipe
java    12690 mwuser   91w     FIFO                0,9       0t0 26472911 pipe
java    12690 mwuser   92u  a_inode               0,10         0     7150 [eventpoll]
java    12690 mwuser   93r     FIFO                0,9       0t0 26472912 pipe
java    12690 mwuser   94w     FIFO                0,9       0t0 26472912 pipe
java    12690 mwuser   95u  a_inode               0,10         0     7150 [eventpoll]
java    12690 mwuser   96r     FIFO                0,9       0t0 26472913 pipe
java    12690 mwuser   97w     FIFO                0,9       0t0 26472913 pipe
java    12690 mwuser   98u  a_inode               0,10         0     7150 [eventpoll]
java    12690 mwuser   99r     FIFO                0,9       0t0 26472914 pipe
java    12690 mwuser  100w     FIFO                0,9       0t0 26472914 pipe
java    12690 mwuser  101u  a_inode               0,10         0     7150 [eventpoll]
java    12690 mwuser  102r     FIFO                0,9       0t0 26472915 pipe
java    12690 mwuser  103w     FIFO                0,9       0t0 26472915 pipe
java    12690 mwuser  104u  a_inode               0,10         0     7150 [eventpoll]
java    12690 mwuser  105r     FIFO                0,9       0t0 26472916 pipe
java    12690 mwuser  106w     FIFO                0,9       0t0 26472916 pipe
java    12690 mwuser  107u  a_inode               0,10         0     7150 [eventpoll]
java    12690 mwuser  108r     FIFO                0,9       0t0 26472917 pipe
java    12690 mwuser  109w     FIFO                0,9       0t0 26472917 pipe
java    12690 mwuser  110u  a_inode               0,10         0     7150 [eventpoll]
java    12690 mwuser  111r     FIFO                0,9       0t0 26472918 pipe
java    12690 mwuser  112w     FIFO                0,9       0t0 26472918 pipe
java    12690 mwuser  113u  a_inode               0,10         0     7150 [eventpoll]
java    12690 mwuser  114r     FIFO                0,9       0t0 26472919 pipe
java    12690 mwuser  115w     FIFO                0,9       0t0 26472919 pipe
java    12690 mwuser  116u  a_inode               0,10         0     7150 [eventpoll]
java    12690 mwuser  117r     FIFO                0,9       0t0 26472920 pipe
java    12690 mwuser  118w     FIFO                0,9       0t0 26472920 pipe
java    12690 mwuser  119u  a_inode               0,10         0     7150 [eventpoll]
java    12690 mwuser  120r     FIFO                0,9       0t0 26472921 pipe
java    12690 mwuser  121w     FIFO                0,9       0t0 26472921 pipe
java    12690 mwuser  122u  a_inode               0,10         0     7150 [eventpoll]
java    12690 mwuser  123r     FIFO                0,9       0t0 26472922 pipe
java    12690 mwuser  124w     FIFO                0,9       0t0 26472922 pipe
java    12690 mwuser  125u  a_inode               0,10         0     7150 [eventpoll]
java    12690 mwuser  126r     FIFO                0,9       0t0 26472923 pipe
java    12690 mwuser  127w     FIFO                0,9       0t0 26472923 pipe
java    12690 mwuser  128u  a_inode               0,10         0     7150 [eventpoll]
java    12690 mwuser  129r     FIFO                0,9       0t0 26472924 pipe
java    12690 mwuser  130w     FIFO                0,9       0t0 26472924 pipe
java    12690 mwuser  131u  a_inode               0,10         0     7150 [eventpoll]
java    12690 mwuser  132r     FIFO                0,9       0t0 26472925 pipe
java    12690 mwuser  133w     FIFO                0,9       0t0 26472925 pipe
java    12690 mwuser  134u  a_inode               0,10         0     7150 [eventpoll]
java    12690 mwuser  135u     unix 0xffff9bc713886800       0t0 26477658 socket
java    12690 mwuser  136u     IPv6           26477660       0t0      TCP *:42904 (LISTEN)
java    12690 mwuser  137u     IPv6           26477661       0t0      TCP *:italk (LISTEN)
java    12690 mwuser  138u     IPv6           26477665       0t0      TCP *:45397 (LISTEN)
java    12690 mwuser  139r      REG              253,0 124865798 34995412 /mwbase/appsystem/chinacri-pcarcore/chinacri-pcarcore-service-service/chinacri-pcarcore-service-service.jar
java    12690 mwuser  140u      REG              253,0  84838296 17525669 /tmp/rule%0045ngine%0043ache.data
java    12690 mwuser  141r     FIFO                0,9       0t0 27767188 pipe
java    12690 mwuser  142u     IPv6           26473526       0t0      TCP SZ-CX-PCARCORE-005:48054->172.27.1.233:11800 (ESTABLISHED)
java    12690 mwuser  143u     unix 0xffff9bcaa5370400       0t0 26472284 socket
java    12690 mwuser  144w      REG              253,0  58047266 34995403 /mwbase/applogs/rtlog/PCARCORESERVICE_PRD_5/common-all.log
java    12690 mwuser  145r      REG              253,0   3135615 17928311 /mwbase/jdk/jdk1.8.0_171/jre/lib/charsets.jar
java    12690 mwuser  146r      REG              253,0  18480356 17928336 /mwbase/jdk/jdk1.8.0_171/jre/lib/ext/jfxrt.jar
java    12690 mwuser  147u     IPv6           35227341       0t0      TCP SZ-CX-PCARCORE-005:60539->172.27.200.22:http (ESTABLISHED)
java    12690 mwuser  148u     IPv6           32507264       0t0      TCP SZ-CX-PCARCORE-005:57302->172.27.1.247:21212 (ESTABLISHED)
java    12690 mwuser  149r      REG              253,0   3860502 17928335 /mwbase/jdk/jdk1.8.0_171/jre/lib/ext/cldrdata.jar
java    12690 mwuser  150r      REG              253,0   1178928 17928342 /mwbase/jdk/jdk1.8.0_171/jre/lib/ext/localedata.jar
java    12690 mwuser  151r     FIFO                0,9       0t0 27908862 pipe
java    12690 mwuser  152w      REG              253,0   4247519 18544350 /mwbase/applogs/rtlog/pcarcore-service/logs/nacos/naming.log
java    12690 mwuser  153r     FIFO                0,9       0t0 28066031 pipe
java    12690 mwuser  154u     IPv6           26477704       0t0      UDP *:37946 
java    12690 mwuser  155r     FIFO                0,9       0t0 28063977 pipe
java    12690 mwuser  156r     FIFO                0,9       0t0 26477706 pipe
java    12690 mwuser  157r     FIFO                0,9       0t0 27731010 pipe
java    12690 mwuser  158w     FIFO                0,9       0t0 27731010 pipe
java    12690 mwuser  159r     FIFO                0,9       0t0 27960775 pipe
二、分析
统计数量较多类型 pipe 、eventpoll是影响因素，但是pipe影响文件无法定位，socket 数量不多，考虑先排除。将应用中httpclient socket 显示关闭
pipe、eventpoll 说明 https://blog.csdn.net/abcyyjjkk/article/details/82181977
//TODO 代码待补充~~
socket 分析处理
实际上 socket 显示关闭后并不明显。
netstat
到线上服务器上，用netstat来统计进程的connection数量。
netstat -antp | grep pid | wc -l
socket https://blog.csdn.net/blankti/article/details/100808475
netstat -tan|awk '$1~/tcp/{print $NF}'|sort|uniq -c|sort -nr
那只剩下文件读写，文件读写不好确认是什么文件。
配置文件、dubbo服务缓存文件、还有当前应用特有的缓存文件（ehcache 缓存文件硬盘存储）
从特有的缓存文件入手应该是找对了。先取消系统的缓存查询。
周四版本后，运行较为稳定~
tips :之前在生产此问题复现比较麻烦,需要一定的业务量，运行一段时间达到系统阈值，才会复现too many open files
单个进程允许打开的最大fd数量设置：ulimit -HSn 50 此指令设置后，在当前终端生效（关闭终端失效），启动待测应用。
三、事后回顾 ehcache
https://blog.csdn.net/a544879146/article/details/109009310
简单说明ehcache，原本适合单体应用与java应用一体，硬盘缓存高并发也会存在too many open files 问题，旧系统采取每天凌晨定时重启避免。新系统分布式一个系统拆分多个，沿用ehcache 应该会加剧 too many open files 的情况。目前调整将缓存数据放到map中，未命中就查询数据库。更合理的方式应该采用redis 或者其他分布式缓存
类似参考
https://blog.csdn.net/iamzhongyong/article/details/84421307?utm_medium=distribute.pc_relevant_t0.none-task-blog-OPENSEARCH-1.channel_param&depth_1-utm_source=distribute.pc_relevant_t0.none-task-blog-OPENSEARCH-1.channel_param