记一次数仓Spark任务报错Caused by: java.lang.OutOfMemoryError: unable to create new native thread

Caused by: java.lang.OutOfMemoryError: unable to create new native thread

Caused by: java.lang.OutOfMemoryError: unable to create new native thread

    at java.lang.Thread.start0(Native Method)

    at java.lang.Thread.start(Thread.java:717)

    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:786)

    ... 20 more

24/03/04 17:42:35 INFO retry.RetryInvocationHandler: Exception while invoking renewLease of class ClientNamenodeProtocolTranslatorPB over cdh106/00.0.00.00:8020 after 3 fail over attempts. Trying to fail over immediately.

java.io.IOException: Failed on local exception: java.io.IOException: Couldn't set up IO streams; Host Details : local host is: "cdh116/00.0.00.00"; destination host is: "cdh106":8020;

   发现每次报错都是在cdh116,检查116机器资源问题，发现内存cpu磁盘IO均正常

   咨询运维同事物理机状况，结果正常

   检查机器配置的句柄数，正常，网上很多。不赘述

   后查询该机器上的句柄打开数量，发现是某个组件打开了太多的句柄未关闭

lsof -n|awk '{print $2}'|sort|uniq -c|sort -nr|more

ps aef | grep 2XX1（最大进程pid查看详情）

    发现kafka-eagle占用1300万个句柄，重启之，解决