ES：Too many open files in system

测试环境es有warn:

[o.e.g.DanglingIndicesState] [elk-app10] failed to list dangling indices
java.nio.file.FileSystemException: /data/elasticsearch/data/nodes/0/indices: Too many open files in system
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[?:?]
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
    at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427) ~[?:?]
    at java.nio.file.Files.newDirectoryStream(Files.java:457) ~[?:?]
    at org.elasticsearch.env.NodeEnvironment.availableIndexFoldersForPath(NodeEnvironment.java:901) ~[elasticsearch-6.7.1.jar:6.7.1]
    at org.elasticsearch.env.NodeEnvironment.availableIndexFolders(NodeEnvironment.java:867) ~[elasticsearch-6.7.1.jar:6.7.1]
    at org.elasticsearch.gateway.MetaStateService.loadIndicesStates(MetaStateService.java:89) ~[elasticsearch-6.7.1.jar:6.7.1]
    at org.elasticsearch.gateway.DanglingIndicesState.findNewDanglingIndices(DanglingIndicesState.java:137) ~[elasticsearch-6.7.1.jar:6.7.1]
    at org.elasticsearch.gateway.DanglingIndicesState.findNewAndAddDanglingIndices(DanglingIndicesState.java:122) ~[elasticsearch-6.7.1.jar:6.7.1]
    at org.elasticsearch.gateway.DanglingIndicesState.processDanglingIndices(DanglingIndicesState.java:87) ~[elasticsearch-6.7.1.jar:6.7.1]
    at org.elasticsearch.gateway.DanglingIndicesState.clusterChanged(DanglingIndicesState.java:191) ~[elasticsearch-6.7.1.jar:6.7.1]
    at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateListeners$7(ClusterApplierService.java:495) ~[elasticsearch-6.7.1.jar:6.7.1]
    at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) [?:?]
    at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) [?:?]
    at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) [?:?]
    at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:492) [elasticsearch-6.7.1.jar:6.7.1]
    at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:475) [elasticsearch-6.7.1.jar:6.7.1]
    at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:419) [elasticsearch-6.7.1.jar:6.7.1]
    at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:163) [elasticsearch-6.7.1.jar:6.7.1]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-6.7.1.jar:6.7.1]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-6.7.1.jar:6.7.1]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-6.7.1.jar:6.7.1]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:?]
    at java.lang.Thread.run(Thread.java:745) [?:?]

这是因为系统打开的文件太多了，可以用ulimit -a查看当前系统文件最大打开数

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 191525
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 512000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 512000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

栏open files不是默认的1024，没毛病。

lsof |wc -l

可以看已经打开文件数量。8万了。

file-max是设置 系统所有进程一共可以打开的文件数量 。

再看看：cat /proc/sys/fs/file-max

65536

这个小了。还需要改。

echo 52000 > /proc/sys/fs/file-max

再来看看，cat /proc/sys/fs/file-max
52000

通常线上环境不会出现这种问题，运维都会提前配好了。