spark链接redis报错Too many open files

原创 2018年04月17日 21:04:42

最近在写spark调用redis的时候总是遇到如下错误:

18/03/28 16:36:40 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.IllegalStateException: failed to create a child event loop
java.lang.IllegalStateException: failed to create a child event loop
	at com.lambdaworks.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
	at com.lambdaworks.io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
	at com.lambdaworks.io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:68)
	at com.lambdaworks.io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:63)
	at com.lambdaworks.io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:54)
	at com.lambdaworks.redis.resource.DefaultEventLoopGroupProvider.createEventLoopGroup(DefaultEventLoopGroupProvider.java:125)
	at com.lambdaworks.redis.resource.DefaultEventLoopGroupProvider.getOrCreate(DefaultEventLoopGroupProvider.java:99)
	at com.lambdaworks.redis.resource.DefaultEventLoopGroupProvider.allocate(DefaultEventLoopGroupProvider.java:48)
	at com.lambdaworks.redis.AbstractRedisClient.getEventLoopGroup(AbstractRedisClient.java:183)
	at com.lambdaworks.redis.AbstractRedisClient.channelType(AbstractRedisClient.java:167)
	at com.lambdaworks.redis.RedisClient.connectAsync(RedisClient.java:448)
	at com.lambdaworks.redis.RedisClient.connectAsync(RedisClient.java:429)
	at com.lambdaworks.redis.RedisClient.connect(RedisClient.java:371)
	at com.lambdaworks.redis.RedisClient.connect(RedisClient.java:344)
	at com.lambdaworks.redis.RedisClient.connect(RedisClient.java:329)
	at Util.Redis$.Init(Redis.scala:28)
	at Util.Redis$.getResource(Redis.scala:34)
	at LogTools.Format.Normal.BasicTask$.FileIsProcessed(BasicTask.scala:97)
	at LogTools.Format.Normal.BasicTask$.filesNotUpdate(BasicTask.scala:635)
	at LogTools.Format.Normal.BasicTask$.processSplitFile(BasicTask.scala:257)
	at LogTools.Format.Normal.BasicTask$.main(BasicTask.scala:221)
	at LogTools.Format.Normal.BasicTask.main(BasicTask.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:552)
Caused by: com.lambdaworks.io.netty.channel.ChannelException: failed to open a new selector
	at com.lambdaworks.io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:151)
	at com.lambdaworks.io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:142)
	at com.lambdaworks.io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:94)
	at com.lambdaworks.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
	... 26 more
Caused by: java.io.IOException: Too many open files
	at sun.nio.ch.IOUtil.makePipe(Native Method)
	at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65)
	at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
	at com.lambdaworks.io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:149)
	... 29 more

原因:在init里面创建redis的链接之后,用完之后没有释放资源,没有close将链接释放,导致在程序在创建过多的redis链接之后,就报too many open files 的错误。最开始我用得是com.lambdaworks.redis来调用redis,用的时候发现这个包在调用close()之后没有释放connection,导致redis的链接越来越多。

修改:使用连接池来维护redis的connection,或者我最后用的是jedis来调用redis。

参考代码如下:

object JRedis extends Serializable{
	case class RedisConfig(
		                      Host: String,
		                      Port: Int,
		                      Password: String
	                      ) extends Serializable

	@transient
	private var j: Jedis = _


	private def Init(host: String, port: Int, password: String): Unit = {
		if (j != null) return
		j = new Jedis(host, port)
		if (password.nonEmpty) j.auth(password)
		j.select(0)
	}

	def getResource(RedisConfig: RedisConfig) = {
		this.synchronized {
			Init(RedisConfig.Host: String, RedisConfig.Port: Int, RedisConfig.Password: String)
			j
		}
	}

	def updateValueByKey(key: String, value: Long, RedisConfig: RedisConfig): Unit = {
		val redis = getResource(RedisConfig: RedisConfig)
		redis.incrBy(key, value)
		redis.close
	}

	def getValueByKey(key: String, RedisConfig: RedisConfig): Long = {
		val redis = getResource(RedisConfig: RedisConfig)
		val num = redis.get(key).toLong
		redis.close()
		num
	}
}


版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/u012289670/article/details/79980803

Spark-sql 运行提示too many open files

设置/etc/security/limits.conf 在结尾# End of file下添加 * soft nofile 65536 * hard nofile 65536 * soft n...
  • wisdom_c_1010
  • wisdom_c_1010
  • 2015-06-17 10:58:24
  • 970

Spark streaming + kafka 运行时报 Too many open files错误的解决方法

什么改limits.conf这些方法我都试过,不管用,错误照常出现,需要修改的是kafka的一个参数叫做num.replica.fetchers,默认为1,改为2或者3就可以了。...
  • hust_mse
  • hust_mse
  • 2017-07-12 15:12:51
  • 288

go 服务器出现too many open files的错误

最近两天服务器总是会出现too many open files的错误。真的很是头疼。在本地开发环境中,也用了压力测试,发现问题不存在。但是为什么服务器会出现这种错误,有些闹不明白。本来想用ulimit...
  • u011511092
  • u011511092
  • 2014-01-15 07:42:54
  • 3630

Redis too many open files error

最近公司的API总是莫名其妙的报告一些redis 打开文件太多的错误。今天老板终于下定决心,必须解决这个bug。经过测试,问题终于出现了。...
  • u011511092
  • u011511092
  • 2014-01-09 20:54:56
  • 2657

too many open files解决办法

在/etc/security/limit.conf 最后加上如下两行: root soft nofile 32000 root hard nofile 32000...
  • hljlzc2007
  • hljlzc2007
  • 2014-09-11 15:01:43
  • 3171

Linux问题 open too many files

部署到linux下的tomcat,今天发现包了“java.net.SocketException: Too many open files”,以前从来没有遇到过,在此记录一下: 彻底解决问题的是...
  • leili0806
  • leili0806
  • 2012-05-04 14:54:41
  • 53755

一样的“Too many open files”,不一样的阿里云和青云

现象系统服务准备在青云上线,突然间发现服务时断时续,登陆服务器,查看日志,原来是‘Too many open files‘,... 青云和阿里云的解决方案不同.......
  • wireless_com
  • wireless_com
  • 2016-05-14 22:04:45
  • 4616

Too many open files 解决方法

  • 2012年12月20日 17:42
  • 2KB
  • 下载

解决Too many open files问题

当用linux做高并发服务器时,会遇到"Too many open files"的错误。 Linux是有文件句柄限制的(open files),而且Linux默认不是很高,一般都是1024,做高并发生...
  • zhuwinmin
  • zhuwinmin
  • 2017-05-25 16:02:23
  • 677

Linux下java报错Too many open files的解决方法

最近发现运行在Linux下的java application报如下错误:   java.io.IOException: Cannot run program "/bin/sh": java.io....
  • qaz1qaz1qaz2
  • qaz1qaz1qaz2
  • 2016-10-15 21:25:12
  • 844
收藏助手
不良信息举报
您举报文章:spark链接redis报错Too many open files
举报原因:
原因补充:

(最多只允许输入30个字)