spark中dataFrame 的join操作出现的问题
val resu1: DataFrame = joined2.filter(($"register_time" + 7 * 86400000 )> $"signin_time") .groupBy("register_time").agg(countDistinct(registered("userUID")).as("num"))
// resu1.show()
val resu2: DataFrame = joined2.groupBy("register_time").agg(countDistinct(registered("userUID")).as("totalNum"))
// resu2.show()
//通过字段register_time join
val frame: DataFrame = resu1.join(resu2,Seq("register_time"))
val frame2: DataFrame = frame.select($"register_time",($"num"/$"totalNum").as("percent"))
JdbcUtils.dataFrameToMysql(frame2,JdbcUtils.table_week_percent,0)// 封装的方法
//如果是下面这种 没有指明字段 show 也能出结果 但是很慢很慢 而且提示要加这句话:config("spark.sql.crossJoin.enabled",true)
//写入数据库的时候就报错了
val frame: DataFrame = resu1.join(resu2)
val frame2: DataFrame = frame.select(registered("register_time"),($"num"/$"totalNum").as("percent"))
frame2.show()
JdbcUtils.dataFrameToMysql(frame2,JdbcUtils.table_week_percent,0)// 封装的方法
//错误信息:
org.hibernate.exception.JDBCConnectionException: Cannot open connection
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:74)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:426)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:139)
Caused by: java.net.BindException: Address already in use: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
The driver was unable to create a connection due to an inability to establish the client portion of a socket.
This is usually caused by a limit on the number of sockets imposed by the operating system. This limit is usually configurable.
For Unix-based platforms, see the manual page for the 'ulimit' command. Kernel or system reconfiguration may also be required.
For Windows-based platforms, see Microsoft Knowledge Base Article 196271 (Q196271).
...
...
原因:join操作虽然在show的时候能出结果但是花费了很长的时间,可见两边关联的时候选择了好几个字段进行关联 ,但是在执行写入mysql的时候就和mysql建立了连接,但是结果迟迟不能出来,最后超时退出,也有解决办法:
https://blog.csdn.net/xiaoxuexu/article/details/84482355?utm_medium=distribute.pc_relevant_t0.none-task-blog-2%7Edefault%7EBlogCommendFromMachineLearnPai2%7Edefault-1.control&depth_1-utm_source=distribute.pc_relevant_t0.none-task-blog-2%7Edefault%7EBlogCommendFromMachineLearnPai2%7Edefault-1.control
但是这个并不是解决本错误案例的最佳方法(其实就是语法不够规范),所以很多时候看问题不能只看表面。看到的问题是表象,要找到产生问题的真正原因才是关键。
计算机的问题有点类似于人的病,看不见摸不着,你是咳嗽了,但是咳嗽的原因有很多,不是别人吃这个药你就吃这个药,找到引起咳嗽的真正原因才能对症下药。