97.Spark大型电商项目-页面单跳转化率-生产环境测试

目录

创建hive表

user_visit_action

 导入数据

user_info

导入数据 

打包测试

spark_page.sh

问题解决

---

---

本篇文章记录页面单跳转化率-生产环境测试。

创建hive表

user_visit_action

create table user_visit_action(
    dates string,
    userid bigint,
    sessionid string,
    pageid bigint,
    action_time string,
    search_key_word string,
    click_category_id bigint,
    click_product_id bigint,
    order_category_ids string,
    order_product_ids string,
    pay_category_ids string,
    pay_product_ids string
);

 导入数据

hive> load data local inpath '/home/hadoop/docker/graduationlogs/user_visit_action.txt' into table user_visit_actiLoading data to table hive.user_visit_action
 

user_info

create table user_info(
    user_id bigint,
    user_name string,
    name string,
    age bigint,
    professional string,
    city string,
    sex string
);

导入数据 

hive> load data local inpath '/home/hadoop/docker/graduationlogs/user_info.txt' into table user_info;

打包测试

spark_page.sh

spark-submit --class graduation.java.spark.page.PageOneStepConvertRateSpark \
--master spark://Master:7077 \
--deploy-mode client \
--driver-memory 1g \
--executor-memory 1g \
--total-executor-cores 4 \
--executor-cores 2 \
--files /usr/local/hive/conf/hive-site.xml \
/home/hadoop/IdeaProjects/BigDataGraduationProject/out/artifacts/BigDataGraduationProject_jar/BigDataGraduationProject.jar \
${1}	

运行权限：sudo chmod 777 spark_page.sh

问题解决

1、Caused by: java.net.ConnectException: 拒绝连接 (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211)
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:300)
    ... 28 more
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

解决方案：发现是数据库的链接地址使用本地的节点有问题，将启改为localhost进行解决，或者是直接使用master机器上的ip地址。

2、Caused by: java.lang.ClassNotFCaoundException: org.apache.spark.sql.hive.HiveContext

解决方法：是在代码中使用了已经弃用了的hiveContext,将spark-hive_2.11-2.3.1.jar中的放入spark/jars中即可解决

3、Caused by: java.lang.ClassNotFCaoundException:com.alibaba.fastjson.JSONObject

解决方案：代码中使用了阿里巴巴的JSON jar包，导入com.alibaba.fastjson1.2.55.jar即可。