spark sql and hive 3g数据测试

最新推荐文章于 2022-06-24 07:00:00 发布
最新推荐文章于 2022-06-24 07:00:00 发布
阅读量1.7k 收藏
点赞数
分类专栏： Hive Spark 文章标签： spark sql
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/wind520/article/details/44081849
版权
Spark 同时被 2 个专栏收录
21 篇文章 0 订阅
订阅专栏
Hive
19 篇文章 0 订阅
订阅专栏

1：上传文件到hdfs

[jifeng@feng01 hadoop]$ hadoop fs -mkdir /user/jifeng/fire
15/03/05 13:29:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[jifeng@feng01 hadoop]$ hadoop fs -put /home/jifeng/hadoop/firewallb-week2-origin.sql /user/jifeng/fire/
15/03/05 13:30:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[jifeng@feng01 hadoop]$

2:hive中建表 


[jifeng@feng01 ~]$ hive
15/03/05 13:31:08 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect.  Use hive.hmshandler.retry.* instead

Logging initialized using configuration in jar:file:/home/jifeng/hive/lib/hive-common-0.13.1.jar!/hive-log4j.properties
hive> CREATE EXTERNAL TABLE firewall (         idauto INT,         dt TIMESTAMP,         dtstr STRING,         timeSeconds INT,         priority STRING,         operation STRING,         messageCode STRING,         protocol STRING,         srcIp STRING,         destIp STRING,         srcIPNumber bigINT,         destIPNumber bigINT,         srcPort INT,         destPort INT,         destService STRING,         direction STRING,         flags STRING,         command STRING,         slot1min INT,         slot5min INT,         slot30min INT,         slot60min INT        ) row format delimited fields terminated by '\t'; 
OK
Time taken: 3.414 seconds
hive> load data inpath '/user/jifeng/fire/firewallb-week2-origin.sql' into table firewall;
Loading data to table default.firewall
Table default.firewall stats: [numFiles=1, numRows=0, totalSize=3415284428, rawDataSize=0]
OK
Time taken: 1.441 seconds

建表语句：

CREATE EXTERNAL TABLE firewall (  
       idauto INT,  
       dt TIMESTAMP,  
       dtstr STRING,  
       timeSeconds INT,  
       priority STRING,  
       operation STRING,  
       messageCode STRING,  
       protocol STRING,  
       srcIp STRING,  
       destIp STRING,  
       srcIPNumber bigINT,  
       destIPNumber bigINT,  
       srcPort INT,  
       destPort INT,  
       destService STRING,  
       direction STRING,  
       flags STRING,  
       command STRING,  
       slot1min INT,  
       slot5min INT,  
       slot30min INT,  
       slot60min INT   
     ) row format delimited fields terminated by '\t';


3：hive中测试 

hive> select * from firewall order by dt desc limit 10; 
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Job = job_1423061535604_0004, Tracking URL = http://feng01:8088/proxy/application_1423061535604_0004/
Kill Command = /home/jifeng/hadoop/hadoop-2.4.1/bin/hadoop job  -kill job_1423061535604_0004
Hadoop job information for Stage-1: number of mappers: 13; number of reducers: 1
2015-03-05 13:35:57,706 Stage-1 map = 0%,  reduce = 0%
2015-03-05 13:36:58,247 Stage-1 map = 0%,  reduce = 0%
2015-03-05 13:37:26,248 Stage-1 map = 3%,  reduce = 0%, Cumulative CPU 5.38 sec
2015-03-05 13:37:34,542 Stage-1 map = 8%,  reduce = 0%, Cumulative CPU 7.63 sec
2015-03-05 13:38:40,840 Stage-1 map = 8%,  reduce = 0%, Cumulative CPU 13.89 sec
2015-03-05 13:39:24,926 Stage-1 map = 10%,  reduce = 0%, Cumulative CPU 79.08 sec
2015-03-05 13:40:00,669 Stage-1 map = 13%,  reduce = 0%, Cumulative CPU 85.37 sec
2015-03-05 13:40:25,540 Stage-1 map = 15%,  reduce = 0%, Cumulative CPU 86.27 sec
2015-03-05 13:40:45,643 Stage-1 map = 26%,  reduce = 0%, Cumulative CPU 107.73 sec
2015-03-05 13:40:46,742 Stage-1 map = 31%,  reduce = 0%, Cumulative CPU 109.79 sec
2015-03-05 13:40:59,426 Stage-1 map = 36%,  reduce = 0%, Cumulative CPU 116.75 sec
2015-03-05 13:41:03,706 Stage-1 map = 38%,  reduce = 0%, Cumulative CPU 120.27 sec
2015-03-05 13:41:07,921 Stage-1 map = 44%,  reduce = 0%, Cumulative CPU 125.38 sec
2015-03-05 13:41:15,830 Stage-1 map = 49%,  reduce = 0%, Cumulative CPU 130.26 sec
2015-03-05 13:41:18,002 Stage-1 map = 54%,  reduce = 0%, Cumulative CPU 131.86 sec
2015-03-05 13:42:18,780 Stage-1 map = 54%,  reduce = 0%, Cumulative CPU 170.98 sec
2015-03-05 13:43:19,428 Stage-1 map = 54%,  reduce = 0%, Cumulative CPU 194.07 sec
2015-03-05 13:44:20,028 Stage-1 map = 54%,  reduce = 0%, Cumulative CPU 197.32 sec
2015-03-05 13:45:20,494 Stage-1 map = 54%,  reduce = 0%, Cumulative CPU 200.93 sec
2015-03-05 13:46:21,209 Stage-1 map = 54%,  reduce = 0%, Cumulative CPU 223.6 sec
2015-03-05 13:46:42,075 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 230.03 sec
2015-03-05 13:46:49,284 Stage-1 map = 56%,  reduce = 15%, Cumulative CPU 218.59 sec
2015-03-05 13:46:51,358 Stage-1 map = 60%,  reduce = 15%, Cumulative CPU 219.07 sec
2015-03-05 13:46:54,448 Stage-1 map = 63%,  reduce = 15%, Cumulative CPU 191.15 sec
2015-03-05 13:46:58,609 Stage-1 map = 64%,  reduce = 15%, Cumulative CPU 193.67 sec
2015-03-05 13:47:05,823 Stage-1 map = 67%,  reduce = 15%, Cumulative CPU 180.53 sec
2015-03-05 13:47:09,938 Stage-1 map = 69%,  reduce = 15%, Cumulative CPU 194.45 sec
2015-03-05 13:47:11,986 Stage-1 map = 69%,  reduce = 18%, Cumulative CPU 194.85 sec
2015-03-05 13:47:47,100 Stage-1 map = 72%,  reduce = 18%, Cumulative CPU 209.86 sec
2015-03-05 13:48:31,364 Stage-1 map = 77%,  reduce = 18%, Cumulative CPU 210.55 sec
2015-03-05 13:48:37,549 Stage-1 map = 77%,  reduce = 21%, Cumulative CPU 210.74 sec
2015-03-05 13:48:53,998 Stage-1 map = 82%,  reduce = 21%, Cumulative CPU 177.87 sec
2015-03-05 13:48:55,021 Stage-1 map = 82%,  reduce = 23%, Cumulative CPU 178.51 sec
2015-03-05 13:49:06,447 Stage-1 map = 90%,  reduce = 23%, Cumulative CPU 143.53 sec
2015-03-05 13:49:11,591 Stage-1 map = 90%,  reduce = 26%, Cumulative CPU 144.64 sec
2015-03-05 13:49:12,612 Stage-1 map = 95%,  reduce = 26%, Cumulative CPU 145.78 sec
2015-03-05 13:49:14,665 Stage-1 map = 95%,  reduce = 28%, Cumulative CPU 145.81 sec
2015-03-05 13:49:28,043 Stage-1 map = 97%,  reduce = 28%, Cumulative CPU 147.61 sec
2015-03-05 13:49:29,075 Stage-1 map = 100%,  reduce = 28%, Cumulative CPU 147.67 sec
2015-03-05 13:50:06,153 Stage-1 map = 100%,  reduce = 67%, Cumulative CPU 148.22 sec
2015-03-05 13:50:09,249 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 149.17 sec
MapReduce Total cumulative CPU time: 2 minutes 29 seconds 170 msec
Ended Job = job_1423061535604_0004
MapReduce Jobs Launched: 
Job 0: Map: 13  Reduce: 1   Cumulative CPU: 149.68 sec   HDFS Read: 3418565375 HDFS Write: 2076 SUCCESS
Total MapReduce CPU Time Spent: 2 minutes 29 seconds 680 msec
OK
16600916        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Warning Deny    ASA-4-106023    TCP     10.6.6.7 10.0.2.2        168166919       167772674       42465   52372   52372_tcp       (empty) (empty) (empty) 227670004553400 758900  379450
16600915        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Warning Deny    ASA-4-106023    TCP     10.13.77.49      10.0.3.2        168643889       167772930       58200   32179   32179_tcp       (empty) (empty) (empty) 22767000 4553400 758900  379450
16600931        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Warning Deny    ASA-4-106023    TCP     10.6.6.7 10.0.2.2        168166919       167772674       42464   33004   33004_tcp       (empty) (empty) (empty) 227670004553400 758900  379450
16600919        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Warning Deny    ASA-4-106023    TCP     10.6.6.7 10.0.2.2        168166919       167772674       42465   38300   38300_tcp       (empty) (empty) (empty) 227670004553400 758900  379450
16600918        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Warning Deny    ASA-4-106023    TCP     10.13.77.49      10.0.3.2        168643889       167772930       58199   25087   25087_tcp       (empty) (empty) (empty) 22767000 4553400 758900  379450
16600912        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Warning Deny    ASA-4-106023    TCP     10.6.6.7 10.0.2.2        168166919       167772674       42465   556     remotefs        (empty) (empty) (empty) 227670004553400 758900  379450
16600913        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Warning Deny    ASA-4-106023    TCP     10.13.77.49      10.0.3.2        168643889       167772930       58199   32179   32179_tcp       (empty) (empty) (empty) 22767000 4553400 758900  379450
16600920        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Info    Teardown        ASA-6-302014    TCP      10.6.6.7        172.10.0.2      168166919       2886336514      42706   3389    3389_tcp        inbound SYN Timeout      (empty) 22767000        4553400 758900  379450
16600930        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Warning Deny    ASA-4-106023    TCP     10.13.77.49      10.0.3.2        168643889       167772930       58200   13817   13817_tcp       (empty) (empty) (empty) 22767000 4553400 758900  379450
16600911        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Warning Deny    ASA-4-106023    TCP     10.13.77.49      10.0.3.2        168643889       167772930       58200   1993    1993_tcp        (empty) (empty) (empty) 22767000 4553400 758900  379450
Time taken: 879.31 seconds, Fetched: 10 row(s)

4：启动spark sql server

[jifeng@feng02 spark-1.2.0-bin-2.4.1]$ ./sbin/start-thriftserver.sh  --hiveconf hive.server2.thrift.port=10000  --hiveconf hive.server2.thrift.bind.host=feng02 --master spark://feng02:7077
starting org.apache.spark.sql.hive.thriftserver.HiveThriftServer2, logging to /home/jifeng/hadoop/spark-1.2.0-bin-2.4.1/sbin/../logs/spark-jifeng-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-feng02.out

运行beeline 

[jifeng@feng02 spark-1.2.0-bin-2.4.1]$ ./bin/beeline
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Beeline version ??? by Apache Hive

连接： 

beeline> !connect jdbc:hive2://feng02:10000 scott tiger org.apache.hive.jdbc.HiveDriver
Connecting to jdbc:hive2://feng02:10000
log4j:WARN No appenders could be found for logger (org.apache.thrift.transport.TSaslTransport).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Connected to: Spark SQL (version 1.2.0)
Driver: null (version null)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://feng02:10000> show tables;
+----------------+
|     result     |
+----------------+
| course         |
| firewall       |
| hbase_table_1  |
| pokes          |
| student        |
+----------------+
5 rows selected (0.489 seconds)


5：spark sql测试 

0: jdbc:hive2://feng02:10000> select * from firewall order by dt desc limit 10; 
+-----------+------------------------+-----------------------+--------------+-----------+------------+---------------+--+
|  idauto   |           dt           |         dtstr         | timeseconds  | priority  | operation  |  messagecode  |  |
+-----------+------------------------+-----------------------+--------------+-----------+------------+---------------+--+
| 16600918  | 2013-04-15 10:00:00.0  | 15/Apr/2013 10:00:00  | 1366020000   | Warning   | Deny       | ASA-4-106023  |  |
| 16600914  | 2013-04-15 10:00:00.0  | 15/Apr/2013 10:00:00  | 1366020000   | Warning   | Deny       | ASA-4-106023  |  |
| 16600917  | 2013-04-15 10:00:00.0  | 15/Apr/2013 10:00:00  | 1366020000   | Warning   | Deny       | ASA-4-106023  |  |
| 16600912  | 2013-04-15 10:00:00.0  | 15/Apr/2013 10:00:00  | 1366020000   | Warning   | Deny       | ASA-4-106023  |  |
| 16600913  | 2013-04-15 10:00:00.0  | 15/Apr/2013 10:00:00  | 1366020000   | Warning   | Deny       | ASA-4-106023  |  |
| 16600915  | 2013-04-15 10:00:00.0  | 15/Apr/2013 10:00:00  | 1366020000   | Warning   | Deny       | ASA-4-106023  |  |
| 16600916  | 2013-04-15 10:00:00.0  | 15/Apr/2013 10:00:00  | 1366020000   | Warning   | Deny       | ASA-4-106023  |  |
| 16600911  | 2013-04-15 10:00:00.0  | 15/Apr/2013 10:00:00  | 1366020000   | Warning   | Deny       | ASA-4-106023  |  |
| 16600920  | 2013-04-15 10:00:00.0  | 15/Apr/2013 10:00:00  | 1366020000   | Info      | Teardown   | ASA-6-302014  |  |
| 16600919  | 2013-04-15 10:00:00.0  | 15/Apr/2013 10:00:00  | 1366020000   | Warning   | Deny       | ASA-4-106023  |  |
+-----------+------------------------+-----------------------+--------------+-----------+------------+---------------+--+
10 rows selected (119.692 seconds)


统计IP出现次数：

0: jdbc:hive2://feng02:10000> select srcIp,count(*) as num  from firewall  group by srcIp order by  num desc limit 100; 
+-----------------+----------+
|      srcIp      |   num    |
+-----------------+----------+
| 10.12.15.152    | 3875019  |
| 10.13.77.49     | 1954742  |
| 10.6.6.7        | 1769857  |
| 10.138.235.111  | 1630920  |
| 10.12.14.15     | 756475   |
| 10.17.15.10     | 720936   |
| 172.10.0.4      | 482568   |
| 10.78.100.150   | 300530   |
| 10.247.58.182   | 292384   |
| 10.38.217.48    | 282588   |
| 172.30.0.4      | 232850   |
| 10.10.11.102    | 205579   |
| 10.170.32.110   | 202735   |
| 10.170.32.181   | 200539   |
| 10.138.214.18   | 197864   |
| 10.247.106.27   | 193912   |
| 10.156.165.120  | 183929   |
| 10.15.7.85      | 172422   |
| 10.0.0.42       | 165063   |
| 10.200.20.2     | 164519   |
| 10.70.68.127    | 161735   |
| 172.20.0.15     | 145924   |
| 10.250.178.101  | 140053   |
| 172.20.0.4      | 122768   |
| 172.30.1.223    | 51329    |
| 172.20.1.81     | 51328    |
| 172.10.2.66     | 51304    |
| 172.20.1.47     | 51298    |
| 172.30.1.218    | 51236    |
| 172.10.2.135    | 51233    |
| 172.20.1.23     | 51199    |
| 172.10.2.106    | 51174    |
| 172.20.0.3      | 20795    |
| 172.10.0.2      | 16296    |
| 10.13.77.4      | 15622    |
| 10.77.101.2     | 15307    |
| 10.77.24.4      | 15274    |
| 10.0.2.181      | 15264    |
| 10.0.3.77       | 15086    |
| 10.58.99.97     | 15021    |
| 10.83.101.25    | 14997    |
| 10.62.15.101    | 14930    |
| 10.1.55.43      | 14783    |
| 172.20.0.2      | 14754    |
| 10.27.200.49    | 14690    |
| 10.18.37.65     | 14645    |
| 10.43.51.79     | 14457    |
| 172.30.0.3      | 11113    |
| 172.30.0.2      | 10129    |
| 172.10.0.3      | 9598     |
| 172.10.0.9      | 3899     |
| 172.10.0.8      | 3852     |
| 172.20.0.6      | 3833     |
| 172.10.0.7      | 3820     |
| 172.10.0.5      | 3805     |
| 10.57.84.113    | 3420     |
| 10.57.105.222   | 3329     |
| 10.170.32.239   | 3231     |
| 172.30.1.215    | 2973     |
| 172.30.1.248    | 2908     |
| 172.30.1.213    | 2890     |
| 172.30.1.217    | 2713     |
| 172.30.1.210    | 2686     |
| 172.20.0.5      | 2668     |
| 172.30.1.205    | 2639     |
| 10.76.187.67    | 2609     |
| 172.30.1.204    | 2556     |
| 172.30.1.227    | 2553     |
| 172.30.1.242    | 2525     |
| 172.30.1.236    | 2521     |
| 10.156.165.212  | 2513     |
| 10.88.52.24     | 2513     |
| 172.30.1.247    | 2513     |
| 10.41.9.89      | 2509     |
| 10.89.101.53    | 2503     |
| 10.88.25.4      | 2494     |
| 172.30.1.243    | 2473     |
| 172.30.1.245    | 2457     |
| 172.30.1.241    | 2442     |
| 10.51.88.167    | 2418     |
| 172.30.1.202    | 2415     |
| 172.30.1.237    | 2414     |
| 10.0.3.178      | 2414     |
| 10.36.156.99    | 2411     |
| 10.32.42.116    | 2410     |
| 10.78.90.9      | 2408     |
| 172.30.1.229    | 2408     |
| 10.33.0.15      | 2406     |
| 10.0.125.6      | 2400     |
| 10.138.78.75    | 2400     |
| 10.51.88.253    | 2400     |
| 10.38.37.13     | 2398     |
| 10.76.187.68    | 2397     |
| 10.76.193.186   | 2397     |
| 10.149.59.54    | 2396     |
| 10.170.148.52   | 2395     |
| 10.179.19.15    | 2395     |
| 10.57.105.44    | 2395     |
| 172.30.1.224    | 2394     |
+-----------------+----------+
|      srcIp      |   num    |
+-----------------+----------+
| 10.38.37.229    | 2393     |
+-----------------+----------+
100 rows selected (65.642 seconds)



0: jdbc:hive2://feng02:10000> select count(*) from firewall;                                                
+-----------+
|    _c0    |
+-----------+
| 16600931  |
+-----------+
1 row selected (47.636 seconds)
wind520
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
专栏目录
Spark SQL常见4种数据源详解
01-19
Spark SQL的DataFrame接口支持多种数据源的操作。一个DataFrame可以进行RDDs方式的操作，也可以被注册为临时表。把DataFrame注册为临时表之后，就可以对该DataFrame执行SQL查询。 Spark SQL的默认数据源为Parquet...
Spark SQL实战(08)-整合Hive
JavaEdge全是干货的技术号
03-26 1606
统计每个人爱好的个数* pk：3* 1）定义函数* 2）注册函数* 3）使用函数。
Spark SQL 与Hive集成
蔡政洁的博客
09-23 1056
目录一、Spark SQL 与Hive集成(spark-shell)（1）添加配置项目（2）启动服务（3）数据测试（4）将数据集写入到MySQL二、Spark SQL 与Hive集成(spark-sql) 一、Spark SQL 与Hive集成(spark-shell) （1）添加配置项目 第一步： 拷贝hive的配置文件hive-site.xml到spark的conf的目录下 cp hive-site.xml /opt/Hadoop/spark/conf 第二步： 添加hive-site.xml中met
Spark对接Hive：整合Hive操作及函数
muyingmiao的专栏
03-12 811
1.拷贝hive-site.xml文件到spark的conf目录下 2.[hadoop@hadoop002 bin]$ ./spark-shell --master local[2] --jars ~/software/mysql-connector-java-5.1.47.jar 注意用5版本的mysql-connector-java scala> spark.sql("sh...
解决:log4j警告:WARN Please initialize the log4j system properly
热门推荐
just_learing的博客
04-26 2万+
解决:log4j警告:WARN Please initialize the log4j system properly
解决执行MapReduce程序控制台没有日志信息WARN Please initialize the log4j system properly
tanxinji的博客
06-24 952
1.在resource目录下创建一个log4j.properties文件 2.添加如下信息:
项目实战——Spark将Hive表的数据写入ElasticSearch（Java版本）
10-24
项目实战：Java一站式解决Hive内用Spark取数，新建ES索引，灌入数据，并且采用ES别名机制，实现ES数据更新的无缝更新，底层采用Spark计算框架，数据较快。
idea中通过spark访问hive
01-07
IDEA中使用Spark SQL 连接Hive 折腾了两个小时，好久没有弄了，记录一下 一、复制hive-site.xml 复制hive-site.xml到idea工程中的resource目录下 二、在windows中安装hadoop并配置环境变量 1、解压hadoop 2、修改...
编译的spark-hive_2.11-2.3.0和 spark-hive-thriftserver_2.11-2.3.0.jar
06-23
spark-hive_2.11-2.3.0 spark-hive-thriftserver_2.11-2.3.0.jar log4j-2.15.0.jar slf4j-api-1.7.7.jar slf4j-log4j12-1.7.25.jar curator-client-2.4.0.jar curator-framework-2.4.0.jar curator-recipes-2.4.0....
Spark SQL源码概览.pdf
01-11
Spark SQL 包含 3 个子项目：Core、Catalyst、Hive。其中 Catalyst 是核心的查询优化引 擎，独立于 Spark 平台；Spark SQL Core 封装 Catalyst，向应用程序提供 SparkSession、Dataset、 DataFrame 等 API...
启动spark服务，解决端口占用的踩坑过程及使用Phoenix操作hbase数据表
阿啄debugIT
07-01 1716
前言 在尝试运行sparksql，发现spark服务没有启动，且发现其端口占用！ 问题处理过程 在liunx系统中，搜索以下是否有spark服务？ [root@A ~]# ps -ef|grep spark root 16970 25666 0 18:19 pts/2 00:00:00 grep --color=auto spark 没有，就去启动spark服务 启动spark服务 [root@A sbin]# cd /home/software/spark2.2/sbi
第73课内幕资料详细版 Spark SQL Thrift Server 实战 每天晚上20：00YY频道现场授课频道68917580
段智华的博客
04-04 2737
第73课 Spark SQL Thrift Server 实战 /* * *王家林老师授课http://weibo.com/ilovepains */   每天晚上20：00YY频道现场授课频道68917580 1.启动hadoop root@master:/usr/local/hadoop-2.6.0/sbin# start-dfs.sh Starting namenodes
Failed to load org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver: org/apache/hadoop/hive/cli/
liyaya0201的博客
04-03 5461
1. 环境介绍 集群：CDH5.12.0 Spark版本：2.4.0（未编译） 执行 spark-sql 的 shell 脚本时，会报如下错误 [root@master bin]# sh spark-sql Warning: Failed to load org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver: org/apache/had...
[spark]Spark thriftserver(HiveServer2) Custom认证
卧龙居
05-27 4856
默认如果不配置的话，是不进行认证的。所以用beeline连接thriftserver时，可以填写任意用户名/密码，均可以认证成功。 下面是进行Custom认证配置的过程： 1.新建一个工程，引入spark-assembly-1.3.0-hadoop2.4.0.jar 需要实现PasswdAuthenticationProvider接口，代码如下： package org.apache.ha
ClassNotFoundException: org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver解决办法
mnasd的博客
05-21 5067
我们通过源码编译的spark-2.3.0来启动spark-sql进行sql操作，结果出现如下错误：Spark assembly has been built with Hive, including Datanucleus jars on classpath java.lang.ClassNotFoundException: org.apache.spark.sql.hive.thriftse...
spark基础之Spark SQL和Hive的集成以及ThriftServer配置
happy19870612's blog
10-27 1万+
如果希望Maven编译Spark时支持Hive,需要给定-Phive -Phive-thriftserver。比如比如：mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0-Phive -Phive-thriftserver -DskipTests clean package   一 Spark SQL和Hive集成 1.1 将hive的配置文件hiv
spark SQL Running the Thrift JDBC/ODBC server
风行天下
03-04 9494
Running the Thrift JDBC/ODBC server 1:运行 ./sbin/start-thriftserver.sh  --hiveconf hive.server2.thrift.port=10000  --hiveconf hive.server2.thrift.bind.host=feng02 --master spark://feng02:7077 --d
awk进行文本处理
风行天下
08-21 1178
1 awk的内置变量 awk 所内建的字段变量及其涵意如下 : 字段变量 含义 $0 一字符串, 其内容为目前 awk 所读入的数据行. $1 $0 上第一个字段的数据. $2 $0 上第二个字段的数据. ... 其余类推
如何用spark sql统计hive表数据量
最新发布
06-06
您可以使用以下步骤来使用Spark SQL统计Hive表数据量： 1. 创建SparkSession对象： ```scala val spark = SparkSession .builder() .appName("Spark SQL Hive Example") .enableHiveSupport() .getOrCreate...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值