spark 1.3.0 测试

最新推荐文章于 2021-08-22 11:41:31 发布
最新推荐文章于 2021-08-22 11:41:31 发布
阅读量7.1k 收藏 1
点赞数
分类专栏： Spark 文章标签： spark sql
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/wind520/article/details/44040261
版权 



[jifeng@feng02 spark-1.3.0-bin-2.4.1]$ master=spark://feng02:7077 ./bin/spark-shell
Spark assembly has been built with Hive, including Datanucleus jars on classpath
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/03/03 15:30:26 INFO SecurityManager: Changing view acls to: jifeng
15/03/03 15:30:26 INFO SecurityManager: Changing modify acls to: jifeng
15/03/03 15:30:26 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jifeng); users with modify permissions: Set(jifeng)
15/03/03 15:30:26 INFO HttpServer: Starting HTTP Server
15/03/03 15:30:27 INFO Server: jetty-8.y.z-SNAPSHOT
15/03/03 15:30:27 INFO AbstractConnector: Started SocketConnector@0.0.0.0:49848
15/03/03 15:30:27 INFO Utils: Successfully started service 'HTTP class server' on port 49848.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.3.0
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_45)
Type in expressions to have them evaluated.
Type :help for more information.
15/03/03 15:30:33 INFO SparkContext: Running Spark version 1.3.0
15/03/03 15:30:33 INFO SecurityManager: Changing view acls to: jifeng
15/03/03 15:30:33 INFO SecurityManager: Changing modify acls to: jifeng
15/03/03 15:30:33 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jifeng); users with modify permissions: Set(jifeng)
15/03/03 15:30:33 INFO Slf4jLogger: Slf4jLogger started
15/03/03 15:30:34 INFO Remoting: Starting remoting
15/03/03 15:30:34 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@feng02:36823]
15/03/03 15:30:34 INFO Utils: Successfully started service 'sparkDriver' on port 36823.
15/03/03 15:30:34 INFO SparkEnv: Registering MapOutputTracker
15/03/03 15:30:34 INFO SparkEnv: Registering BlockManagerMaster
15/03/03 15:30:34 INFO DiskBlockManager: Created local directory at /tmp/spark-303ca9ef-f76b-4c9c-a315-a10351265aa4/blockmgr-2c1283b6-9466-40f2-83af-bc408a5e4ec0
15/03/03 15:30:34 INFO MemoryStore: MemoryStore started with capacity 267.3 MB
15/03/03 15:30:34 INFO HttpFileServer: HTTP File server directory is /tmp/spark-e7ee2d00-4c21-4221-9f65-3e86d509235a/httpd-729605bc-2cdf-4d64-9b83-842bfedc5223
15/03/03 15:30:34 INFO HttpServer: Starting HTTP Server
15/03/03 15:30:34 INFO Server: jetty-8.y.z-SNAPSHOT
15/03/03 15:30:34 INFO AbstractConnector: Started SocketConnector@0.0.0.0:38734
15/03/03 15:30:34 INFO Utils: Successfully started service 'HTTP file server' on port 38734.
15/03/03 15:30:34 INFO SparkEnv: Registering OutputCommitCoordinator
15/03/03 15:30:34 INFO Server: jetty-8.y.z-SNAPSHOT
15/03/03 15:30:35 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
15/03/03 15:30:35 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/03/03 15:30:35 INFO SparkUI: Started SparkUI at http://feng02:4040
15/03/03 15:30:35 INFO Executor: Starting executor ID <driver> on host localhost
15/03/03 15:30:35 INFO Executor: Using REPL class URI: http://10.6.3.201:49848
15/03/03 15:30:35 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@feng02:36823/user/HeartbeatReceiver
15/03/03 15:30:35 INFO NettyBlockTransferService: Server created on 51265
15/03/03 15:30:35 INFO BlockManagerMaster: Trying to register BlockManager
15/03/03 15:30:35 INFO BlockManagerMasterActor: Registering block manager localhost:51265 with 267.3 MB RAM, BlockManagerId(<driver>, localhost, 51265)
15/03/03 15:30:35 INFO BlockManagerMaster: Registered BlockManager
15/03/03 15:30:35 INFO SparkILoop: Created spark context..
Spark context available as sc.
15/03/03 15:30:36 INFO SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.

scala> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
sqlContext: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@7e6c0e3

scala> import sqlContext.createSchemaRDD
<console>:23: error: value createSchemaRDD is not a member of org.apache.spark.sql.SQLContext
       import sqlContext.createSchemaRDD
              ^

scala> case class Person(name: String, age: Int)
defined class Person

scala> val people = sc.textFile("examples/src/main/resources/people.txt").map(_.split(",")).map(p => Person(p(0), p(1).trim.toInt))
15/03/03 15:42:12 INFO MemoryStore: ensureFreeSpace(163705) called with curMem=0, maxMem=280248975
15/03/03 15:42:12 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 159.9 KB, free 267.1 MB)
15/03/03 15:42:12 INFO MemoryStore: ensureFreeSpace(22736) called with curMem=163705, maxMem=280248975
15/03/03 15:42:12 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 22.2 KB, free 267.1 MB)
15/03/03 15:42:12 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:51265 (size: 22.2 KB, free: 267.2 MB)
15/03/03 15:42:12 INFO BlockManagerMaster: Updated info of block broadcast_0_piece0
15/03/03 15:42:12 INFO SparkContext: Created broadcast 0 from textFile at <console>:23
people: org.apache.spark.rdd.RDD[Person] = MapPartitionsRDD[3] at map at <console>:23

scala> people.registerTempTable("people")
<console>:26: error: value registerTempTable is not a member of org.apache.spark.rdd.RDD[Person]
              people.registerTempTable("people")
                     ^

scala> val teenagers = sqlContext.sql("SELECT name FROM people WHERE age >= 13 AND age <= 19")
java.lang.RuntimeException: Table Not Found: people
        at scala.sys.package$.error(package.scala:27)
        at org.apache.spark.sql.catalyst.analysis.SimpleCatalog$$anonfun$1.apply(Catalog.scala:111)
        at org.apache.spark.sql.catalyst.analysis.SimpleCatalog$$anonfun$1.apply(Catalog.scala:111)
        at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
        at scala.collection.AbstractMap.getOrElse(Map.scala:58)
        at org.apache.spark.sql.catalyst.analysis.SimpleCatalog.lookupRelation(Catalog.scala:111)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.getTable(Analyzer.scala:258)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:270)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:265)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:187)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:187)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:50)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:186)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:207)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
        at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
        at scala.collection.AbstractIterator.to(Iterator.scala:1157)
        at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:236)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:192)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:207)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
        at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
        at scala.collection.AbstractIterator.to(Iterator.scala:1157)
        at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:236)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:192)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:177)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:265)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:255)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1$$anonfun$apply$2.apply(RuleExecutor.scala:61)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1$$anonfun$apply$2.apply(RuleExecutor.scala:59)
        at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111)
        at scala.collection.immutable.List.foldLeft(List.scala:84)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1.apply(RuleExecutor.scala:59)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1.apply(RuleExecutor.scala:51)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.apply(RuleExecutor.scala:51)
        at org.apache.spark.sql.SQLContext$QueryExecution.analyzed$lzycompute(SQLContext.scala:1063)
        at org.apache.spark.sql.SQLContext$QueryExecution.analyzed(SQLContext.scala:1063)
        at org.apache.spark.sql.DataFrameImpl.<init>(DataFrameImpl.scala:63)
        at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:39)
        at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:915)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:23)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:28)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:30)
        at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:32)
        at $iwC$$iwC$$iwC$$iwC.<init>(<console>:34)
        at $iwC$$iwC$$iwC.<init>(<console>:36)
        at $iwC$$iwC.<init>(<console>:38)
        at $iwC.<init>(<console>:40)
        at <init>(<console>:42)
        at .<init>(<console>:46)
        at .<clinit>(<console>)
        at .<init>(<console>:7)
        at .<clinit>(<console>)
        at $print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
        at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
        at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)


scala> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
sqlContext: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@50f71e2b

scala> val people = sc.textFile("examples/src/main/resources/people.txt")val people = sc.textFile("examples/src/main/resources/people.txt")
<console>:1: error: ';' expected but 'val' found.
       val people = sc.textFile("examples/src/main/resources/people.txt")val people = sc.textFile("examples/src/main/resources/people.txt")
                                                                         ^

scala> val people = sc.textFile("examples/src/main/resources/people.txt")
15/03/03 16:16:00 INFO MemoryStore: ensureFreeSpace(163753) called with curMem=186441, maxMem=280248975
15/03/03 16:16:00 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 159.9 KB, free 266.9 MB)
15/03/03 16:16:00 INFO MemoryStore: ensureFreeSpace(22736) called with curMem=350194, maxMem=280248975
15/03/03 16:16:00 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 22.2 KB, free 266.9 MB)
15/03/03 16:16:00 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:51265 (size: 22.2 KB, free: 267.2 MB)
15/03/03 16:16:00 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0
15/03/03 16:16:00 INFO SparkContext: Created broadcast 1 from textFile at <console>:21
people: org.apache.spark.rdd.RDD[String] = examples/src/main/resources/people.txt MapPartitionsRDD[5] at textFile at <console>:21

scala> val schemaString = "name age"
schemaString: String = name age

scala> import org.apache.spark.sql._
import org.apache.spark.sql._

scala> val schema =
     |   StructType(
     |     schemaString.split(" ").map(fieldName => StructField(fieldName, StringType, true)))
<console>:25: error: not found: value StructType
         StructType(
         ^

scala> val schema = StructType(schemaString.split(" ").map(fieldName => StructField(fieldName, StringType, true)))
<console>:24: error: not found: value StructType
       val schema = StructType(schemaString.split(" ").map(fieldName => StructField(fieldName, StringType, true)))
                    ^

scala> val rowRDD = people.map(_.split(",")).map(p => Row(p(0), p(1).trim))
rowRDD: org.apache.spark.rdd.RDD[org.apache.spark.sql.Row] = MapPartitionsRDD[7] at map at <console>:26

scala> val peopleSchemaRDD = sqlContext.applySchema(rowRDD, schema)
<console>:30: error: package schema is not a value
       val peopleSchemaRDD = sqlContext.applySchema(rowRDD, schema)
                                                            ^

scala> peopleSchemaRDD.registerTempTable("people")
<console>:23: error: not found: value peopleSchemaRDD
              peopleSchemaRDD.registerTempTable("people")
              ^

scala> val results = sqlContext.sql("SELECT name FROM people")
java.lang.RuntimeException: Table Not Found: people
        at scala.sys.package$.error(package.scala:27)
        at org.apache.spark.sql.catalyst.analysis.SimpleCatalog$$anonfun$1.apply(Catalog.scala:111)
        at org.apache.spark.sql.catalyst.analysis.SimpleCatalog$$anonfun$1.apply(Catalog.scala:111)
        at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
        at scala.collection.AbstractMap.getOrElse(Map.scala:58)
        at org.apache.spark.sql.catalyst.analysis.SimpleCatalog.lookupRelation(Catalog.scala:111)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.getTable(Analyzer.scala:258)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:270)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:265)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:187)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:187)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:50)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:186)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:207)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
        at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
        at scala.collection.AbstractIterator.to(Iterator.scala:1157)
        at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:236)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:192)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:177)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:265)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:255)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1$$anonfun$apply$2.apply(RuleExecutor.scala:61)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1$$anonfun$apply$2.apply(RuleExecutor.scala:59)
        at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111)
        at scala.collection.immutable.List.foldLeft(List.scala:84)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1.apply(RuleExecutor.scala:59)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1.apply(RuleExecutor.scala:51)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.apply(RuleExecutor.scala:51)
        at org.apache.spark.sql.SQLContext$QueryExecution.analyzed$lzycompute(SQLContext.scala:1063)
        at org.apache.spark.sql.SQLContext$QueryExecution.analyzed(SQLContext.scala:1063)
        at org.apache.spark.sql.DataFrameImpl.<init>(DataFrameImpl.scala:63)
        at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:39)
        at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:915)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:26)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:33)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:35)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:37)
        at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:39)
        at $iwC$$iwC$$iwC$$iwC.<init>(<console>:41)
        at $iwC$$iwC$$iwC.<init>(<console>:43)
        at $iwC$$iwC.<init>(<console>:45)
        at $iwC.<init>(<console>:47)
        at <init>(<console>:49)
        at .<init>(<console>:53)
        at .<clinit>(<console>)
        at .<init>(<console>:7)
        at .<clinit>(<console>)
        at $print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
        at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
        at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)


scala> results.map(t => "Name: " + t(0)).collect().foreach(println)
<console>:23: error: not found: value results
              results.map(t => "Name: " + t(0)).collect().foreach(println)
              ^

scala> rowRDD.collect()
15/03/03 16:19:12 INFO FileInputFormat: Total input paths to process : 1
15/03/03 16:19:12 INFO SparkContext: Starting job: collect at <console>:29
15/03/03 16:19:12 INFO DAGScheduler: Got job 0 (collect at <console>:29) with 1 output partitions (allowLocal=false)
15/03/03 16:19:12 INFO DAGScheduler: Final stage: Stage 0(collect at <console>:29)
15/03/03 16:19:12 INFO DAGScheduler: Parents of final stage: List()
15/03/03 16:19:12 INFO DAGScheduler: Missing parents: List()
15/03/03 16:19:12 INFO DAGScheduler: Submitting Stage 0 (MapPartitionsRDD[7] at map at <console>:26), which has no missing parents
15/03/03 16:19:12 INFO MemoryStore: ensureFreeSpace(3056) called with curMem=372930, maxMem=280248975
15/03/03 16:19:12 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 3.0 KB, free 266.9 MB)
15/03/03 16:19:12 INFO MemoryStore: ensureFreeSpace(2122) called with curMem=375986, maxMem=280248975
15/03/03 16:19:12 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.1 KB, free 266.9 MB)
15/03/03 16:19:12 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:51265 (size: 2.1 KB, free: 267.2 MB)
15/03/03 16:19:12 INFO BlockManagerMaster: Updated info of block broadcast_2_piece0
15/03/03 16:19:12 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:839
15/03/03 16:19:12 INFO DAGScheduler: Submitting 1 missing tasks from Stage 0 (MapPartitionsRDD[7] at map at <console>:26)
15/03/03 16:19:12 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
15/03/03 16:19:12 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, PROCESS_LOCAL, 1349 bytes)
15/03/03 16:19:12 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
15/03/03 16:19:12 INFO HadoopRDD: Input split: file:/home/jifeng/hadoop/spark-1.3.0-bin-2.4.1/examples/src/main/resources/people.txt:0+32
15/03/03 16:19:12 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
15/03/03 16:19:12 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
15/03/03 16:19:12 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
15/03/03 16:19:12 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
15/03/03 16:19:12 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
15/03/03 16:19:12 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 2016 bytes result sent to driver
15/03/03 16:19:12 INFO DAGScheduler: Stage 0 (collect at <console>:29) finished in 0.196 s
15/03/03 16:19:12 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 147 ms on localhost (1/1)
15/03/03 16:19:12 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
15/03/03 16:19:12 INFO DAGScheduler: Job 0 finished: collect at <console>:29, took 0.348604 s
res3: Array[org.apache.spark.sql.Row] = Array([Michael,29], [Andy,30], [Justin,19])


Hive Tables

[jifeng@feng02 spark-1.3.0-bin-2.4.1]$ master=spark://feng02:7077 ./bin/spark-shell
Spark assembly has been built with Hive, including Datanucleus jars on classpath
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/03/03 22:55:12 INFO SecurityManager: Changing view acls to: jifeng
15/03/03 22:55:12 INFO SecurityManager: Changing modify acls to: jifeng
15/03/03 22:55:12 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jifeng); users with modify permissions: Set(jifeng)
15/03/03 22:55:12 INFO HttpServer: Starting HTTP Server
15/03/03 22:55:12 INFO Server: jetty-8.y.z-SNAPSHOT
15/03/03 22:55:12 INFO AbstractConnector: Started SocketConnector@0.0.0.0:56298
15/03/03 22:55:12 INFO Utils: Successfully started service 'HTTP class server' on port 56298.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.3.0
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_45)
Type in expressions to have them evaluated.
Type :help for more information.
15/03/03 22:55:18 INFO SparkContext: Running Spark version 1.3.0
15/03/03 22:55:18 INFO SecurityManager: Changing view acls to: jifeng
15/03/03 22:55:18 INFO SecurityManager: Changing modify acls to: jifeng
15/03/03 22:55:18 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jifeng); users with modify permissions: Set(jifeng)
15/03/03 22:55:19 INFO Slf4jLogger: Slf4jLogger started
15/03/03 22:55:19 INFO Remoting: Starting remoting
15/03/03 22:55:19 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@feng02:59702]
15/03/03 22:55:19 INFO Utils: Successfully started service 'sparkDriver' on port 59702.
15/03/03 22:55:19 INFO SparkEnv: Registering MapOutputTracker
15/03/03 22:55:19 INFO SparkEnv: Registering BlockManagerMaster
15/03/03 22:55:19 INFO DiskBlockManager: Created local directory at /tmp/spark-cbda6663-0eeb-46f3-942a-b02b202adb53/blockmgr-3cf465f3-5759-49a5-8d7e-4b35d57e9a4a
15/03/03 22:55:19 INFO MemoryStore: MemoryStore started with capacity 267.3 MB
15/03/03 22:55:19 INFO HttpFileServer: HTTP File server directory is /tmp/spark-529754e8-50f5-45cb-aa62-4f55a6035c61/httpd-d35daa35-8e66-41a9-aa84-dd40e2273599
15/03/03 22:55:19 INFO HttpServer: Starting HTTP Server
15/03/03 22:55:19 INFO Server: jetty-8.y.z-SNAPSHOT
15/03/03 22:55:19 INFO AbstractConnector: Started SocketConnector@0.0.0.0:42197
15/03/03 22:55:19 INFO Utils: Successfully started service 'HTTP file server' on port 42197.
15/03/03 22:55:19 INFO SparkEnv: Registering OutputCommitCoordinator
15/03/03 22:55:20 INFO Server: jetty-8.y.z-SNAPSHOT
15/03/03 22:55:20 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
15/03/03 22:55:20 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/03/03 22:55:20 INFO SparkUI: Started SparkUI at http://feng02:4040
15/03/03 22:55:20 INFO Executor: Starting executor ID <driver> on host localhost
15/03/03 22:55:20 INFO Executor: Using REPL class URI: http://10.6.3.201:56298
15/03/03 22:55:20 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@feng02:59702/user/HeartbeatReceiver
15/03/03 22:55:20 INFO NettyBlockTransferService: Server created on 41403
15/03/03 22:55:20 INFO BlockManagerMaster: Trying to register BlockManager
15/03/03 22:55:20 INFO BlockManagerMasterActor: Registering block manager localhost:41403 with 267.3 MB RAM, BlockManagerId(<driver>, localhost, 41403)
15/03/03 22:55:20 INFO BlockManagerMaster: Registered BlockManager
15/03/03 22:55:20 INFO SparkILoop: Created spark context..
Spark context available as sc.
15/03/03 22:55:21 INFO SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.

scala> val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
sqlContext: org.apache.spark.sql.hive.HiveContext = org.apache.spark.sql.hive.HiveContext@3d9ece38

scala> sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
15/03/03 22:55:59 WARN HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect.  Use hive.hmshandler.retry.* instead
15/03/03 22:55:59 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
15/03/03 22:55:59 INFO ObjectStore: ObjectStore, initialize called
15/03/03 22:56:00 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
15/03/03 22:56:00 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
15/03/03 22:56:00 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
        at org.apache.spark.sql.hive.HiveContext$$anonfun$5.apply(HiveContext.scala:216)
        at org.apache.spark.sql.hive.HiveContext$$anonfun$5.apply(HiveContext.scala:212)
        at scala.Option.orElse(Option.scala:257)
        at org.apache.spark.sql.hive.HiveContext.x$4$lzycompute(HiveContext.scala:212)
        at org.apache.spark.sql.hive.HiveContext.x$4(HiveContext.scala:210)
        at org.apache.spark.sql.hive.HiveContext.hiveconf$lzycompute(HiveContext.scala:210)
        at org.apache.spark.sql.hive.HiveContext.hiveconf(HiveContext.scala:210)
        at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:71)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:24)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
        at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:33)
        at $iwC$$iwC$$iwC$$iwC.<init>(<console>:35)
        at $iwC$$iwC$$iwC.<init>(<console>:37)
        at $iwC$$iwC.<init>(<console>:39)
        at $iwC.<init>(<console>:41)
        at <init>(<console>:43)
        at .<init>(<console>:47)
        at .<clinit>(<console>)
        at .<init>(<console>:7)
        at .<clinit>(<console>)
        at $print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
        at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
        at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
        ... 54 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
        ... 59 more
Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
NestedThrowables:
java.lang.reflect.InvocationTargetException
        at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:587)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:788)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
        at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
        at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
        at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:310)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:339)
        at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:248)
        at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:223)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
        at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58)
        at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:497)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:475)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
        ... 64 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
        at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:325)
        at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:282)
        at org.datanucleus.store.AbstractStoreManager.<init>(AbstractStoreManager.java:240)
        at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:286)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
        at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
        at org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1187)
        at org.datanucleus.NucleusContext.initialise(NucleusContext.java:356)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:775)
        ... 93 more
Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "dbcp-builtin" plugin to create a ConnectionPool gave an error : The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:259)
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:131)
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:85)
        ... 111 more
Caused by: org.datanucleus.store.rdbms.connectionpool.DatastoreDriverNotFoundException: The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
        at org.datanucleus.store.rdbms.connectionpool.AbstractConnectionPoolFactory.loadDriver(AbstractConnectionPoolFactory.java:58)
        at org.datanucleus.store.rdbms.connectionpool.DBCPBuiltinConnectionPoolFactory.createConnectionPool(DBCPBuiltinConnectionPoolFactory.java:49)
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:238)
        ... 113 more



wind520
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 3
    评论
专栏目录
maven创建项目报错问题
RoseOnly
07-14 8842
报错显示如下： Description Resource Path Location Type The container 'Maven Dependencies' references non existing library 'C:\Users\CCai\.m2\repository\com\github\pagehelper\pagehelper\3.4.2-fix\pagehelp
maven 打包scala报错问题：error: not found: object xxx
祁东握力的博客
05-13 1444
我的完整报错是：E:\daydayup\spark\src\main\scala\com\tt\KafkaSparkStreamTest.scala:3: error: not found: object kafka 就这些信息根本看不出啥。。还是靠百度，找到一篇资料说是maven仓库移动一下，然后他的文章中我看他的仓库是从路径有中文的移动到了全英文的路径下。。然后就没问题了，我也移动了maven仓库到英文路径下，然后idea setting重新设置仓库位置为英文下的那个，然后就成功了。，，可气.
小白学scala日记——List is not a value
qq_42515611的博客
08-22 485
出现List is not a value的问题 解决办法导包（import scala.collection.immutable.List）：
在spark-shell中解决error: not found: value StructType/StructField/StringType问题
someby的博客
02-25 2173
问题描述 在Spark-shell中运行官方文档提供的例子时，出现error: not found: value StructType/StructField/StringType的错误 解决方案 将相关的类导入即可 import org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType...
RDD
Realoyou的博客
05-18 747
RDD概念 RDD源码中的描述： A Resilient Distributed Dataset (RDD), the basic abstraction in Spark. Represents an immutable,partitioned collection of elements that can be operated on in parallel. This class c...
spark1.3源码
12-27
Spark是Apache软件基金会下的一个开源大数据处理框架，其1.3.0版本是Spark发展历史上的一个重要里程碑。这个版本在前一版本的基础上进行了许多优化和改进，为数据科学家和工程师提供了更高效、易用的数据处理工具。...
spark安装部署手册
07-07
- 可以参考Spark官方文档中的示例程序来进行测试。 #### 五、注意事项 - **版本兼容性**: - 在安装Scala前，请确保选择与Spark版本兼容的Scala版本。 - **操作系统差异**: - 在不同的操作系统上安装Spark时，...
spark-integration-tests:Spark 的集成测试
06-02
Spark 集成测试 该项目包含基于的 Spark 集成测试，包括 Spark 独立集群管理器的容错测试。 安装/设置 安装 Docker 该项目依赖于 Docker >= 1.3.0（它可能适用于早期版本，但尚未经过测试）。 在 Linux 上 安装 ...
spark开发环境搭建
12-09
<spark.version>1.3.0</spark.version> <scala.version>2.10.4 <!-- 其他配置 --> <groupId>org.apache.spark <artifactId>spark-core_2.10 ${spark.version} <!-- 其他依赖 --> ``` 通过这种方式，...
spark_integration
05-01
CDH 5.4.2和Spark 1.3.0（作业运行时） Elasticsearch 1.7.2 Apache Kafka 0.8.2.2 集会 与Maven组装。 mvn clean install 将spark_integration-1.0.jar与包含的依赖项一起使用 配置 可以从命令行提供参数来配置...
Scala编程中常见错误：Error:(24, 29) value foreach is not a member of java.util.Set[String]
DataFlow范式
01-16 1万+
问题: 在Scala编程开发中, 经常会出现类似如下的错误, Error:(24, 29) value foreach is not a member of java.util.Set[String]     for (key                             ^ 或 Error:(21, 22) value filter is not a member of
使用mongoose.schema出现 ‘Schema is not a constructor’错误
InterestAndFun的博客
07-14 1073
问题描述 在学习 mongoose 的用法时，发现模仿一些教程的编写方式报错，无法使用schema var Schema = mongoose.schema var userSchema = new Schema({ }) 报错截图 百度找了好久，都没找到出现同样问题的人，于是我去查看官方文档 解决方法 修改成如下形式即可 // var Schema = mongoose.schema const { Schema } = mongoose; ...
服务器上Python和matlab混编时，遇到问题：‘matlab‘ is not a package的解决方法
qq_37395293的博客
10-06 2951
服务器上使用import matlab.engine 命令遇到’matlab’ is not a package的解决方法 前言 最近在服务器上使用python和matlab混编时遇到了这个问题 网上很多解决方案，一一试了过去，发现并不好用（动不动就劝人卸载重按-_-） 我的matlab版本是2020a，python版本的话3.6和3.7我都试了都可以，现在也不存在一些回答中的版本支持问题（都0202年了） 解决过程 与大多的回答一样，以下几步是必须的： 在matlab软件安装路径下，找到文件目录“./e
Package name is not valid解决
热门推荐
刘程的博客
04-16 3万+
刚刚新建项目，将com打成con，提示Package name is not valid，在eclipse中有些单词不能命名。
Spark Sql中DataSet与RDD互操作的两种方式
weixin_45568872的博客
02-07 510
介绍 Spark Sql支持两种将现有RDD转换为DataSet的方式 1.使用反射来推断包含特定对象类型RDD的数据结构 当在编写Spark应用时已经了解数据结构，即应用可以推断RDD的数据结构时，即可使用此方法，这种基于反射的方法可以使代码更为简洁 2.通过编程界面来创建DataSet，然后将其映射到RDD上 当在运行Spark应用时才得知数据结构，即应用无法推断RDD的数据结构时，则使用此方...
scala 基本语法-cp
hzdxw的专栏
07-05 7941
scala 基本语法-cp
value toDF is not a member of org.apache.spark.rdd.RDD[People]
ronaldo4511的博客
11-28 6192
编译如下代码时，出现value toDF is not a member of org.apache.spark.rdd.RDD[People]  错误 val rdd : RDD[People]= sparkSession.sparkContext.textFile(hdfsFile,2).map(line => line.split(",")).map(arr => People(arr(0
在spark中使用Hive报错error: not found: value sqlContext
图特摩斯科技-博客
05-15 1万+
<console>:16: error: not found: value sqlContext import sqlContext.implicits._ ^ <console>:16: error: not found: value sqlContext import sqlContext.sql ^ raini@biyuzhe:~$ spark-shell --jars /home/raini/spa
Spark之日志数据清洗及分析（详细解说）
绿萝蔓蔓绕枝生
11-19 4275
一、日志数据清洗及分析 1、数据清洗 基本步骤： 按照Tab切割数据 过滤掉字段数量少于8个的数据 按照第一列和第二列对数据进行去重 过滤掉状态码非200的数据 过滤掉event_time为空的数据 将url按照&以及=切割 保存数据：将数据写入mysql表中 日志拆分字段： event_time url method status sip user_uip action_prepend action_client 如下是日志中的一条数据按照Tab分隔后的示例，每一行代表一个字段，分别以上一
OpenFlow交换机规范v1.3.0解析
"OpenFlow交换机规范(概要)，Version1.3.0" OpenFlow是一种网络虚拟化技术，用于实现网络设备（如交换机）的编程和控制，以实现更灵活的流量管理和网络服务。在OpenFlow 1.3.0版本中，交换机的核心组成部分包括流表...
评论 3
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值