Spark操作Hive，Hdfs，Elasticsearch，Mongodb，mysql，kafka，hbase（持续更新）

操作hive

碰到的报错： tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.mapred.FileInputFormat

解决办法，添加低版本Hadoop依赖

    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>2.7.7</version>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-mapreduce-client-core</artifactId>
      <version>2.7.7</version>
    </dependency>

hive-site.xml配置

  <property>
          <name>hive.metastore.uris</name>
          <value>thrift://master:9083</value>
   </property>

启动metastore

nohup hive --service metastore &

hive里面要创建好外部表，hdfs里面才会有warehouse路径，然后配置hive.metastore.uris和spark.sql.warehouse.dir参数就可以了。