spark java hive job_EMR：如何将Spark与Hive集成？

只需确保使用AWS的dynamo连接器opensource . 默认情况下，它可在EMR AFAIK上使用 .

使用DynamoDBStorageHandler类创建表的语法：

CREATE EXTERNAL TABLE hive_tablename (

hive_column1_name column1_datatype,

hive_column2_name column2_datatype

)

STORED BY 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler'

TBLPROPERTIES (

"dynamodb.table.name" = "dynamodb_tablename",

"dynamodb.column.mapping" =

"hive_column1_name:dynamodb_attribute1_name,hive_column2_name:dynamodb_attribute2_name"

);

对于任何Spark Job，您需要具备以下内容：

$ spark-shell --jars /usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar

...

import org.apache.hadoop.io.Text;

import org.apache.hadoop.dynamodb.DynamoDBItemWritable

import org.apache.hadoop.dynamodb.read.DynamoDBInputFormat

import org.apache.hadoop.dynamodb.write.DynamoDBOutputFormat

import org.apache.hadoop.mapred.JobConf

import org.apache.hadoop.io.LongWritable

var jobConf = new JobConf(sc.hadoopConfiguration)

jobConf.set("dynamodb.input.tableName", "myDynamoDBTable")

jobConf.set("mapred.output.format.class", "org.apache.hadoop.dynamodb.write.DynamoDBOutputFormat")

jobConf.set("mapred.input.format.class", "org.apache.hadoop.dynamodb.read.DynamoDBInputFormat")

var orders = sc.hadoopRDD(jobConf, classOf[DynamoDBInputFormat], classOf[Text], classOf[DynamoDBItemWritable])

orders.count()