Springboot with Impala and Kudu

转自

How to run this app

Please follow the instructions below to get kudu running locally with java and impala driver.

Download the kudu quickstart VM

Kudu Quick Start VM

Create the department file to load

You can use this python script to create the department file to load: name it generate.py

#!/usr/local/bin/python3
default_path = '/Users/ryang1/Desktop'
def generateDepartmentMasterFile():
    """
    Company Id | Department Code | Department Description
    """
    file = open(default_path+'/dept.txt','a')
    for x in range (1,501):
        file.write('Company{0}|{0}|Dept{0} 
Description\n'.format(str(x)))
generateDepartmentMasterFile()

Run it using

$ python3 generate.py

Load the department file

Start your kudu quickstart VM. Once it has started run this command to find the vm ip. Should start with 192.168.*

$ ip addr show
[demo@quickstart ~]$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue 
state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
pfifo_fast state UP qlen 1000
    link/ether 08:00:27:8a:40:9a brd ff:ff:ff:ff:ff:ff
    inet 192.168.57.100/24 brd 192.168.57.255 scope global 
eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
pfifo_fast state UP qlen 1000
    link/ether 08:00:27:16:94:b5 brd ff:ff:ff:ff:ff:ff
    inet 10.0.3.15/24 brd 10.0.3.255 scope global eth1

Once you figure out your vm ip address, we will copy over the dept.txt file we created earlier with the python script and put it into hdfs.

ssh username:demo password:demo

$ scp dept.txt demo@192.168.57.100:~/
$ ssh demo@192.168.57.100
$ hdfs dfs -put -f dept.txt /data/dept

Create the department table in kudu/impala

CREATE EXTERNAL TABLE department_raw (
    company_id string,
    department_code int,
    department_description string)
    ROW FORMAT DELIMITED
    FIELDS TERMINATED BY '|'
    LOCATION '/data/dept/';
CREATE TABLE department
DISTRIBUTE BY HASH (company_id) INTO 16 BUCKETS
TBLPROPERTIES(
'storage_handler' = 
'com.cloudera.kudu.hive.KuduStorageHandler',
'kudu.table_name' = 'department',
'kudu.master_addresses' = '127.0.0.1',
'kudu.key_columns' = 'company_id'
 ) AS SELECT * FROM department_raw;
# Set the database configuration

In the project search for src/main/java/config/DatabaseConfig.java and set the datasource url to your kudu/impala VM url

@Bean(name="dataSource")
public DataSource dataSource() {
    DataSourceBuilder dataSourceBuilder = DataSourceBuilder.create();

    //CHANGE ME!
    dataSourceBuilder.url("jdbc:impala://192.168.57.100:21050");

    dataSourceBuilder.username("demo");
    dataSourceBuilder.password("demo");
    dataSourceBuilder.driverClassName("com.cloudera.impala.jdbc4.Driver");
    return dataSourceBuilder.build();
}

Setting up intellij

In order to run the spring boot app you will need to set the run configuration Run -> Edit Configurations Press the + icon create a new maven entry name it to springboot-run change the command line input field to

spring-boot:run

hit apply/save.

You are all set. Hit run.

To see it in action go to your browser and visit url: http://localhost:8080/department?companyId=Company1