电动车传感器数据处理,入库手动版

基于hue

Hue是cdh专门的一套web管理器，它包括3个部分hue ui，hue server，hue
db。hue提供所有的cdh组件的shell界面的接口。你可以在hue编写mr，查看修改hdfs的文件，管理hive的元数据，运行Sqoop，编写Oozie工作流等大量工作。

2.在服务器中处理数据，并且上传

之前的环境安装

python3.x以上

安装豆瓣的包

pip3 install matplotlib -i http://pypi.douban.com/simple --trusted-host pypi.douban.com
pip3 install numpy -i http://pypi.douban.com/simple --trusted-host pypi.douban.com
pip3 install pandas -i http://pypi.douban.com/simple --trusted-host pypi.douban.com
pip3 install seaborn scipy  -i http://pypi.douban.com/simple --trusted-host pypi.douban.com
pip3 install copy  -i http://pypi.douban.com/simple --trusted-host pypi.douban.com
pip3 install xlrd  -i http://pypi.douban.com/simple --trusted-host pypi.douban.com

数据处理（复用逍遥大神的代码）

python3 data_process1.py ./data_9_2019041020_csv.csv ./2.csv

上传数据到hdfs上

sudo -u hdfs hdfs dfs -put 2.csv /user/admin/data

在hive中建表

create table ele_car_sensor_data2(
Time_string string,
Time string,
Max_Single_Voltage string,
Min_Single_Voltage string,
Max_Single_Temp string,
Min_Single_Temp string,
Charger_Input_Current string,
Charger_Input_Volt string,
Charger_Output_Current string,
Charger_Output_Volt string,
PTC_Power string,
DCDC_High_Voltage string,
DCDC_High_Current string,
DCDC_Low_Voltage string,
DCDC_Low_Current string,
DCDC_Temp string,
Before_Filter_DCDC_Volt string,
After_Filter_DCDC_Volt string,
Before_Filter_TM_Current string,
After_Filter_TM_Current string,
EDU_Coolant_Temp string,
TM_Stator_Temp string,
TM_Inverter_Temp string,
Drive_Motor_Torque string,
Drive_Motor_RPM string,
Battery_Total_volt string,
Battery_Total_Current string,
Battery_SOC string,
Battery_SOH string,
Battery_Resistance string,
Accelerator_Pedal string,
Brake_Pedal string,
Charging_State_AC string,
Charging_State_DC string,
Endurance_Mileage string,
Key_Station string,
Gear string,
Speed string,
Steering_wheel_angle string,
ODO string,
RUN_MODE string,
Air_Condition string,
Air_Power string,
Energy_Recovery string,
Wiper_State string,
Air_Set_Temp string,
Cruise_Control_Flag string,
Cruise_Control_Speed string,
Longitudinal_Acceleration string,
Transverse_Acceleration string
)
PARTITIONED BY (day string)
row format delimited fields terminated by ',';

导入数据

load data inpath "/user/admin/data/2.csv" into table ele_car_sensor_data2 partition(day='20190410');

查询试试

SELECT * FROM ele_car_sensor_data2	

简单理解下分区后的文件系统：（分区后）

下一步：写一个shell脚本按天自动上传到数仓中

3. 在线查询的impala(基于内存)

4.spark进展

1. 升级到了spark2.2(支持sparkSQL和更多的api)