6、移动命令
#hadoop dfs –mv /user/test.txt /user/test/
7、拷贝命令
#hadoop dfs –copytolocal /user/test.txt /opt/
查看spark上面的运行情况;
http://192.168.31.10:8080/#
wc -l judgedoc_litigant.txt 查看文件有多少行;
htop查看当前spark占用的cpu,退出按q
命令:
vim ~/.bashrc
source ~/.bashrc
ps aux | grep spark
pkill -f "spark"
sudo chown -R sc:sc spark-2.3.1-bin-hadoop2.7/
sudo mv /home/sc/Downloads/spark-2.3.1-bin-hadoop2.7 /opt/
$SPARK_HOME 查看spark的路径
network_all_link_cnt.csv
network_all_link_judgedoc_cnt.csv *
network_share_com_inv_com_judgedoc_cnt.csv
network_c_inv_c_revoke_cnt.csv
network_c_inv_c_shixin_cnt.csv
network_c_inv_c_zhixing_cnt.csv
network_share_established_years_mean.csv
network_share_judgedoc_cnt.csv *
network_share_registered_capital_mean.csv
network_share_revoke_cnt.csv
network_share_shixin_cnt.csv
network_share_zhixing_cnt.csv
/home/sc/PycharmProjects/risk-model/scpirt/test_build_data_scripts/shixin_cox/shixin_cox_get_features/shixin_cox_biz/sub/shixin_the_last_evevt_time_merge_shixin_time_and_shixin_cnt.py
http://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-2.3.1/spark-2.3.1-bin-hadoop2.7.tgz
drwxr-xr-x - scdata supergroup 0 2018-08-08 14:16 /hdfs/riskModelAuto/2018-07-24/feature/feature_extract_bidding.csv
drwxr-xr-x - scdata supergroup 0 2018-08-08 14:15 /hdfs/riskModelAuto/2018-07-24/feature/feature_extract_courtnotice.csv
drwxr-xr-x - scdata supergroup 0 2018-08-09 16:14 /hdfs/riskModelAuto/2018-07-24/feature/feature_extract_judgedoc.csv
drwxr-xr-x - scdata supergroup 0 2018-08-08 14:16 /hdfs/riskModelAuto/2018-07-24/feature/feature_extract_shixin.csv
drwxr-xr-x - scdata supergroup 0 2018-08-08 14:15 /hdfs/riskModelAuto/2018-07-24/feature/feature_extract_trademark.csv
drwxr-xr-x - scdata supergroup 0 2018-08-09 10:39 /hdfs/riskModelAuto/2018-07-24/feature/feature_extract_sue_by_rank_judgedoc.csv
drwxr-xr-x - scdata supergroup 0 2018-08-06 23:04 /hdfs/riskModelAuto/2018-07-24/feature/network_all_link_cnt.csv
drwxr-xr-x - scdata supergroup 0 2018-08-09 11:14 /hdfs/riskModelAuto/2018-07-24/feature/network_c_inv_c_revoke_cnt.csv
drwxr-xr-x - scdata supergroup 0 2018-08-09 11:29 /hdfs/riskModelAuto/2018-07-24/feature/network_c_inv_c_shixin_cnt.csv
drwxr-xr-x - scdata supergroup 0 2018-08-09 11:25 /hdfs/riskModelAuto/2018-07-24/feature/network_c_inv_c_zhixing_cnt.csv
drwxr-xr-x - scdata supergroup 0 2018-08-09 15:24 /hdfs/riskModelAuto/2018-07-24/feature/network_share_com_inv_com_judgedoc_cnt.csv
drwxr-xr-x - scdata supergroup 0 2018-08-09 14:08 /hdfs/riskModelAuto/2018-07-24/feature/network_share_registered_capital_mean.csv
drwxr-xr-x - scdata supergroup 0 2018-08-09 14:31 /hdfs/riskModelAuto/2018-07-24/feature/network_share_revoke_cnt.csv
drwxr-xr-x - scdata supergroup 0 2018-08-08 19:07 /hdfs/riskModelAuto/2018-07-24/feature/network_share_shixin_cnt.csv
drwxr-xr-x - scdata supergroup 0 2018-08-07 12:32 /hdfs/riskModelAuto/2018-07-24/feature/network_share_zhixing_cnt.csv
查看spark进程运行状态以及安装spark
最新推荐文章于 2023-05-31 08:27:02 发布