Hadoop常用存储格式测试

最新推荐文章于 2024-06-03 10:52:11 发布
最新推荐文章于 2024-06-03 10:52:11 发布
阅读量386 收藏 1
点赞数
分类专栏： Hadoop
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/xiaoxiongaa0/article/details/89401879
版权 
hive> set hive.default.fileformat;
hive.default.fileformat=TextFile

先关闭压缩 

hive> SET hive.exec.compress.output
    > ;
hive.exec.compress.output=true

hive>
    > set hive.exec.compress.output=fasle;

hive> set hive.exec.compress.output;
hive.exec.compress.output=fasle

SEQUENCEFILE 

hive> create table page_views_seq
    > stored as SEQUENCEFILE
    > as select * from page_views;
Query ID = hadoop_20190419143939_53d67293-92de-4697-a791-f9a1afe7be01
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1555643336639_0005, Tracking URL = http://hadoop004:8088/proxy/application_1555643336639_0005/
Kill Command = /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/bin/hadoop job  -kill job_1555643336639_0005
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2019-04-19 15:11:26,067 Stage-1 map = 0%,  reduce = 0%
2019-04-19 15:11:33,351 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.85 sec
MapReduce Total cumulative CPU time: 2 seconds 850 msec
Ended Job = job_1555643336639_0005
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to: hdfs://hadoop004:9000/user/hive/warehouse/.hive-staging_hive_2019-04-19_15-11-19_748_1458653148881308947-1/-ext-10001
Moving data to: hdfs://hadoop004:9000/user/hive/warehouse/page_views_seq
Table default.page_views_seq stats: [numFiles=1, numRows=100000, totalSize=20501449, rawDataSize=18914993]
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1   Cumulative CPU: 2.85 sec   HDFS Read: 19018400 HDFS Write: 20501537 SUCCESS
Total MapReduce CPU Time Spent: 2 seconds 850 msec
OK
Time taken: 14.846 seconds

[hadoop@hadoop004 hadoop]$ hdfs dfs -ls /user/hive/warehouse/page_views_seq
Found 1 items
-rwxr-xr-x   1 hadoop supergroup   20501449 2019-04-19 15:11 /user/hive/warehouse/page_views_seq/000000_0

[hadoop@hadoop004 hadoop]$ hdfs dfs -du -s -h /user/hive/warehouse/page_views_seq/*
19.6 M  19.6 M  /user/hive/warehouse/page_views_seq/000000_0

RCFILE 

hive>
    > create table page_views_rcfile
    > stored as RCFILE
    > as select * from page_views;
Query ID = hadoop_20190419143939_53d67293-92de-4697-a791-f9a1afe7be01
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1555643336639_0006, Tracking URL = http://hadoop004:8088/proxy/application_1555643336639_0006/
Kill Command = /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/bin/hadoop job  -kill job_1555643336639_0006
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2019-04-19 15:14:00,203 Stage-1 map = 0%,  reduce = 0%
2019-04-19 15:14:06,565 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.68 sec
MapReduce Total cumulative CPU time: 2 seconds 680 msec
Ended Job = job_1555643336639_0006
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to: hdfs://hadoop004:9000/user/hive/warehouse/.hive-staging_hive_2019-04-19_15-13-53_788_8414345647588210415-1/-ext-10001
Moving data to: hdfs://hadoop004:9000/user/hive/warehouse/page_views_rcfile
Table default.page_views_rcfile stats: [numFiles=1, numRows=100000, totalSize=18799578, rawDataSize=18314993]
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1   Cumulative CPU: 2.68 sec   HDFS Read: 19018443 HDFS Write: 18799669 SUCCESS
Total MapReduce CPU Time Spent: 2 seconds 680 msec
OK
Time taken: 15.03 seconds

[hadoop@hadoop004 hadoop]$ hdfs dfs -ls /user/hive/warehouse/page_views_rcfile
Found 1 items
-rwxr-xr-x   1 hadoop supergroup   18799578 2019-04-19 15:14 /user/hive/warehouse/page_views_rcfile/000000_0

[hadoop@hadoop004 hadoop]$ hdfs dfs -du -s -h /user/hive/warehouse/page_views_rcfile/*
17.9 M  17.9 M  /user/hive/warehouse/page_views_rcfile/000000_0

ORC

 ORC默认的压缩格式是ZLIB

hive> set hive.exec.orc.default.compress;
hive.exec.orc.default.compress=ZLIB

 

hive>
    >
    > create table page_views_orc
    > stored as ORC
    > as select * from page_views;
Query ID = hadoop_20190419143939_53d67293-92de-4697-a791-f9a1afe7be01
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1555643336639_0007, Tracking URL = http://hadoop004:8088/proxy/application_1555643336639_0007/
Kill Command = /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/bin/hadoop job  -kill job_1555643336639_0007
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2019-04-19 15:21:21,912 Stage-1 map = 0%,  reduce = 0%
2019-04-19 15:21:29,295 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 4.27 sec
MapReduce Total cumulative CPU time: 4 seconds 270 msec
Ended Job = job_1555643336639_0007
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to: hdfs://hadoop004:9000/user/hive/warehouse/.hive-staging_hive_2019-04-19_15-21-16_165_6566607993878562298-1/-ext-10001
Moving data to: hdfs://hadoop004:9000/user/hive/warehouse/page_views_orc
Table default.page_views_orc stats: [numFiles=1, numRows=100000, totalSize=2914012, rawDataSize=76900000]
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1   Cumulative CPU: 4.27 sec   HDFS Read: 19018431 HDFS Write: 2914100 SUCCESS
Total MapReduce CPU Time Spent: 4 seconds 270 msec
OK
Time taken: 15.427 seconds

[hadoop@hadoop004 hadoop]$ hdfs dfs -ls /user/hive/warehouse/page_views_orc
Found 1 items
-rwxr-xr-x   1 hadoop supergroup    2914012 2019-04-19 15:21 /user/hive/warehouse/page_views_orc/000000_0

[hadoop@hadoop004 hadoop]$ hdfs dfs -du -s -h /user/hive/warehouse/page_views_orc/*
2.8 M  2.8 M  /user/hive/warehouse/page_views_orc/000000_0

hive> create table page_views_orc_none
    > stored as ORC tblproperties ("orc.compress"="NONE")
    > as select * from page_views;
Query ID = hadoop_20190419143939_53d67293-92de-4697-a791-f9a1afe7be01
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1555643336639_0008, Tracking URL = http://hadoop004:8088/proxy/application_1555643336639_0008/
Kill Command = /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/bin/hadoop job  -kill job_1555643336639_0008
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2019-04-19 15:27:25,281 Stage-1 map = 0%,  reduce = 0%
2019-04-19 15:27:32,558 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 4.28 sec
MapReduce Total cumulative CPU time: 4 seconds 280 msec
Ended Job = job_1555643336639_0008
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to: hdfs://hadoop004:9000/user/hive/warehouse/.hive-staging_hive_2019-04-19_15-27-19_598_2293440779180455372-1/-ext-10001
Moving data to: hdfs://hadoop004:9000/user/hive/warehouse/page_views_orc_none
Table default.page_views_orc_none stats: [numFiles=1, numRows=100000, totalSize=8101548, rawDataSize=76900000]
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1   Cumulative CPU: 4.28 sec   HDFS Read: 19018456 HDFS Write: 8101641 SUCCESS
Total MapReduce CPU Time Spent: 4 seconds 280 msec
OK
Time taken: 14.205 seconds

[hadoop@hadoop004 hadoop]$ hdfs dfs -ls /user/hive/warehouse/page_views_orc_none
Found 1 items
-rwxr-xr-x   1 hadoop supergroup    8101548 2019-04-19 15:27 /user/hive/warehouse/page_views_orc_none/000000_0

[hadoop@hadoop004 hadoop]$ hdfs dfs -du -s -h /user/hive/warehouse/page_views_orc_none/*
7.7 M  7.7 M  /user/hive/warehouse/page_views_orc_none/000000_0

hive> set parquet.compression;
parquet.compression is undefined


hive> create table page_views_parquet
    > stored as PARQUET
    > as select * from page_views;
Query ID = hadoop_20190419143939_53d67293-92de-4697-a791-f9a1afe7be01
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1555643336639_0009, Tracking URL = http://hadoop004:8088/proxy/application_1555643336639_0009/
Kill Command = /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/bin/hadoop job  -kill job_1555643336639_0009
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2019-04-19 15:37:15,140 Stage-1 map = 0%,  reduce = 0%
2019-04-19 15:37:23,556 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 5.34 sec
MapReduce Total cumulative CPU time: 5 seconds 340 msec
Ended Job = job_1555643336639_0009
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to: hdfs://hadoop004:9000/user/hive/warehouse/.hive-staging_hive_2019-04-19_15-37-08_429_4827863228315749856-1/-ext-10001
Moving data to: hdfs://hadoop004:9000/user/hive/warehouse/page_views_parquet
Table default.page_views_parquet stats: [numFiles=1, numRows=100000, totalSize=4050771, rawDataSize=700000]
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1   Cumulative CPU: 5.34 sec   HDFS Read: 19018481 HDFS Write: 4050861 SUCCESS
Total MapReduce CPU Time Spent: 5 seconds 340 msec
OK
Time taken: 16.359 seconds

[hadoop@hadoop004 hadoop]$ hdfs dfs -ls /user/hive/warehouse/page_views_parquet
Found 1 items
-rwxr-xr-x   1 hadoop supergroup    4050771 2019-04-19 15:37 /user/hive/warehouse/page_views_parquet/000000_0

[hadoop@hadoop004 hadoop]$ hdfs dfs -du -s -h /user/hive/warehouse/page_views_parquet/*
3.9 M  3.9 M  /user/hive/warehouse/page_views_parquet/000000_0

hive> set parquet.compression=gzip;
hive> set parquet.compression
    > ;
parquet.compression=gzip

hive> create table page_views_parquet_gzip
    > stored as PARQUET
    > as select * from page_views;
Query ID = hadoop_20190419143939_53d67293-92de-4697-a791-f9a1afe7be01
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1555643336639_0010, Tracking URL = http://hadoop004:8088/proxy/application_1555643336639_0010/
Kill Command = /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/bin/hadoop job  -kill job_1555643336639_0010
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2019-04-19 15:41:16,722 Stage-1 map = 0%,  reduce = 0%
2019-04-19 15:41:25,119 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 5.18 sec
MapReduce Total cumulative CPU time: 5 seconds 180 msec
Ended Job = job_1555643336639_0010
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to: hdfs://hadoop004:9000/user/hive/warehouse/.hive-staging_hive_2019-04-19_15-41-09_976_9091496854159646261-1/-ext-10001
Moving data to: hdfs://hadoop004:9000/user/hive/warehouse/page_views_parquet_gzip
Table default.page_views_parquet_gzip stats: [numFiles=1, numRows=100000, totalSize=4050771, rawDataSize=700000]
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1   Cumulative CPU: 5.18 sec   HDFS Read: 19018486 HDFS Write: 4050866 SUCCESS
Total MapReduce CPU Time Spent: 5 seconds 180 msec
OK
Time taken: 16.374 seconds

[hadoop@hadoop004 hadoop]$ hdfs dfs -ls /user/hive/warehouse/page_views_parquet_gzip
Found 1 items
-rwxr-xr-x   1 hadoop supergroup    4050771 2019-04-19 15:41 /user/hive/warehouse/page_views_parquet_gzip/000000_0

[hadoop@hadoop004 hadoop]$ hdfs dfs -du -s -h /user/hive/warehouse/page_views_parquet_gzip/*
3.9 M  3.9 M  /user/hive/warehouse/page_views_parquet_gzip/000000_0
hive> set parquet.compression=bzip2;
hive> set parquet.compression
    > ;

hive> create table page_views_parquet_bzip2
    > stored as PARQUET
    > as select * from page_views;
Query ID = hadoop_20190419143939_53d67293-92de-4697-a791-f9a1afe7be01
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1555643336639_0012, Tracking URL = http://hadoop004:8088/proxy/application_1555643336639_0012/
Kill Command = /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/bin/hadoop job  -kill job_1555643336639_0012
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2019-04-19 15:44:59,446 Stage-1 map = 0%,  reduce = 0%
2019-04-19 15:45:20,283 Stage-1 map = 100%,  reduce = 0%
Ended Job = job_1555643336639_0012 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1555643336639_0012_m_000000 (and more) from job job_1555643336639_0012

Task with the most failures(4):
-----
Task ID:
  task_1555643336639_0012_m_000000

URL:
  http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1555643336639_0012&tipid=task_1555643336639_0012_m_000000
-----
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"track_times":"2013-05-19 13:00:00","url":"http://www.taobao.com/17/?tracker_u=1624169&type=1","session_id":"B58W48U4WKZCJ5D1T3Z9ZY88RU7QA7B1","referer":"http://hao.360.cn/","ip":"1.196.34.243","end_user_id":"NULL","city_id":"-1"}
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"track_times":"2013-05-19 13:00:00","url":"http://www.taobao.com/17/?tracker_u=1624169&type=1","session_id":"B58W48U4WKZCJ5D1T3Z9ZY88RU7QA7B1","referer":"http://hao.360.cn/","ip":"1.196.34.243","end_user_id":"NULL","city_id":"-1"}
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
	... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: No enum constant parquet.hadoop.metadata.CompressionCodecName.BZIP2
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:525)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:623)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
	at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
	at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
	at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
	... 9 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: No enum constant parquet.hadoop.metadata.CompressionCodecName.BZIP2
	at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:248)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:570)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:514)
	... 16 more
Caused by: java.lang.IllegalArgumentException: No enum constant parquet.hadoop.metadata.CompressionCodecName.BZIP2
	at java.lang.Enum.valueOf(Enum.java:236)
	at parquet.hadoop.metadata.CompressionCodecName.valueOf(CompressionCodecName.java:24)
	at parquet.hadoop.metadata.CompressionCodecName.fromConf(CompressionCodecName.java:34)
	at parquet.hadoop.codec.CodecConfig.getParquetCompressionCodec(CodecConfig.java:81)
	at parquet.hadoop.codec.CodecConfig.getCodec(CodecConfig.java:88)
	at parquet.hadoop.ParquetOutputFormat.getCodec(ParquetOutputFormat.java:233)
	at parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:287)
	at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.<init>(ParquetRecordWriterWrapper.java:65)
	at org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat.getParquerRecordWriterWrapper(MapredParquetOutputFormat.java:125)
	at org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat.getHiveRecordWriter(MapredParquetOutputFormat.java:114)
	at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:260)
	at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:245)
	... 18 more


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

 

AaronLwx
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
spark写hive分区表，文件move失败
stonehigher125的专栏
04-23 849
1.写分区文件失败 错误日志 出现org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: Load Data failed for hdfs://***:8020/warehouse/tablespace/managed/hive/***/.hive-staging_hive_ 20
hadoop存储测试
数据技术控
11-26 1041
目标 测试hadoop常见的几种存储格式textfile、orcfile、parquet、sequencefile占用存储空间以及在hive、impala、presto的查询效率 测试结果 存储格式 未压缩 压缩后(snappy) impala presto hive textfile 35G 10.1G 5s 5s 47s orc
Hadoop压缩存储测试
何以问_的博客
04-25 235
Hadoop压缩存储测试执行 1.放大测试数据 放大之后的大小 : 6.26 内容 : 重复的测序数据 上传到服务器 文件名 : test1_R1.fq 2.部署Hadoop环境 2.1上传Hadoop包 3.执行各个格式压缩解压 设置HDFS的副本数为1 查看可用的压缩格式 hadoop checknative 配置压缩格式之后 3.1代码实现打成Jar包上传到服务器 3.2Gzip hejiahao@MountHua:~/package$ java -jar
Hadoop文件存储格式
最新发布
06-03 311
相比 TEXTFILE 和 SEQUENCEFILE，RCFILE 由于列式存储方式，数据加载时性能消耗较大，但是具有较好的压缩比和查询响应。数据仓库的特点是一次写入、多次读取，因此，整体来看，RCFILE 相比其余两种格式具有较明显的优势。
Hadoop之存储格式
Nice的博客
05-20 675
Hive中不同存储格式的使用 Hive Wiki 中 https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL 在定义SQL语句之后，可以通过 [STORED AS file_format] 进行制定存储格式 file_format: : SEQUENCEFILE | TEXTFILE -- (Defau...
【踩坑实录】-hive参数产生2个文件问题
chimchim66的博客
05-09 676
背景： 执行任务完成后，经常会发现hdfs生成了多个文件 设置以下参数防止hive生成2个文件： set hive.exec.stagingdir=/tmp/hive-staging/.hive-staging; 处理方法：在hive-site.xml中添加以下参数。 <property> <name>hive.exec.stagingdir</name> <value>/tm...
hadoop-testing:说明 Hadoop 中的测试示例
07-05
`hadoop-testing`通常指的是一个包含Hadoop项目中各种测试用例、测试框架和工具的模块，它可以帮助开发者有效地测试分布式环境中的数据处理和存储功能。在这个项目中，我们可能会看到`hadoop-testing-master`这个...
hadoop2.8.3
10-12
4. **格式化NameNode**：在启动Hadoop集群前，需要对NameNode进行格式化，这会初始化HDFS元数据并清除任何现有的HDFS状态。 5. **启动和关闭Hadoop服务**：通过执行相应的命令，如`start-dfs.sh`和`start-yarn.sh`...
Hadoop期末学习重点.pdf
05-30
- **步骤**：安装前准备、使用root账户登录、修改IP、修改host主机名、配置ssh免密登录、关闭防火墙、安装JDK并配置环境变量、解压Hadoop安装包、配置Hadoop的核心文件、配置Hadoop环境变量、格式化Hadoop namenode...
实验1 安装Hadoop.doc
09-24
- 格式化HDFS文件系统：`hadoop namenode -format` - 启动Hadoop服务：`start-dfs.sh`和`start-yarn.sh` 3. **验证Hadoop安装**： - 使用`jps`命令检查进程是否正常启动。 - 测试写入数据：`hadoop fs -put ...
Hadoop文件的存储格式实例详解
10-16
主要介绍了Hadoop文件的存储格式，本文给大家介绍的非常详细，具有一定的参考借鉴价值,需要的朋友可以参考下
Hive 静态分区&动态分区（含常见错误）
wzj_wp的博客
12-25 1744
静态分区 1.创建分区表 hive (wzj)> CREATE TABLE emp_partition( > empno int, > ename string, > job string, > mgr int, > hiredate string, ...
Hadoop基础教程-第11章 Hive：SQL on Hadoop（11.3 Hive 快速入门）
程裕强的专栏
08-06 1864
第11章 Hive：SQL on Hadoop11.3 Hive数据类型与应用11.3.1 基本数据类型Hive 支持关系型数据中大多数基本数据类型， 类型 描述 示例 BOOLEAN true/false TRUE TINYINT 1字节的有符号整数 -128~127 1Y SMALLINT 2个字节的有符号整数，-32768~32767 1S
Hadoop文件存储格式（Avro、Parquet、ORC及其他）
JacksonKing的专栏
07-30 3746
相比TEXTFILE和SEQUENCEFILE，RCFILE由于列式存储方式，数据加载时性能消耗较大，但是具有较好的压缩比和查询响应。数据仓库的特点是一次写入、多次读取，因此，整体来看，
Hadoop生态圈（四）- HDFS数据存储与数据管理
程序园@大Null
01-13 5687
Hadoop常用文件存储格式，Hadoop支持的压缩算法，HDFS压缩如何抉择，压缩算法比较，HDFS存储类型和存储策略，冷热温三阶段数据存储，HDFS中的内存存储支持，HDFS存储策略命令，HDFS使用懒持久化存储策略，BigData File Viewer介绍、安装及功能，Hadoop丰富的存储格式，Text File，Sequence File，Avro File，RCFile，ORC File，Parquet File，ProtoBuf和Thrift，Arrow是如何提升数据移动性能的
Hadoop文件的存储格式
Zmj的博客
04-01 5394
sequence文件存储格式txt纯文本格式，若干行记录。默认用字符编码存储SequenceFile格式（顺序文件格式，可进行切割）key-value 格式进行存储，最终形成的是一个二进制文件， 需用hadoop提供的api进行写入存储。编写 写入 seq文件案例。 Configuration configuration = new Configuration(); configu...
通用数据存储格式: Hadoop SequenceFile、HFile
weixin_33860147的博客
11-02 214
Hadoop SequenceFile 详见：http://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/io/SequenceFile.html 3个概念：记录(Record)、块(Block)、文件(File)。 记录（Record）: Hadoop SequenceFile的存储格式是通用的KV数据存储格式，key和valu...
Apache Hadoop 实战指南
接着，逐步介绍了下载Hadoop、更新bash配置、配置Hadoop（如修改`core-site.xml`, `hdfs-site.xml`, `mapred-site.xml`, `yarn-site.xml`等）、格式化HDFS文件系统、启动Hadoop服务、测试MapReduce作业以及停止...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值