hadoop 文件存放策略_Hdfs磁盘存储策略和预留空间配置

一、Hdfs磁盘存储策略

1、 指定本地目录存储策略

data目录为Hot策略对应DISK；

data1目录为Cold策略对应ARCHIVE；

dfs.datanode.data.dir

[DISK]/opt/beh/data/namenode/dfs/data,[ARCHIVE]/opt/beh/data/namenode/dfs/data1

重启hdfs

$ stop-dfs.sh

$ start-dfs.sh

2、指定hdfs目录的存储策略

查看hdfs存储策略

$ hdfs storagepolicies -listPolicies

Block Storage Policies:

BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]}

BlockStoragePolicy{WARM:5, storageTypes=[DISK, ARCHIVE], creationFallbacks=[DISK, ARCHIVE], replicationFallbacks=[DISK, ARCHIVE]}

BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}

BlockStoragePolicy{ONE_SSD:10, storageTypes=[SSD, DISK], creationFallbacks=[SSD, DISK], replicationFallbacks=[SSD, DISK]}

BlockStoragePolicy{ALL_SSD:12, storageTypes=[SSD], creationFallbacks=[DISK], replicationFallbacks=[DISK]}

BlockStoragePolicy{LAZY_PERSIST:15, storageTypes=[RAM_DISK, DISK], creationFallbacks=[DISK], replicationFallbacks=[DISK]}

创建2个hdfs目录

$ hadoop fs -mkdir /Cold_data

$ hadoop fs -mkdir /Hot_data

指定hdfs目录存储策略

$ hdfs storagepolicies -setStoragePolicy -path hdfs://breath:9000/Cold_data -policy COLD

Set storage policy COLD on hdfs://breath:9000/Cold_data

$ hdfs storagepolicies -setStoragePolicy -path hdfs://breath:9000/Hot_data -policy HOT

Set storage policy HOT on hdfs://breath:9000/Hot_data

查看2个目录的存储策略是否正确

$ hdfs storagepolicies -getStoragePolicy -path /Cold_data

The storage policy of /Cold_data:

B