Hadoop的存储策略
策略 ID | 策略名称 | 块分布 | creationFallbacks | replicationFallbacks |
---|---|---|---|---|
15 | Lazy_Persist | RAM_DISK: 1, DISK: n-1 | DISK | DISK |
12 | All_SSD | SSD: n | DISK | DISK |
10 | One_SSD | SSD: 1, DISK: n-1 | SSD, DISK | SSD, DISK |
7 | Hot (default) | DISK: n | < none > | ARCHIVE |
5 | Warm | DISK: 1, ARCHIVE: n-1 | ARCHIVE, DISK | ARCHIVE, DISK |
2 | Cold | ARCHIVE:n | < none > | < none > |
策略ID数值越大其存取速度越快
1、查看机器支持的异构存储策略
#查看机器支持的异构存储策略
hdfs storagepolicies -listPolicies
#----------------------------------------------------------------------------------------
Block Storage Policies:
BlockStoragePolicy{PROVIDED:1, storageTypes=[PROVIDED, DISK], creationFallbacks=[PROVIDED, DISK], replicationFallbacks=[PROVIDED, DISK]}
BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]}
BlockStoragePolicy{WARM:5, storageTypes=[DISK, ARCHIVE], creationFallbacks=[DISK, ARCHIVE], replicationFallbacks=[DISK, ARCHIVE]}
BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}
BlockStoragePolicy{ONE_SSD:10, storageTypes=[SSD, DISK], creationFallbacks=[SSD, DISK], replicationFallbacks=[SSD, DISK]}
BlockStoragePolicy{ALL_SSD:12, storageTypes=[SSD], creationFallbacks=[DISK], replicationFallbacks=[DISK]}
BlockStoragePolicy{LAZY_PERSIST:15, storageTypes=[RAM_DISK, DISK], creationFallbacks=[DISK], replicationFallbacks=[DISK]}
#----------------------------------------------------------------------------------------
2、COLD
COLD存储策略主要用于存储冷数据,存储类型是ARCHIVE
BlockStoragePolicy{
COLD:2,
storageTypes=[ARCHIVE],
creationFallbacks=[],
replicationFallbacks=[]
}
3、Warm
Warm存储策略主要用于存储半热半冷数据,存储类型是DISK和ARCHIVE
BlockStoragePolicy{
WARM:5,
storageTypes=[DISK, ARCHIVE],
creationFallbacks=[DISK, ARCHIVE],
replicationFallbacks=[DISK, ARCHIVE]
}
4、HOT
HOT存储策略主要用于存储热数据,存储类型是DISK
BlockStoragePolicy{
HOT:7,
storageTypes=[DISK],
creationFallbacks=[],
replicationFallbacks=[ARCHIVE]
}
5、ONE_SSD
ONE_SSD的存储策略就是将一个主数据放在SSD介质上,其他备份文件放在DISK上。这是HDFS的默认数据存储策略
BlockStoragePolicy{
ONE_SSD:10,
storageTypes=[SSD, DISK],
creationFallbacks=[SSD, DISK],
replicationFallbacks=[SSD, DISK]
}
6、ALL_SSD
这个数据的存储策略就是,将数据全部存储在SSD介质上,包括备份数据的数据,当SSD空间不足时才采取将数据放在DISK上
BlockStoragePolicy
ALL_SSD:12,
storageTypes=[SSD],
creationFallbacks=[DISK],
replicationFallbacks=[DISK]
}
7、LAZY_PERSIST
LAZY_PERSIST的存储策略就是将新建的数据块放在RAM_DISK上,而备份的数据放在DISK上。RAM_DISK为随机读写磁盘,其是内存的主要构成单元,因此LAZY_PERSIST存储策略的数据其读写速度最快。
BlockStoragePolicy{
LAZY_PERSIST:15,
storageTypes=[RAM_DISK, DISK],
creationFallbacks=[DISK],
replicationFallbacks=[DISK]
}