Hadoop的存储策略

Hadoop的存储策略

策略 ID	策略名称	块分布	creationFallbacks	replicationFallbacks
15	Lazy_Persist	RAM_DISK: 1, DISK: n-1	DISK	DISK
12	All_SSD	SSD: n	DISK	DISK
10	One_SSD	SSD: 1, DISK: n-1	SSD, DISK	SSD, DISK
7	Hot (default)	DISK: n	< none >	ARCHIVE
5	Warm	DISK: 1, ARCHIVE: n-1	ARCHIVE, DISK	ARCHIVE, DISK
2	Cold	ARCHIVE:n	< none >	< none >

策略ID数值越大其存取速度越快

1、查看机器支持的异构存储策略

#查看机器支持的异构存储策略
hdfs storagepolicies -listPolicies
#----------------------------------------------------------------------------------------
Block Storage Policies:
	BlockStoragePolicy{PROVIDED:1, storageTypes=[PROVIDED, DISK], creationFallbacks=[PROVIDED, DISK], replicationFallbacks=[PROVIDED, DISK]}
	BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]}
	BlockStoragePolicy{WARM:5, storageTypes=[DISK, ARCHIVE], creationFallbacks=[DISK, ARCHIVE], replicationFallbacks=[DISK, ARCHIVE]}
	BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}
	BlockStoragePolicy{ONE_SSD:10, storageTypes=[SSD, DISK], creationFallbacks=[SSD, DISK], replicationFallbacks=[SSD, DISK]}
	BlockStoragePolicy{ALL_SSD:12, storageTypes=[SSD], creationFallbacks=[DISK], replicationFallbacks=[DISK]}
	BlockStoragePolicy{LAZY_PERSIST:15, storageTypes=[RAM_DISK, DISK], creationFallbacks=[DISK], replicationFallbacks=[DISK]}
#----------------------------------------------------------------------------------------

2、COLD

COLD存储策略主要用于存储冷数据，存储类型是ARCHIVE

BlockStoragePolicy{
    COLD:2,
    storageTypes=[ARCHIVE],
	creationFallbacks=[],
	replicationFallbacks=[]
}

3、Warm

Warm存储策略主要用于存储半热半冷数据，存储类型是DISK和ARCHIVE

BlockStoragePolicy{
	WARM:5, 
	storageTypes=[DISK, ARCHIVE], 
	creationFallbacks=[DISK, ARCHIVE], 
	replicationFallbacks=[DISK, ARCHIVE]
}

4、HOT

HOT存储策略主要用于存储热数据，存储类型是DISK

BlockStoragePolicy{
	HOT:7, 
	storageTypes=[DISK], 
	creationFallbacks=[], 
	replicationFallbacks=[ARCHIVE]
}

5、ONE_SSD

ONE_SSD的存储策略就是将一个主数据放在SSD介质上，其他备份文件放在DISK上。这是HDFS的默认数据存储策略

BlockStoragePolicy{
	ONE_SSD:10, 
	storageTypes=[SSD, DISK], 
	creationFallbacks=[SSD, DISK], 
	replicationFallbacks=[SSD, DISK]
}

6、ALL_SSD

这个数据的存储策略就是，将数据全部存储在SSD介质上，包括备份数据的数据，当SSD空间不足时才采取将数据放在DISK上

BlockStoragePolicy
	ALL_SSD:12, 
	storageTypes=[SSD], 
	creationFallbacks=[DISK], 
	replicationFallbacks=[DISK]
}

7、LAZY_PERSIST

LAZY_PERSIST的存储策略就是将新建的数据块放在RAM_DISK上，而备份的数据放在DISK上。RAM_DISK为随机读写磁盘，其是内存的主要构成单元，因此LAZY_PERSIST存储策略的数据其读写速度最快。

BlockStoragePolicy{
	LAZY_PERSIST:15, 
	storageTypes=[RAM_DISK, DISK], 
	creationFallbacks=[DISK], 
	replicationFallbacks=[DISK]
}