[hadoop2.7.1]I/O之tfile

在这儿做个记录吧

org.apache.hadoop.io.file.tfile.TFile

@Public
@Evolving
ATFileisacontainerofkey-valuepairs.Bothkeysandvaluesaretype-lessbytes.Keysarerestrictedto64KB,valuelengthisnotrestricted(practicallylimitedtotheavailablediskstorage).TFilefurtherprovidesthefollowingfeatures:
BlockCompression.
Namedmetadatablocks.
Sortedorunsortedkeys.
Seekbykeyorbyfileoffset.
ThememoryfootprintofaTFileincludesthefollowing:
Someconstantoverheadofreadingorwritingacompressedblock.
Eachcompressedblockrequiresonecompression/decompressioncodecforI/O.
Temporaryspacetobufferthekey.
Temporaryspacetobufferthevalue(forTFile.Writeronly).Valuesarechunkencoded,sothatwebufferatmostonechunkofuserdata.Bydefault,thechunkbufferis1MB.Readingchunkedvaluedoesnotrequireadditionalmemory.
TFileindex,whichisproportionaltothetotalnumberofDataBlocks.Thetotalamountofmemoryneededtoholdtheindexcanbeestimatedas(56+AvgKeySize)*NumBlocks.MetaBlockindex,whichisproportionaltothetotalnumberofMetaBlocks.ThetotalamountofmemoryneededtoholdtheindexforMetaBlockscanbeestimatedas(40+AvgMetaBlockName)*NumMetaBlock.
ThebehaviorofTFilecanbecustomizedbythefollowingvariablesthroughConfiguration:
tfile.io.chunk.size:Valuechunksize.Integer(inbytes).Defaultto1MB.Valuesofthelengthlessthanthechunksizeisguaranteedtohaveknownvaluelengthinreadtime(SeeTFile.Reader.Scanner.Entry.isValueLengthKnown()).
tfile.fs.output.buffer.size:BuffersizeusedforFSDataOutputStream.Integer(inbytes).Defaultto256KB.
tfile.fs.input.buffer.size:BuffersizeusedforFSDataInputStream.Integer(inbytes).Defaultto256KB.
Suggestionsonperformanceoptimization.
Minimumblocksize.Werecommendasettingofminimumblocksizebetween256KBto1MBforgeneralusage.Largerblocksizeispreferrediffilesareprimarilyforsequentialaccess.However,itwouldleadtoinefficientrandomaccess(becausetherearemoredatatodecompress).Smallerblocksaregoodforrandomaccess,butrequiremorememorytoholdtheblockindex,andmaybeslowertocreate(becausewemustflushthecompressorstreamattheconclusionofeachdatablock,whichleadstoanFSI/Oflush).Further,duetotheinternalcachinginCompressioncodec,thesmallestpossibleblocksizewouldbearound20KB-30KB.
Thecurrentimplementationdoesnotoffertruemulti-threadingforreading.TheimplementationusesFSDataInputStreamseek()+read(),whichisshowntobemuchfasterthanpositioned-readcallinsinglethreadmode.However,italsomeansthatifmultiplethreadsattempttoaccessthesameTFile(usingmultiplescanners)simultaneously,theactualI/OiscarriedoutsequentiallyeveniftheyaccessdifferentDFSblocks.
Compressioncodec.Use"none"ifthedataisnotverycompressable(bycompressable,Imeanacompressionratioatleast2:1).Generally,use"lzo"asthestartingpointforexperimenting."gz"oversslightlybettercompressionratioover"lzo"butrequires4xCPUtocompressand2xCPUtodecompress,comparingto"lzo".
Filesystembuffering,iftheunderlyingFSDataInputStreamandFSDataOutputStreamisalreadyadequatelybuffered;orifapplicationsreads/writeskeysandvaluesinlargebuffers,wecanreducethesizesofinput/outputbufferinginTFilelayerbysettingtheconfigurationparameters"tfile.fs.input.buffer.size"and"tfile.fs.output.buffer.size".