HDPCD-Java-复习笔记（12）

Advanced MapReduce Features

Overview of Localization

Localization is the process of copying or downloading remote resources onto the local file-system.

It is copied to the local machine so the resource can be accessed locally.Localization is implemented using LocalResources.

A LocalResource represents a file or library required torun a container.

The NodeManager is responsible for localizing the resource prior to launching thecontainer.

Typical examples of LocalResources include:

--Libraries required for starting the container such as a jar file

--Configuration files required to configure the container once started (remote service URLs or application default configs, for example)

--A static dictionary file

Therefore if an external system has to update the remote resource it should be done via versioning. YARN will fail containers that depend on modified remote resources to prevent inconsistencies.

Each LocalResource can be of one of the following types:

FILE -- A regular file, either textual or binary.
ARCHIVE -- An archive, which is automatically unarchived by the NodeManager. As of now,NodeManager recognizes jars, tars, tar.gz files and .zip files.

PATTERN -- A hybrid of ARCHIVE and FILE types. 

Distributing Files as a LocalResource

There is a set of static methods in the  Job class for adding files and archives to the cache, including:

addCacheFile -- Adds a single file
<