NM初始化过程中涉及到的子service列表
- DeletionService 文件清理服务
- NodeHealthCheckerService 节点健康检查服务
- NodeResourceMonitor 节点资源监控服务
- ContainerManager 容器管理服务
–ResourceLocalizationService 资源本地服务
–ContainersLauncher container启动服务
–ContianerMonitor container监控服务
–AuxServices NM附属服务(如shuffle服务)
–LogHandler(NonAggregatingLogHandler | LogAggregationService) NM本地log处理服务
–SharedCacheUploadService - WebServer web服务
- AsyncDispatcher 事件分发器
- NodeStatusUpdater 节点状态更新服务
- NMStateStoreService NM状态持久化服务
子service初始化分析
DeletionService
作用:删除本地文件
@Override
protected void serviceInit(Configuration conf) throws Exception {
//构建线程工厂
ThreadFactory tf = new ThreadFactoryBuilder()
.setNameFormat("DeletionService #%d")
.build();
if (conf != null) {
//创建线程池,默认4个线程
sched = new DelServiceSchedThreadPoolExecutor(
conf.getInt(YarnConfiguration.NM_DELETE_THREAD_COUNT,
YarnConfiguration.DEFAULT_NM_DELETE_THREAD_COUNT), tf);
debugDelay = conf.getInt(YarnConfiguration.DEBUG_NM_DELETE_DELAY_SEC, 0);
} else {
sched = new DelServiceSchedThreadPoolExecutor(
YarnConfiguration.DEFAULT_NM_DELETE_THREAD_COUNT, tf);
}
sched.setExecuteExistingDelayedTasksAfterShutdownPolicy(false);
sched.setKeepAliveTime(60L, SECONDS);
if (stateStore.canRecover()) {
recover(stateStore.loadDeletionServiceState());
}
super.serviceInit(conf);
}
相关参数
yarn.nodemanager.delete.thread-count 默认4
yarn.nodemanager.delete.debug-delay-sec 默认0
NodeHealthCheckerService
作用:检查节点健康状况并汇报
包括两个子service:
(1)NodeHealthScriptRunner–使用配置的脚本进行健康检查
(2)LocalDirsHandlerService–检查磁盘健康状况
包括一个方法getHealthReport():用来获取(1)(2)两service的健康检查结果
@Override
protected void serviceInit(Configuration conf) throws Exception {
//判读是否有配置健康检查脚本,有的话将脚本检查服务添加到NM的服务列表中
if (NodeHealthScriptRunner.shouldRun(conf)) {
nodeHealthScriptRunner = new NodeHealthScriptRunner();
addService(nodeHealthScriptRunner);
}
addService(dirsHandler);
super.serviceInit(conf);
}
NodeHealthScriptRunner
作用:使用配置的脚本来检查节点的健康状况并汇报结果给汇报者。
实现:使用Timer定时的去执行健康检查脚本,健康检查脚本封装为一个TimerTask,timer支持超时检查功能,如果超过设定的时间脚本没有执行完,则timer会kill脚本并返回结果为timeout。
serviceInit分析
/*
* Method which initializes the values for the script path and interval time.
*/
@Override
protected void serviceInit(Configuration conf) throws Exception {
this.conf = conf;
this.nodeHealthScript =
conf.get(YarnConfiguration.NM_HEALTH_CHECK_SCRIPT_PATH);
this.intervalTime = conf.getLong(YarnConfiguration.NM_HEALTH_CHECK_INTERVAL_MS,
YarnConfiguration.DEFAULT_NM_HEALTH_CHECK_INTERVAL_MS);
this.scriptTimeout = conf.getLong(
YarnConfiguration.NM_HEALTH_CHECK_SCRIPT_TIMEOUT_MS,
YarnConfiguration.DEFAULT_NM_HEALTH_CHECK_SCRIPT_TIMEOUT_MS);
String[] args = conf.getStrings(YarnConfiguration.NM_HEALTH_CHECK_SCRIPT_OPTS,
new String[] {});
//NodeHealthMonitorExecutor是TimerTask的继承类,即创建了一个TimerTask
timer = new NodeHealthMonitorExecutor(args);
super.serviceInit(conf);
}
相关参数
yarn.nodemanager.health-checker.script.path 脚本路径
yarn.nodemanager.health-checker.interval-ms 健康检查脚本执行间隔时间,默认10分钟
yarn.nodemanager.health-checker.script.timeout-ms 监控检查脚本执行超时时间,默认2*yarn.nodemanager.health-checker.interval-ms
yarn.nodemanager.health-checker.script.opts 给脚本传递的参数
LocalDirsHandlerService
作用:检查节点的本地目录健康状况,周期性的检查nodemanager-local-dirs和nodemanager-log-dirs两种配置目录。
实现:使用Timer定期去检查两个目录的使用率和是否为只读模式了
/**
* Method which initializes the timertask and its interval time.
*
*/
@Override
protected void serviceInit(Configuration config) throws Exception {
// Clone the configuration as we may do modifications to dirs-list
Configuration conf = new Configuration(config);
diskHealthCheckInterval = conf.getLong(
YarnConfiguration.NM_DISK_HEALTH_CHECK_INTERVAL_MS,
YarnConfiguration.DEFAULT_NM_DISK_HEALTH_CHECK_INTERVAL_MS);
//真正的健康检查TimerTask实例
monitoringTimerTask = new MonitoringTimerTask(conf);
isDiskHealthCheckerEnabled = conf.getBoolean(
YarnConfiguration.NM_DISK_HEALTH_CHECK_ENABLE, true);
minNeededHealthyDisksFactor = conf.getFloat(
YarnConfiguration.NM_MIN_HEALTHY_DISKS_FRACTION,
YarnConfiguration.DEFAULT_NM_MIN_HEALTHY_DISKS_FRACTION);
lastDisksCheckTime = System.currentTimeMillis();
super.serviceInit(conf);
//创建本地文件上下文实例
FileContext localFs;
try {
localFs = FileContext.getLocalFSFileContext(config);
} catch (IOException e) {
throw new YarnRuntimeException("Unable to get the local filesystem", e);
}
//创建出数据目录和log目录
FsPermission perm = new FsPermission((short)0755);
boolean createSucceeded = localDirs.createNonExistentDirs(localFs, perm);
createSucceeded &= logDirs.createNonExistentDirs(localFs, perm);
if (!createSucceeded) {
updateDirsAfterTest();
}
// Check the disk health immediately to weed out bad directories
// before other init code attempts to use them.
//执行一次目录健康检查
checkDirs();
}
MonitoringTimerTask
public MonitoringTimerTask(Configuration conf) throws YarnRuntimeException {
float maxUsableSpacePercentagePerDisk =
conf.getFloat(
YarnConfiguration.NM_MAX_PER_DISK_UTILIZATION_PERCENTAGE,
YarnConfiguration.DEFAULT_NM_MAX_PER_DISK_UTILIZATION_PERCENTAGE);
long minFreeSpacePerDiskMB =
conf.getLong(YarnConfiguration.NM_MIN_PER_DISK_FREE_SPACE_MB,
YarnConfiguration.DEFAULT_NM_MIN_PER_DISK_FREE_SPACE_MB);
localDirs =
new DirectoryCollection(
validatePaths(conf
.getTrimmedStrings(YarnConfiguration.NM_LOCAL_DIRS)),
maxUsableSpacePercentagePerDisk, minFreeSpacePerDiskMB);
logDirs =
new DirectoryCollection(
validatePaths(conf.getTrimmedStrings(YarnConfiguration.NM_LOG_DIRS)),
maxUsableSpacePercentagePerDisk, minFreeSpacePerDiskMB);
localDirsAllocator = new LocalDirAllocator(
YarnConfiguration.NM_LOCAL_DIRS);
logDirsAllocator = new LocalDirAllocator(YarnConfiguration.NM_LOG_DIRS);
}
相关参数:
yarn.nodemanager.disk-health-checker.interval-ms 磁盘检查间隔时间,默认:2 * 60 * 1000
yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage 最大磁盘使用率,默认90%
yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb 最小可用空间,默认0
yarn.nodemanager.local-dirs 本地数据目录,默认/tmp/nm-local-dir
yarn.nodemanager.log-dirs 本地log目录,默认/tmp/logs
yarn.nodemanager.disk-health-checker.enable 是否开始磁盘检查,默认true
min-healthy-disks 最小健康磁盘,最少的健康磁盘个数占比,小于这个比例的化NM就不能启动container了,默认25%
NodeResourceMonitor
NodeResourceMonitorImp是个空服务,没有任何实现
ContainerManager
作用:container管理器
子服务:
(1)资源本地服务ResourceLocalizationService
(2)container启动服务ContainersLauncher
(3)container监控服务ContianerMonitor
(4)配置服务AuxServices
(5)LogHandler(NonAggregatingLogHandler | LogAggregationService)
(6)SharedCacheUploadService
public ContainerManagerImpl(Context context, ContainerExecutor exec,
DeletionService deletionContext, NodeStatusUpdater nodeStatusUpdater,
NodeManagerMetrics metrics, ApplicationACLsManager aclsManager,
LocalDirsHandlerService dirsHandler) {
super(ContainerManagerImpl.class.getName());
this.context = context;
//引入LocalDirsHandlerService检查本地目录健康状况
this.dirsHandler = dirsHandler;
// ContainerManager level dispatcher.
//ContainerManager级别的分发器
dispatcher = new AsyncDispatcher();
//引入本地文件删除服务
this.deletionService = deletionContext;
//引入NM的metrics统计功能
this.metrics = metrics;
//创建资源本地服务
rsrcLocalizationSrvc =
createResourceLocalizationService(exec, deletionContext, context);
addService(rsrcLocalizationSrvc);
//创建container启动服务
containersLauncher = createContainersLauncher(context, exec);
addService(containersLauncher);
//引入nodeStatusUpdater服务
this.nodeStatusUpdater = nodeStatusUpdater;
this.aclsManager = aclsManager;
// Start configurable services
auxiliaryServices = new AuxServices();
auxiliaryServices.registerServiceListener(this);
addService(auxiliaryServices);
//创建containerMonitor服务
this.containersMonitor =
new ContainersMonitorImpl(exec, dispatcher, this.context);
addService(this.containersMonitor);
//dispatcher注册各个子服务
dispatcher.register(ContainerEventType.class,
new ContainerEventDispatcher());
dispatcher.register(ApplicationEventType.class,
new ApplicationEventDispatcher());
dispatcher.register(LocalizationEventType.class, rsrcLocalizationSrvc);
dispatcher.register(AuxServicesEventType.class, auxiliaryServices);
dispatcher.register(ContainersMonitorEventType.class, containersMonitor);
dispatcher.register(ContainersLauncherEventType.class, containersLauncher);
addService(dispatcher);
//创建一个读写锁 TODO:用在哪?
ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
this.readLock = lock.readLock();
this.writeLock = lock.writeLock();
}
@Override
public void serviceInit(Configuration conf) throws Exception {
//创建logHandler服务并注册到分发器
LogHandler logHandler =
createLogHandler(conf, this.context, this.deletionService);
addIfService(logHandler);
dispatcher.register(LogHandlerEventType.class, logHandler);
// add the shared cache upload service (it will do nothing if the shared
// cache is disabled)
//创建共享缓存上传服务并注册到分发器
SharedCacheUploadService sharedCacheUploader =
createSharedCacheUploaderService();
addService(sharedCacheUploader);
dispatcher.register(SharedCacheUploadEventType.class, sharedCacheUploader);
//container kill等待时间
waitForContainersOnShutdownMillis =
conf.getLong(YarnConfiguration.NM_SLEEP_DELAY_BEFORE_SIGKILL_MS,
YarnConfiguration.DEFAULT_NM_SLEEP_DELAY_BEFORE_SIGKILL_MS) +
conf.getLong(YarnConfiguration.NM_PROCESS_KILL_WAIT_MS,
YarnConfiguration.DEFAULT_NM_PROCESS_KILL_WAIT_MS) +
SHUTDOWN_CLEANUP_SLOP_MS;
super.serviceInit(conf);
recover();
}
资源本地服务ResourceLocalizationService
作用:负责本地初始化,创建相应目录
子服务:LocalizerTracker
@Override
public void serviceInit(Configuration conf) throws Exception {
this.validateConf(conf);
this.publicRsrc = new LocalResourcesTrackerImpl(null, null, dispatcher,
true, conf, stateStore);
this.recordFactory = RecordFactoryProvider.getRecordFactory(conf);
try {
lfs = getLocalFileContext(conf);
lfs.setUMask(new FsPermission((short) FsPermission.DEFAULT_UMASK));
if (!stateStore.canRecover()|| stateStore.isNewlyCreated()) {
//清理本地数据目录(即删除)
cleanUpLocalDirs(lfs, delService);
//创建本地数据目录
initializeLocalDirs(lfs);
//创建本地log目录
initializeLogDirs(lfs);
}
} catch (Exception e) {
throw new YarnRuntimeException(
"Failed to initialize LocalizationService", e);
}
//本地化cache大小 localizer.cache.target-size-mb
cacheTargetSize =
conf.getLong(YarnConfiguration.NM_LOCALIZER_CACHE_TARGET_SIZE_MB, YarnConfiguration.DEFAULT_NM_LOCALIZER_CACHE_TARGET_SIZE_MB) << 20;
//cache清除间隔 localizer.cache.cleanup.interval-ms
cacheCleanupPeriod =
conf.getLong(YarnConfiguration.NM_LOCALIZER_CACHE_CLEANUP_INTERVAL_MS, YarnConfiguration.DEFAULT_NM_LOCALIZER_CACHE_CLEANUP_INTERVAL_MS);
//NM进程绑定地址,IP:端口
localizationServerAddress = conf.getSocketAddr(
YarnConfiguration.NM_BIND_HOST,
YarnConfiguration.NM_LOCALIZER_ADDRESS,
YarnConfiguration.DEFAULT_NM_LOCALIZER_ADDRESS,
YarnConfiguration.DEFAULT_NM_LOCALIZER_PORT);
//创建LocalizerTracker服务并注册事件 TODO:作用?
localizerTracker = createLocalizerTracker(conf);
addService(localizerTracker);
dispatcher.register(LocalizerEventType.class, localizerTracker);
super.serviceInit(conf);
}
相关参数:
yarn.nodemanager.localizer.cache.target-size-mb
yarn.nodemanager.localizer.cache.cleanup.interval-ms
yarn.nodemanager.bind-host NM绑定ip
localizer.address NM地址,默认0.0.0.0:8040
container启动服务ContainersLauncher
作用:负责container启动,初始化过程比较简单
@Override
protected void serviceInit(Configuration conf) throws Exception {
try {
//TODO Is this required?
FileContext.getLocalFSFileContext(conf);
} catch (UnsupportedFileSystemException e) {
throw new YarnRuntimeException("Failed to start ContainersLauncher", e);
}
super.serviceInit(conf);
}
container监控服务ContianerMonitor
作用:container状态监控,主要是内存合理性方面的校验
public ContainersMonitorImpl(ContainerExecutor exec,
AsyncDispatcher dispatcher, Context context) {
super("containers-monitor");
this.containerExecutor = exec;
this.eventDispatcher = dispatcher;
this.context = context;
this.containersToBeAdded = new HashMap<ContainerId, ProcessTreeInfo>();
this.containersToBeRemoved = new ArrayList<ContainerId>();
//创建一个线程
this.monitoringThread = new MonitoringThread();
}
@Override
protected void serviceInit(Configuration conf) throws Exception {
//监控间隔
this.monitoringInterval =
conf.getLong(YarnConfiguration.NM_CONTAINER_MON_INTERVAL_MS,
YarnConfiguration.DEFAULT_NM_CONTAINER_MON_INTERVAL_MS);
Class<? extends ResourceCalculatorPlugin> clazz =
conf.getClass(YarnConfiguration.NM_CONTAINER_MON_RESOURCE_CALCULATOR, null,
ResourceCalculatorPlugin.class);
//实例化资源计算插件,两个实现:LinuxResourceCalculatorPlugin和WindowsResourceCalculatorPlugin
this.resourceCalculatorPlugin =
ResourceCalculatorPlugin.getResourceCalculatorPlugin(clazz, conf);
LOG.info(" Using ResourceCalculatorPlugin : "
+ this.resourceCalculatorPlugin);
//进程树类
processTreeClass = conf.getClass(YarnConfiguration.NM_CONTAINER_MON_PROCESS_TREE, null,
ResourceCalculatorProcessTree.class);
this.conf = conf;
LOG.info(" Using ResourceCalculatorProcessTree : "
+ this.processTreeClass);
//container metrics开启
this.containerMetricsEnabled =
conf.getBoolean(YarnConfiguration.NM_CONTAINER_METRICS_ENABLE,
YarnConfiguration.DEFAULT_NM_CONTAINER_METRICS_ENABLE);
this.containerMetricsPeriodMs =
conf.getLong(YarnConfiguration.NM_CONTAINER_METRICS_PERIOD_MS,
YarnConfiguration.DEFAULT_NM_CONTAINER_METRICS_PERIOD_MS);
//container的物理内存配置
long configuredPMemForContainers = conf.getLong(
YarnConfiguration.NM_PMEM_MB,
YarnConfiguration.DEFAULT_NM_PMEM_MB) * 1024 * 1024l;
//container的虚拟核数配置
long configuredVCoresForContainers = conf.getLong(
YarnConfiguration.NM_VCORES,
YarnConfiguration.DEFAULT_NM_VCORES);
// Setting these irrespective of whether checks are enabled. Required in
// the UI.
// / Physical memory configuration //
this.maxPmemAllottedForContainers = configuredPMemForContainers;
this.maxVCoresAllottedForContainers = configuredVCoresForContainers;
// / Virtual memory configuration //
float vmemRatio = conf.getFloat(YarnConfiguration.NM_VMEM_PMEM_RATIO,
YarnConfiguration.DEFAULT_NM_VMEM_PMEM_RATIO);
Preconditions.checkArgument(vmemRatio > 0.99f,
YarnConfiguration.NM_VMEM_PMEM_RATIO + " should be at least 1.0");
this.maxVmemAllottedForContainers =
(long) (vmemRatio * configuredPMemForContainers);
pmemCheckEnabled = conf.getBoolean(YarnConfiguration.NM_PMEM_CHECK_ENABLED,
YarnConfiguration.DEFAULT_NM_PMEM_CHECK_ENABLED);
vmemCheckEnabled = conf.getBoolean(YarnConfiguration.NM_VMEM_CHECK_ENABLED,
YarnConfiguration.DEFAULT_NM_VMEM_CHECK_ENABLED);
LOG.info("Physical memory check enabled: " + pmemCheckEnabled);
LOG.info("Virtual memory check enabled: " + vmemCheckEnabled);
nodeCpuPercentageForYARN =
NodeManagerHardwareUtils.getNodeCpuPercentage(conf);
if (pmemCheckEnabled) {
// Logging if actual pmem cannot be determined.
long totalPhysicalMemoryOnNM = UNKNOWN_MEMORY_LIMIT;
if (this.resourceCalculatorPlugin != null) {
//从/proc/meminfo文件中读出MemTotal值即为机器总物理内存大小
totalPhysicalMemoryOnNM = this.resourceCalculatorPlugin
.getPhysicalMemorySize();
if (totalPhysicalMemoryOnNM <= 0) {
LOG.warn("NodeManager's totalPmem could not be calculated. "
+ "Setting it to " + UNKNOWN_MEMORY_LIMIT);
totalPhysicalMemoryOnNM = UNKNOWN_MEMORY_LIMIT;
}
}
//最大使用内存超过物理内存的80%,则发出警告,因为这很有可能引起OOM
if (totalPhysicalMemoryOnNM != UNKNOWN_MEMORY_LIMIT &&
this.maxPmemAllottedForContainers > totalPhysicalMemoryOnNM * 0.80f) {
LOG.warn("NodeManager configured with "
+ TraditionalBinaryPrefix.long2String(maxPmemAllottedForContainers,
"", 1)
+ " physical memory allocated to containers, which is more than "
+ "80% of the total physical memory available ("
+ TraditionalBinaryPrefix.long2String(totalPhysicalMemoryOnNM, "",
1) + "). Thrashing might happen.");
}
}
super.serviceInit(conf);
}
相关参数:
container-monitor.interval-ms container监控间隔,默认3000ms
container-monitor.resource-calculator.class container资源计算器类,默认ResourceCalculatorPlugin
container-monitor.process-tree.class 进程数类,默认ResourceCalculatorProcessTree
container-metrics.enable 是否开启container metrics统计,默认true
container-metrics.period-ms container metrics flush周期,默认-1在完成是flush
resource.memory-mb container NM为container可分配的内存大小,默认8 * 1024MB
resource.cpu-vcores NM为container可分配的vcore数量,默认8
vmem-pmem-ratio 虚拟内存和物理内存的比率,默认2.1
pmem-check-enabled 是否开启物理内容检查,默认true
vmem-check-enabled 是否开启虚拟内存检查,默认true
resource.percentage-physical-cpu-limit 为container分配的物理cpu占比,默认100%
AuxServices
public class AuxServices extends AbstractService
implements ServiceStateChangeListener, EventHandler<AuxServicesEvent> {
static final String STATE_STORE_ROOT_NAME = "nm-aux-services";
private static final Log LOG = LogFactory.getLog(AuxServices.class);
protected final Map<String,AuxiliaryService> serviceMap;
protected final Map<String,ByteBuffer> serviceMetaData;
private final Pattern p = Pattern.compile("^[A-Za-z_]+[A-Za-z0-9_]*$");
public AuxServices() {
super(AuxServices.class.getName());
serviceMap =
Collections.synchronizedMap(new HashMap<String,AuxiliaryService>());
serviceMetaData =
Collections.synchronizedMap(new HashMap<String,ByteBuffer>());
// Obtain services from configuration in init()
}
@Override
public void serviceInit(Configuration conf) throws Exception {
final FsPermission storeDirPerms = new FsPermission((short)0700);
Path stateStoreRoot = null;
FileSystem stateStoreFs = null;
//NM是否开启container恢复,yarn.nodemanager.recovery.enabled
boolean recoveryEnabled = conf.getBoolean(
YarnConfiguration.NM_RECOVERY_ENABLED,
YarnConfiguration.DEFAULT_NM_RECOVERY_ENABLED);
//NM保存container状态的根路径,stateStoreRoot=${yarn.nodemanager.recovery.dir}/${nm-aux-services}
//NM保存container状态的文件系统,本地,stateStoreFs=localFileSystem
if (recoveryEnabled) {
stateStoreRoot = new Path(conf.get(YarnConfiguration.NM_RECOVERY_DIR),
STATE_STORE_ROOT_NAME);
stateStoreFs = FileSystem.getLocal(conf);
}
//NM上运行的附属服务 yarn.nodemanager.aux-services,需要配置成mapreduce_shuffle才能运行mr程序
Collection<String> auxNames = conf.getStringCollection(
YarnConfiguration.NM_AUX_SERVICES);
for (final String sName : auxNames) {
try {
Preconditions
.checkArgument(
validateAuxServiceName(sName),
"The ServiceName: " + sName + " set in " +
YarnConfiguration.NM_AUX_SERVICES +" is invalid." +
"The valid service name should only contain a-zA-Z0-9_ " +
"and can not start with numbers");
//获取配置的附属服务对于类 yarn.nodemanager.aux-services.%s.class,如yarn.nodemanager.aux-services.mapreduce_shuffle.class=org.apache.hadoop.mapred.ShuffleHandler
Class<? extends AuxiliaryService> sClass = conf.getClass(
String.format(YarnConfiguration.NM_AUX_SERVICE_FMT, sName), null,
AuxiliaryService.class);
if (null == sClass) {
throw new RuntimeException("No class defined for " + sName);
}
//实例化AuxiliaryService对象
AuxiliaryService s = ReflectionUtils.newInstance(sClass, conf);
// TODO better use s.getName()?
if(!sName.equals(s.getName())) {
LOG.warn("The Auxilurary Service named '"+sName+"' in the "
+"configuration is for "+sClass+" which has "
+"a name of '"+s.getName()+"'. Because these are "
+"not the same tools trying to send ServiceData and read "
+"Service Meta Data may have issues unless the refer to "
+"the name in the config.");
}
//添加到serviceMap
addService(sName, s);
//如果启动恢复container功能则创建状态存储路径(恢复功能开启后,在NM重启后会进行恢复container工作)
if (recoveryEnabled) {
Path storePath = new Path(stateStoreRoot, sName);
stateStoreFs.mkdirs(storePath, storeDirPerms);
s.setRecoveryPath(storePath);
}
//AuxiliaryService对象初始化,看看ShuffleHandler.init
s.init(conf);
} catch (RuntimeException e) {
LOG.fatal("Failed to initialize " + sName, e);
throw e;
}
}
super.serviceInit(conf);
}
ShuffleHandler的初始化
//ShuffleHandler.java
@Override
protected void serviceInit(Configuration conf) throws Exception {
//是否开启shuffle读操作系统缓存,mapreduce.shuffle.manage.os.cache,默认true
manageOsCache = conf.getBoolean(SHUFFLE_MANAGE_OS_CACHE,
DEFAULT_SHUFFLE_MANAGE_OS_CACHE);
//预先读取数据大小,mapreduce.shuffle.readahead.bytes,默认4M
readaheadLength = conf.getInt(SHUFFLE_READAHEAD_BYTES,
DEFAULT_SHUFFLE_READAHEAD_BYTES);
//响应reduce下载数据请求的最大连接数,mapreduce.shuffle.max.connections,默认0代表连接数无限制。注意:该配置是针对每个NM配置的,而非每个作业配置
maxShuffleConnections = conf.getInt(MAX_SHUFFLE_CONNECTIONS,
DEFAULT_MAX_SHUFFLE_CONNECTIONS);
//响应reduce下载数据请求的最大线程数,mapreduce.shuffle.max.threads,默认0大代表为机器cpu和数的2倍。注意:该配置是针对每个NM配置的,而非每个作业配置
int maxShuffleThreads = conf.getInt(MAX_SHUFFLE_THREADS,
DEFAULT_MAX_SHUFFLE_THREADS);
if (maxShuffleThreads == 0) {
maxShuffleThreads = 2 * Runtime.getRuntime().availableProcessors();
}
//传输数据的buffer大小,mapreduce.shuffle.transfer.buffer.size,默认128k
shuffleBufferSize = conf.getInt(SHUFFLE_BUFFER_SIZE,
DEFAULT_SHUFFLE_BUFFER_SIZE);
//mapreduce.shuffle.transferTo.allowed,默认true
shuffleTransferToAllowed = conf.getBoolean(SHUFFLE_TRANSFERTO_ALLOWED,
(Shell.WINDOWS)?WINDOWS_DEFAULT_SHUFFLE_TRANSFERTO_ALLOWED:
DEFAULT_SHUFFLE_TRANSFERTO_ALLOWED);
ThreadFactory bossFactory = new ThreadFactoryBuilder()
.setNameFormat("ShuffleHandler Netty Boss #%d")
.build();
ThreadFactory workerFactory = new ThreadFactoryBuilder()
.setNameFormat("ShuffleHandler Netty Worker #%d")
.build();
//创建出selector
selector = new NioServerSocketChannelFactory(
Executors.newCachedThreadPool(bossFactory),
Executors.newCachedThreadPool(workerFactory),
maxShuffleThreads);
super.serviceInit(new Configuration(conf));
}
以上主要是shuffle的相关参数初始化,netty服务初始化。
再来看serviceStart():配置shuffle httpserver,绑定端口并启动server来监听请求。
@Override
protected void serviceStart() throws Exception {
Configuration conf = getConfig();
userRsrc = new ConcurrentHashMap<String,String>();
secretManager = new JobTokenSecretManager();
//恢复状态
recoverState(conf);
//netty服务的httpserver
ServerBootstrap bootstrap = new ServerBootstrap(selector);
try {
//处理http请求的pipelineFactory,内部创建出Shuffle对象
pipelineFact = new HttpPipelineFactory(conf);
} catch (Exception ex) {
throw new RuntimeException(ex);
}
bootstrap.setPipelineFactory(pipelineFact);
//mapreduce.shuffle.port,默认13562
port = conf.getInt(SHUFFLE_PORT_CONFIG_KEY, DEFAULT_SHUFFLE_PORT);
//server绑定监听端口构建出接受请求的channel
Channel ch = bootstrap.bind(new InetSocketAddress(port));
//将channel加入到channelgroup中国呢
accepted.add(ch);
port = ((InetSocketAddress)ch.getLocalAddress()).getPort();
//mapreduce.shuffle.port
conf.set(SHUFFLE_PORT_CONFIG_KEY, Integer.toString(port));
//为Shuffle对象设置端口
pipelineFact.SHUFFLE.setPort(port);
LOG.info(getName() + " listening on port " + port);
super.serviceStart();
//mapreduce.shuffle.ssl.file.buffer.size,默认60K
sslFileBufferSize = conf.getInt(SUFFLE_SSL_FILE_BUFFER_SIZE_KEY,
DEFAULT_SUFFLE_SSL_FILE_BUFFER_SIZE);
//apreduce.shuffle.connection-keep-alive.enable,默认false
connectionKeepAliveEnabled =
conf.getBoolean(SHUFFLE_CONNECTION_KEEP_ALIVE_ENABLED,
DEFAULT_SHUFFLE_CONNECTION_KEEP_ALIVE_ENABLED);
//mapreduce.shuffle.connection-keep-alive.timeout,默认5
connectionKeepAliveTimeOut =
Math.max(1, conf.getInt(SHUFFLE_CONNECTION_KEEP_ALIVE_TIME_OUT,
DEFAULT_SHUFFLE_CONNECTION_KEEP_ALIVE_TIME_OUT));
//mapreduce.shuffle.mapoutput-info.meta.cache.size,默认1000
mapOutputMetaInfoCacheSize =
Math.max(1, conf.getInt(SHUFFLE_MAPOUTPUT_META_INFO_CACHE_SIZE,
DEFAULT_SHUFFLE_MAPOUTPUT_META_INFO_CACHE_SIZE));
}
recoverState(conf)
//ShuffleHandler.java
private void recoverState(Configuration conf) throws IOException {
//recoverRoot=${yarn.nodemanager.recovery.dir}/mapreduce_shuffle
Path recoveryRoot = getRecoveryPath();
if (recoveryRoot != null) {
//创建levelDB对象
startStore(recoveryRoot);
//jobPattern=job_[0-9]+_[0-9]+
Pattern jobPattern = Pattern.compile(JobID.JOBID_REGEX);
LeveldbIterator iter = null;
try {
//为DB创建读取数据的迭代器
iter = new LeveldbIterator(stateDb);
iter.seek(bytes(JobID.JOB));
while (iter.hasNext()) {
Map.Entry<byte[],byte[]> entry = iter.next();
String key = asString(entry.getKey());
if (!jobPattern.matcher(key).matches()) {
break;
}
//解析出需要恢复的job,放入成员变量userRsrc中,userRsrc是个map
recoverJobShuffleInfo(key, entry.getValue());
}
} catch (DBException e) {
throw new IOException("Database error during recovery", e);
} finally {
if (iter != null) {
iter.close();
}
}
}
}
startStore(recoveryRoot)
private void startStore(Path recoveryRoot) throws IOException {
Options options = new Options();
options.createIfMissing(false);
options.logger(new LevelDBLogger());
//dbPath=${yarn.nodemanager.recovery.dir}/mapreduce_shuffle/mapreduce_shuffle_state
Path dbPath = new Path(recoveryRoot, STATE_DB_NAME);
LOG.info("Using state database at " + dbPath + " for recovery");
File dbfile = new File(dbPath.toString());
try {
//创建出DB实例
stateDb = JniDBFactory.factory.open(dbfile, options);
} catch (NativeDB.DBException e) {
if (e.isNotFound() || e.getMessage().contains(" does not exist ")) {
LOG.info("Creating state database at " + dbfile);
options.createIfMissing(true);
try {
stateDb = JniDBFactory.factory.open(dbfile, options);
storeVersion();
} catch (DBException dbExc) {
throw new IOException("Unable to create state store", dbExc);
}
} else {
throw e;
}
}
checkVersion();
}
log聚合服务 LogHandler(LogAggregationService)
在ContainerManagerImpl.serviceInit中进行LogHandler的创建
//ContainerManagerImpl.java
LogHandler logHandler =
createLogHandler(conf, this.context, this.deletionService);
addIfService(logHandler);
dispatcher.register(LogHandlerEventType.class, logHandler);
//ContainerManagerImpl.java
protected LogHandler createLogHandler(Configuration conf, Context context,
DeletionService deletionService) {
//是否开启log聚合功能,yarn.log-aggregation-enable 默认false
if (conf.getBoolean(YarnConfiguration.LOG_AGGREGATION_ENABLED,
YarnConfiguration.DEFAULT_LOG_AGGREGATION_ENABLED)) {
//log聚合service
return new LogAggregationService(this.dispatcher, context,
deletionService, dirsHandler);
} else {
//非聚合log的handler
return new NonAggregatingLogHandler(this.dispatcher, deletionService,
dirsHandler,
context.getNMStateStore());
}
}
LogAggregationService的创建、初始化和启动
//LogAggregationService.java
public LogAggregationService(Dispatcher dispatcher, Context context,
DeletionService deletionService, LocalDirsHandlerService dirsHandler) {
super(LogAggregationService.class.getName());
this.dispatcher = dispatcher;
this.context = context;
this.deletionService = deletionService;
this.dirsHandler = dirsHandler;
this.appLogAggregators =
new ConcurrentHashMap<ApplicationId, AppLogAggregator>();
//创建线程池
this.threadPool = Executors.newCachedThreadPool(
new ThreadFactoryBuilder()
.setNameFormat("LogAggregationService #%d")
.build());
}
protected void serviceInit(Configuration conf) throws Exception {
//applog存放路径,yarn.nodemanager.remote-app-log-dir,默认/tmp/logs
this.remoteRootLogDir =
new Path(conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR,
YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR));
//applog存放文件后缀名,yarn.nodemanager.remote-app-log-dir-suffix,默认logs
//log会被存放在NM_REMOTE_APP_LOG_DIR/${user}/NM_REMOTE_APP_LOG_DIR_SUFFIX/${appId}
this.remoteRootLogDirSuffix =
conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR_SUFFIX,
YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR_SUFFIX);
super.serviceInit(conf);
}
@Override
protected void serviceStart() throws Exception {
// NodeId is only available during start, the following cannot be moved
// anywhere else.
this.nodeId = this.context.getNodeId();
super.serviceStart();
}
NonAggregatingLogHandler创建、初始化、启动
//NonAggregatingLogHandler.java
public NonAggregatingLogHandler(Dispatcher dispatcher,
DeletionService delService, LocalDirsHandlerService dirsHandler,
NMStateStoreService stateStore) {
super(NonAggregatingLogHandler.class.getName());
this.dispatcher = dispatcher;
this.delService = delService;
this.dirsHandler = dirsHandler;
this.stateStore = stateStore;
this.appOwners = new ConcurrentHashMap<ApplicationId, String>();
}
@Override
protected void serviceInit(Configuration conf) throws Exception {
// Default 3 hours.
//container的log保存事件,yarn.nodemanager.log.retain-seconds,默认3小时
this.deleteDelaySeconds =
conf.getLong(YarnConfiguration.NM_LOG_RETAIN_SECONDS,
YarnConfiguration.DEFAULT_NM_LOG_RETAIN_SECONDS);
//创建ScheduledThreadPoolExecutor对象
sched = createScheduledThreadPoolExecutor(conf);
super.serviceInit(conf);
//log恢复
recover();
}
进入createScheduledThreadPoolExecutor(conf)
ScheduledThreadPoolExecutor createScheduledThreadPoolExecutor(
Configuration conf) {
//log删除线程工厂对象
ThreadFactory tf =
new ThreadFactoryBuilder().setNameFormat("LogDeleter #%d").build();
//log删除线程数,yarn.nodemanager.log.deletion-threads-count,默认4
sched =
new ScheduledThreadPoolExecutor(conf.getInt(
YarnConfiguration.NM_LOG_DELETION_THREADS_COUNT,
YarnConfiguration.DEFAULT_NM_LOG_DELETE_THREAD_COUNT), tf);
return sched;
}
进入recover()
/**
* 启动线程池调度,到时间后删除每个app的log
* @throws IOException
*/
private void recover() throws IOException {
if (stateStore.canRecover()) {
//加载LevelDB存储的LogDeleters/路径中每个app的状态
RecoveredLogDeleterState state = stateStore.loadLogDeleterState();
long now = System.currentTimeMillis();
for (Map.Entry<ApplicationId, LogDeleterProto> entry :
state.getLogDeleterMap().entrySet()) {
ApplicationId appId = entry.getKey();
LogDeleterProto proto = entry.getValue();
//计算每个app的删除剩余时间
long deleteDelayMsec = proto.getDeletionTime() - now;
if (LOG.isDebugEnabled()) {
LOG.debug("Scheduling deletion of " + appId + " logs in "
+ deleteDelayMsec + " msec");
}
//为每个app创建log删除的Runner对象
LogDeleterRunnable logDeleter =
new LogDeleterRunnable(proto.getUser(), appId);
//调度线程到时间了进行app的log删除
try {
sched.schedule(logDeleter, deleteDelayMsec, TimeUnit.MILLISECONDS);
} catch (RejectedExecutionException e) {
// Handling this event in local thread before starting threads
// or after calling sched.shutdownNow().
logDeleter.run();
}
}
}
}
具体的log删除过程就要看NonAggregatingLogHandler L o g D e l e t e r R u n n a b l e . r u n ( ) 了 ( 1 ) A p p l i c a t o n I m p l 发 送 A p p l i c a t i o n E v e n t T y p e . A P P L I C A T I O N L O G H A N D L I N G F I N I S H E D 事 件 通 知 ( 2 ) 调 用 D e l e t i o n S e r v i c e 服 务 真 正 的 删 除 本 地 a p p 的 l o g ( 3 ) l o g 删 除 完 成 后 从 l e v e l d b 中 删 除 每 个 a p p i d 对 应 的 记 录 , 即 删 除 k e y = L o g D e l e t e r s / LogDeleterRunnable.run()了 (1)ApplicatonImpl发送ApplicationEventType.APPLICATION_LOG_HANDLING_FINISHED事件通知 (2)调用DeletionService服务真正的删除本地app的log (3)log删除完成后从leveldb中删除每个appid对应的记录,即删除key=LogDeleters/ LogDeleterRunnable.run()了(1)ApplicatonImpl发送ApplicationEventType.APPLICATIONLOGHANDLINGFINISHED事件通知(2)调用DeletionService服务真正的删除本地app的log(3)log删除完成后从leveldb中删除每个appid对应的记录,即删除key=LogDeleters/appID的记录
//NonAggregatingLogHandler.java$LogDeleterRunnable
@Override
@SuppressWarnings("unchecked")
/**
* 删除app的本地log
*/
public void run() {
List<Path> localAppLogDirs = new ArrayList<Path>();
//获取本地文件系统
FileContext lfs = getLocalFileContext(getConfig());
for (String rootLogDir : dirsHandler.getLogDirsForCleanup()) {
//获取每个app的本地log路径
Path logDir = new Path(rootLogDir, applicationId.toString());
try {
lfs.getFileStatus(logDir);
localAppLogDirs.add(logDir);
} catch (UnsupportedFileSystemException ue) {
LOG.warn("Unsupported file system used for log dir " + logDir, ue);
continue;
} catch (IOException ie) {
continue;
}
}
// Inform the application before the actual delete itself, so that links
// to logs will no longer be there on NM web-UI.
// 真正删除log前,先向ApplicatonImpl发送ApplicationEventType.APPLICATION_LOG_HANDLING_FINISHED事件通知
NonAggregatingLogHandler.this.dispatcher.getEventHandler().handle(
new ApplicationEvent(this.applicationId,
ApplicationEventType.APPLICATION_LOG_HANDLING_FINISHED));
// 调用DeletionService服务去真正的删除log(就是删除本地文件的过程)
if (localAppLogDirs.size() > 0) {
NonAggregatingLogHandler.this.delService.delete(user, null,
(Path[]) localAppLogDirs.toArray(new Path[localAppLogDirs.size()]));
}
//log删除完成后从leveldb中删除每个appid对应的记录
try {
NonAggregatingLogHandler.this.stateStore.removeLogDeleter(
this.applicationId);
} catch (IOException e) {
LOG.error("Error removing log deletion state", e);
}
}
最后看下ApplicationImpl收到ApplicationEventType.APPLICATION_LOG_HANDLING_FINISHED事件后的处理
// Transitions from FINISHED state
.addTransition(ApplicationState.FINISHED,
ApplicationState.FINISHED,
ApplicationEventType.APPLICATION_LOG_HANDLING_FINISHED,
new AppLogsAggregatedTransition())
进入AppLogsAggregatedTransition
(1)从NMContext的applications列表中删除本地log已清理完的app
(2)从aclsManagers中删除本地log已清理完的app
(3)删除levelDB中key=ContainerManager/applications/$appID的记录
static class AppLogsAggregatedTransition implements
SingleArcTransition<ApplicationImpl, ApplicationEvent> {
@Override
public void transition(ApplicationImpl app, ApplicationEvent event) {
ApplicationId appId = event.getApplicationID();
app.context.getApplications().remove(appId);
app.aclsManager.removeApplication(appId);
try {
app.context.getNMStateStore().removeApplication(appId);
} catch (IOException e) {
LOG.error("Unable to remove application from state store", e);
}
}
}
WebServer
作用:NM webserver
public WebServer(Context nmContext, ResourceView resourceView,
ApplicationACLsManager aclsManager,
LocalDirsHandlerService dirsHandler) {
super(WebServer.class.getName());
this.nmContext = nmContext;
this.nmWebApp = new NMWebApp(resourceView, aclsManager, dirsHandler);
}
该service无serviceInit方法,初始化工作和启动都放在了serviceStart中,后续分析。
AsyncDispatcher
NodeStatusUpdater
作用:
public NodeStatusUpdaterImpl(Context context, Dispatcher dispatcher,
NodeHealthCheckerService healthChecker, NodeManagerMetrics metrics) {
super(NodeStatusUpdaterImpl.class.getName());
this.healthChecker = healthChecker;
this.context = context;
this.dispatcher = dispatcher;
this.metrics = metrics;
this.recentlyStoppedContainers =
new LinkedHashMap<ContainerId, Long>();
this.pendingCompletedContainers =
new HashMap<ContainerId, ContainerStatus>();
}
@Override
protected void serviceInit(Configuration conf) throws Exception {
//获得内存和虚拟CPU核心相关配置
int memoryMb =
conf.getInt(
YarnConfiguration.NM_PMEM_MB, YarnConfiguration.DEFAULT_NM_PMEM_MB);
float vMemToPMem =
conf.getFloat(
YarnConfiguration.NM_VMEM_PMEM_RATIO,
YarnConfiguration.DEFAULT_NM_VMEM_PMEM_RATIO);
int virtualMemoryMb = (int)Math.ceil(memoryMb * vMemToPMem);
int virtualCores =
conf.getInt(
YarnConfiguration.NM_VCORES, YarnConfiguration.DEFAULT_NM_VCORES);
//创建Resource对象,记录上述资源配置
this.totalResource = Resource.newInstance(memoryMb, virtualCores);
//将资源记录至监控系统
metrics.addResource(totalResource);
this.tokenKeepAliveEnabled = isTokenKeepAliveEnabled(conf);
this.tokenRemovalDelayMs =
conf.getInt(YarnConfiguration.RM_NM_EXPIRY_INTERVAL_MS,
YarnConfiguration.DEFAULT_RM_NM_EXPIRY_INTERVAL_MS);
this.minimumResourceManagerVersion = conf.get(
YarnConfiguration.NM_RESOURCEMANAGER_MINIMUM_VERSION,
YarnConfiguration.DEFAULT_NM_RESOURCEMANAGER_MINIMUM_VERSION);
// Default duration to track stopped containers on nodemanager is 10Min.
// This should not be assigned very large value as it will remember all the
// containers stopped during that time.
//容器执行完毕的检测时间间隔,默认10分钟
durationToTrackStoppedContainers =
conf.getLong(YARN_NODEMANAGER_DURATION_TO_TRACK_STOPPED_CONTAINERS,
600000);
if (durationToTrackStoppedContainers < 0) {
String message = "Invalid configuration for "
+ YARN_NODEMANAGER_DURATION_TO_TRACK_STOPPED_CONTAINERS + " default "
+ "value is 10Min(600000).";
LOG.error(message);
throw new YarnException(message);
}
if (LOG.isDebugEnabled()) {
LOG.debug(YARN_NODEMANAGER_DURATION_TO_TRACK_STOPPED_CONTAINERS + " :"
+ durationToTrackStoppedContainers);
}
super.serviceInit(conf);
LOG.info("Initialized nodemanager for " + nodeId + ":" +
" physical-memory=" + memoryMb + " virtual-memory=" + virtualMemoryMb +
" virtual-cores=" + virtualCores);
}
相关参数:
duration-to-track-stopped-containers container执行完毕的检测时间间隔,默认10分钟
NMStateStoreService
NodeManager在serviceInit方法中进行NMStateStoreService的初始化
//NodeManager.java
initAndStartRecoveryStore(conf);
//NodeManager.java
private void initAndStartRecoveryStore(Configuration conf)
throws IOException {
//是否开启app恢复,yarn.nodemanager.recovery.enabled 默认false
boolean recoveryEnabled = conf.getBoolean(
YarnConfiguration.NM_RECOVERY_ENABLED,
YarnConfiguration.DEFAULT_NM_RECOVERY_ENABLED);
if (recoveryEnabled) {
//获取本地文件系统,file:///
FileSystem recoveryFs = FileSystem.getLocal(conf);
//恢复目录,yarn.nodemanager.recovery.dir
String recoveryDirName = conf.get(YarnConfiguration.NM_RECOVERY_DIR);
if (recoveryDirName == null) {
throw new IllegalArgumentException("Recovery is enabled but " +
YarnConfiguration.NM_RECOVERY_DIR + " is not set.");
}
Path recoveryRoot = new Path(recoveryDirName);
//创建出恢复目录
recoveryFs.mkdirs(recoveryRoot, new FsPermission((short)0700));
//创建LevelDB存储服务
nmStore = new NMLeveldbStateStoreService();
} else {
nmStore = new NMNullStateStoreService();
}
nmStore.init(conf);
nmStore.start();
}
备注:NM的app状态存储服务NMStateStoreService有三种实现:NMLeveldbStateStoreService、NMNullStateStoreService、NMMemoryStateStoreService,实际使用的只有NMLeveldbStateStoreService,其他两个只是用来支持单测的。
nmStore.init(conf);会调到父类的serviceInit
//NMStateStoreService.java
/** Initialize the state storage */
@Override
public void serviceInit(Configuration conf) throws IOException {
initStorage(conf);
}
再调到子类
//NMLeveldbStateStoreService.java
protected void initStorage(Configuration conf)
throws IOException {
//创建出${yarn.nodemanager.recovery.dir}/yarn-nm-state目录,其中yarn-nm-state为DB_NAME
Path storeRoot = createStorageDir(conf);
Options options = new Options();
options.createIfMissing(false);
options.logger(new LeveldbLogger());
LOG.info("Using state database at " + storeRoot + " for recovery");
File dbfile = new File(storeRoot.toString());
try {
//通过leveldbjni三方库创建出DB对象
db = JniDBFactory.factory.open(dbfile, options);
} catch (NativeDB.DBException e) {
if (e.isNotFound() || e.getMessage().contains(" does not exist ")) {
LOG.info("Creating state database at " + dbfile);
isNewlyCreated = true;
options.createIfMissing(true);
try {
db = JniDBFactory.factory.open(dbfile, options);
// store version
storeVersion();
} catch (DBException dbErr) {
throw new IOException(dbErr.getMessage(), dbErr);
}
} else {
throw e;
}
}
checkVersion();
}
接下来看nmStore.start();
调到父类的serviceStart
/** Start the state storage for use */
@Override
public void serviceStart() throws IOException {
startStorage();
}
再调到子类
@Override
protected void startStorage() throws IOException {
}
发现是个空方法,这其实是对leveldb的功能的一个展现:leveldb其实不是一个数据库服务,而仅仅是一个kv存储系统,在使用leveldb的时候外部程序继承进它的库就可以使用它的存储功能了。