we know,hadoop will show only abstract result info about mapreduce(represent in http://xxx:19888/clsuter by default),so it's unconvenient to track the number of mappers,where mappers run,how to track the exception logs etc.note ,this is the default behavior by hadoop(maybe decrease the resouces occupied).
and yep,there is a deamon to process these logs files,within it u can find what u can see in hadoop-1.x.here are some steps to enable this feature below:
1.add a property in yarn-site.xml
yarn.log-aggregation-enable=true
note:if u specify tihs ,the containers logs under 'userlogs' will be removed after job completed.
also,u can specify some relative items:
<property>
<description>How long to keep aggregation logs before deleting them. -1 disables.
Be careful set this too small and you will spam the name node.</description>
<name>yarn.log-aggregation.retain-seconds</name>
<value>-1</value>
</property>
<property>
<description>How long to wait between aggregated log retention checks.
If set to 0 or a negative value then the value is computed as one-tenth
of the aggregated log retention time. Be careful set this too small and
you will spam the name node.</description>
<name>yarn.log-aggregation.retain-check-interval-seconds</name>
<value>-1</value>
</property>
<property>
<description>Time in seconds to retain user logs. Only applicable if
log aggregation is disabled
</description>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>10800</value>
</property>
<property>
<description>Where to aggregate logs to.</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/tmp/logs</value>
</property>
<property>
<description>The remote log dir will be created at
{yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam}
</description>
<name>yarn.nodemanager.remote-app-log-dir-suffix</name>
<value>logs</value>
</property>
2. specify the host where the 'JobHistoryServer' to run by mapred-site.xml
<property>
<name>mapreduce.jobhistory.address</name>
<value>host:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>host:19888</value>
</property>
3.spawn mr history server
mr-jobhistory-daemon.sh start historyserver
now u can see a daemon named JobHistoryServer in the host configured by mapreduce.jobhistory.address
after all above complete,u can go to the historyserver by
http://host:19888/jobhistory
now some figures are shown here:
if u click the link 'history'(ie.http://host:50030/proxy/application_1418972108758_0001/jobhistory/job/job_1418972108758_0001),then u will be redirected to the job history server 'http://host:19888/jobhistory/job/job_1418972108758_0001/jobhistory/job/job_1418972108758_0001'
ref: