异常信息:
*********************************
MapReduce异常信息
*********************************
2015-07-20 14:16:01,045 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert; Ignoring.
2015-07-20 14:16:01,049 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-07-20 14:16:01,050 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.client.conf; Ignoring.
2015-07-20 14:16:01,053 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.keystores.factory.class; Ignoring.
2015-07-20 14:16:01,057 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.server.conf; Ignoring.
2015-07-20 14:16:01,081 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-07-20 14:16:01,358 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2015-07-20 14:16:01,463 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2015-07-20 14:16:01,463 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2015-07-20 14:16:01,479 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2015-07-20 14:16:01,479 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1435737529069_0025, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@63a6d923)
2015-07-20 14:16:01,631 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2015-07-20 14:16:02,065 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /mnt/sdb1/yarn/nmx/usercache/root/appcache/application_1435737529069_0025,/mnt/sdc1/yarn/nmx/usercache/root/appcache/application_1435737529069_0025,/mnt/sdd1/yarn/nmx/usercache/root/appcache/application_1435737529069_0025,/mnt/sde1/yarn/nmx/usercache/root/appcache/application_1435737529069_0025,/mnt/sdf1/yarn/nmx/usercache/root/appcache/application_1435737529069_0025,/mnt/sdg1/yarn/nmx/usercache/root/appcache/application_1435737529069_0025
2015-07-20 14:16:02,271 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert; Ignoring.
2015-07-20 14:16:02,274 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-07-20 14:16:02,275 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.client.conf; Ignoring.
2015-07-20 14:16:02,276 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.keystores.factory.class; Ignoring.
2015-07-20 14:16:02,279 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.server.conf; Ignoring.
2015-07-20 14:16:02,295 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-07-20 14:16:02,668 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2015-07-20 14:16:03,198 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2015-07-20 14:16:03,454 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: com.linkedin.camus.etl.kafka.mapred.EtlSplit@d9f20e5
2015-07-20 14:16:03,582 INFO [main] org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2015-07-20 14:16:03,584 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
2015-07-20 14:16:03,595 INFO [main] com.linkedin.camus.etl.kafka.mapred.EtlRecordReader:
topic:hive-xm-syslog5 partition:0 beginOffset:80 estimatedLastOffset:120
2015-07-20 14:16:03,603 INFO [main] com.linkedin.camus.etl.kafka.common.KafkaReader: bufferSize=10485760
2015-07-20 14:16:03,603 INFO [main] com.linkedin.camus.etl.kafka.common.KafkaReader: timeout=30000
2015-07-20 14:16:04,029 INFO [main] com.linkedin.camus.etl.kafka.common.KafkaReader: Connected to leader tcp://m105:9092 beginning reading at offset 80 latest offset=120
2015-07-20 14:16:05,657 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2015-07-20 14:16:05,702 INFO [main] org.apache.hadoop.hive.metastore.ObjectStore: ObjectStore, initialize called
2015-07-20 14:16:06,018 INFO [main] DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
2015-07-20 14:16:06,019 INFO [main] DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
2015-07-20 14:16:06,686 ERROR [BoneCP-pool-watch-thread] com.jolbox.bonecp.PoolWatchThread: Error in trying to obtain a connection. Retrying in 7000ms
java.sql.SQLException: A read-only user or a user in a read-only datab
ase is not permitted to disable read-only mode on a connection.
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.setReadOnly(Unknown Source)
at com.jolbox.bonecp.ConnectionHandle.setReadOnly(ConnectionHandle.java:1324)
at com.jolbox.bonecp.ConnectionHandle.<init>(ConnectionHandle.java:262)
at com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:115)
at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:82)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
... 13 more
Caused by: ERROR 25505: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.sql.conn.GenericAuthorizer.setReadOnlyConnection(Unknown Source)
at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.setReadOnly(Unknown Source)
... 8 more
原因及方案解决:
m102节点找不到对应的Hive表元数据地址,造成程序偶尔运行异常,在hive-site.xml中配置Hive元数据信息
<property>
<name>hive.metastore.uris</name>
<value>thrift://m103:9083,thrift://m105:9083</value>
</property>