flume相关报错

最新推荐文章于 2023-10-14 21:26:21 发布

不要海带和香菜

最新推荐文章于 2023-10-14 21:26:21 发布

阅读量367

点赞数

分类专栏： flume 文章标签： flume java

本文链接：https://blog.csdn.net/qq_38855557/article/details/113884036

版权

flume 专栏收录该内容

4 篇文章 0 订阅

订阅专栏

（1）Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight

（2）org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 1000 full, consider committing more frequently, increasing capacity, or increasing thread count.

原因：

这两个问题的原因都与capacity和transacCapacity和batchSize有关，

我们的环境是syslogtcp source+ memory channel + hdfs sink，

syslogtcp source每次向channel搬数据的eventSize为2500，则我们设置memory channel的transactionCapacity需大于或等于2500，

而且可能会存在source连续多次抢到锁向channel放数据的情况，所以，capacity应大于transactionCapacity，Hdfs.batchSize需小于transactionCapacity。

（3）Flume "java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.setWriteToWAL"

解决方法：

找到hbaseSink.java, 通过find查找哪里有setWriteWAL, 发现有3个地方存在,

private void putEventsAndCommit(final List<Row> actions,
                                final List<Increment> incs, Transaction txn) throws Exception {

  privilegedExecutor.execute(new PrivilegedExceptionAction<Void>() {
    @Override
    public Void run() throws Exception {
      for (Row r : actions) {
        if (r instanceof Put) {
          // ((Put) r).setWriteToWAL(enableWal);
        }
        // Newer versions of HBase - Increment implements Row.
        if (r instanceof Increment) {
          // ((Increment) r).setWriteToWAL(enableWal);
        }
      }
      table.batch(actions);
      return null;
    }
  });

  privilegedExecutor.execute(new PrivilegedExceptionAction<Void>() {
    @Override
    public Void run() throws Exception {

      List<Increment> processedIncrements;
      if (batchIncrements) {
        processedIncrements = coalesceIncrements(incs);
      } else {
        processedIncrements = incs;
      }

      // Only used for unit testing.
      if (debugIncrCallback != null) {
        debugIncrCallback.onAfterCoalesce(processedIncrements);
      }

      for (final Increment i : processedIncrements) {
        // i.setWriteToWAL(enableWal);
        table.increment(i);
      }
      return null;
    }
  });

  txn.commit();
  sinkCounter.addToEventDrainSuccessCount(actions.size());
}

注释掉上面3个setWriteWAL，然后再重新打一个包进行替换，官方名字叫：flume-ng-hbase-sink-1.8.0.jar。重新启动Flume。