一、内存不足
报错信息:
java.lang.OutOfMemoryError: unable to create new native thread
解决:
调用的IOB多的时候就会发生报错,可通过调整gc来控制:
if [ -z "$PENTAHO_DI_JAVA_OPTIONS" ]; then
PENTAHO_DI_JAVA_OPTIONS="-Xms4096m -Xmx4096m -XX:+UseSerialGC"
fi
二、kettle JOB 假死,需要到系统上面杀死进程
报错信息: JOB一直处理执行状态
解决:
1.修改kettle配置文件/home/hadoopap/data-integration/system/karaf/system/pentaho-karaf-features/pentaho-big-data-plugin-osgi/8.0.0.0-28/pentaho-big-data-plugin-osgi-8.0.0.0-28-features.xml
2.修改ulimit -u和ulimit -n,修改系统配置文件/etc/security/limits.conf和/etc/security/limits.d/20-nproc.conf
三、ETL调度kettle时报Java.lang.Exception: Error running remote command.
解决方案:
①登录作业数据源设置的服务器,即通用shell服务器
②修改/etc/ssh/sshd_config中#MaxStartups,将其改为MaxStartups 200
③重启SSH服务:systemctl restart sshd.service
四、ETL调度kettle时报No suitable driver found for jdbc:hive2://xx.xx.xx.xx:21050/dbname;auth=noSasl;impala_db=true
kettle的impala连接问题
1.在kettle目录data-integration\simple-jndi下将如下语句加入jdbc.properties
JNDI_p4_impala_p4edwadm/type=javax.sql.DataSource
JNDI_p4_impala_p4edwadm/driver=com.cloudera.impala.jdbc41.Driver
JNDI_p4_impala_p4edwadm/url=jdbc:impala://xx.xx.xx.xx:25004/dbname;request_pool="hadoopap"
JNDI_p4_impala_p4edwadm/user=hadoopap
JNDI_p4_impala_p4edwadm/password=12344
2.将impala jdbc驱动ImpalaJDBC41.jar添加到data-integration\lib目录下
3.数据库连接方式替换为Cloudera Impala中的jndi模式,并配置jndi为JNDI_p4_impala_p4edwadm,点击测试
4.分别将驱动和jdbc.properties拷贝到linux环境的kettle相同目录中
五:kettle报提示需要安装libwebkitgtk
需要安装libwebkitgtk,具体安装步骤见https://blog.csdn.net/m0_37618809/article/details/81015492