kettle-java代码执行hive相关ktr时报错： database type with plugin id [HIVE2] couldn‘t be found!

最新推荐文章于 2024-05-25 11:16:40 发布

置顶 lisery_nj

最新推荐文章于 2024-05-25 11:16:40 发布

阅读量6.5k

点赞数 8

分类专栏： kettle 文章标签： kettle hive kettle ktr

本文链接：https://blog.csdn.net/m0_37618809/article/details/81632342

版权

kettle 专栏收录该内容

11 篇文章 1 订阅

订阅专栏

1.在java项目中执行hive相关的ktr:

 
		KettleEnvironment.init();

		TransMeta transMeta = new TransMeta("C:\\Users\\lixiang\\Desktop\\hivetorizhi.ktr");
		// 设置执行模式
		TransExecutionConfiguration config = new TransExecutionConfiguration();
		config.setExecutingClustered(true);
		config.setExecutingLocally(false);
		config.setExecutingRemotely(false);
		config.setClusterPosting(true);
		config.setClusterPreparing(true);
		config.setClusterStarting(true);
		TransSplitter transSplitter = Trans.executeClustered(transMeta, config);
		System.out.println(transSplitter.getCarteObjectMap());
		System.out.println(transSplitter.getMaster());
		System.out.println(transSplitter.getSlaves()[0]);
		System.out.println(transSplitter.getSlaves()[1].getStepNames()[0]);
}

2.报错：

Exception in thread "main" org.pentaho.di.core.exception.KettleXMLException: 
错误从XML文件读取转换

错误从XML文件读取转换

Unable to load database connection info from XML node

Unable to create new database interface

database type with plugin id [HIVE2] couldn't be found!

	at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:3402)
	at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2758)
	at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2710)
	at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2687)
	at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2667)
	at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2632)
	at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2595)
	at com.kettle.KettleExamle.getRemoteServer(KettleExamle.java:55)
	at com.kettle.KettleExamle.main(KettleExamle.java:140)
Caused by: org.pentaho.di.core.exception.KettleXMLException: 
错误从XML文件读取转换

Unable to load database connection info from XML node

Unable to create new database interface

database type with plugin id [HIVE2] couldn't be found!

	at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:3384)
	... 8 more
Caused by: org.pentaho.di.core.exception.KettleXMLException: 
Unable to load database connection info from XML node

Unable to create new database interface

database type with plugin id [HIVE2] couldn't be found!

	at org.pentaho.di.core.database.DatabaseMeta.<init>(DatabaseMeta.java:1013)
	at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:2976)
	... 8 more
Caused by: org.pentaho.di.core.exception.KettleXMLException: 
Unable to create new database interface

database type with plugin id [HIVE2] couldn't be found!

	at org.pentaho.di.core.database.DatabaseMeta.<init>(DatabaseMeta.java:977)
	... 9 more
Caused by: org.pentaho.di.core.exception.KettleDatabaseException: 
database type with plugin id [HIVE2] couldn't be found!

	at org.pentaho.di.core.database.DatabaseMeta.findDatabaseInterface(DatabaseMeta.java:584)
	at org.pentaho.di.core.database.DatabaseMeta.getDatabaseInterface(DatabaseMeta.java:558)
	at org.pentaho.di.core.database.DatabaseMeta.<init>(DatabaseMeta.java:974)
	... 9 more

3.报错分析：kettle程序在读取database type的过程中读不到hive2这个类型。

查看kettle-core-7.1.0.0-12.jar中的kettle-database-types.xml文件，此文件定义了database type ,其中没hive2.。

<database-types>

   <database-type id="AS/400">                                                        
    <description>AS/400</description>                                                
    <classname>org.pentaho.di.core.database.AS400DatabaseMeta</classname>            
  </database-type>                                                                   

  <database-type id="CACHE">
    <description>Intersystems Cache</description>
    <classname>org.pentaho.di.core.database.CacheDatabaseMeta</classname>
  </database-type>                                                       

  <database-type id="DB2">
    <description>IBM DB2</description>
    <classname>org.pentaho.di.core.database.DB2DatabaseMeta</classname>
  </database-type>                                                     

  <database-type id="DBASE">
    <description>dBase III, IV or 5</description>
    <classname>org.pentaho.di.core.database.DbaseDatabaseMeta</classname>
  </database-type>                                                       

  <database-type id="DERBY">
    <description>Apache Derby</description>
    <classname>org.pentaho.di.core.database.DerbyDatabaseMeta</classname>
  </database-type>                                                       

  <database-type id="EXASOL4">
    <description>Exasol 4</description>
    <classname>org.pentaho.di.core.database.Exasol4DatabaseMeta</classname>
  </database-type>                                                         

  <database-type id="EXTENDB">
    <description>ExtenDB</description>
    <classname>org.pentaho.di.core.database.ExtenDBDatabaseMeta</classname>
  </database-type>                                                         

  <database-type id="FIREBIRD">
    <description>Firebird SQL</description>
    <classname>org.pentaho.di.core.database.FirebirdDatabaseMeta</classname>
  </database-type>                                                          

  <database-type id="GENERIC">
    <description>Generic database</description>
    <classname>org.pentaho.di.core.database.GenericDatabaseMeta</classname>
  </database-type>                                                         

  <database-type id="GREENPLUM">
    <description>Greenplum</description>
    <classname>org.pentaho.di.core.database.GreenplumDatabaseMeta</classname>
  </database-type>                                                           

  <database-type id="INFINIDB">
    <description>Calpont InfiniDB</description>
    <classname>org.pentaho.di.core.database.InfiniDbDatabaseMeta</classname>
  </database-type>
 
  <database-type id="SQLBASE">
    <description>Gupta SQL Base</description>
    <classname>org.pentaho.di.core.database.GuptaDatabaseMeta</classname>
  </database-type>                                                        

  <database-type id="H2">
    <description>H2</description>
    <classname>org.pentaho.di.core.database.H2DatabaseMeta</classname>
  </database-type>                                                    

  <database-type id="HYPERSONIC">
    <description>Hypersonic</description>
    <classname>org.pentaho.di.core.database.HypersonicDatabaseMeta</classname>
  </database-type>                                                            

  <database-type id="INFOBRIGHT">
    <description>Infobright</description>
    <classname>org.pentaho.di.core.database.InfobrightDatabaseMeta</classname>
  </database-type>                                                            

  <database-type id="INFORMIX">
    <description>Informix</description>
    <classname>org.pentaho.di.core.database.InformixDatabaseMeta</classname>
  </database-type>                                                          

  <database-type id="INGRES">
    <description>Ingres</description>
    <classname>org.pentaho.di.core.database.IngresDatabaseMeta</classname>
  </database-type>                                                        

  <database-type id="VECTORWISE">
    <description>Ingres VectorWise</description>
    <classname>org.pentaho.di.core.database.VectorWiseDatabaseMeta</classname>
  </database-type>                                                        

  <database-type id="INTERBASE">
    <description>Borland Interbase</description>
    <classname>org.pentaho.di.core.database.InterbaseDatabaseMeta</classname>
  </database-type>                                                           

  <database-type id="KINGBASEES">
    <description>KingbaseES</description>
    <classname>org.pentaho.di.core.database.KingbaseESDatabaseMeta</classname>
  </database-type>                                                            

  <database-type id="LucidDB">
    <description>LucidDB</description>
    <classname>org.pentaho.di.core.database.LucidDBDatabaseMeta</classname>
  </database-type>                                                         

  <database-type id="MARIADB">
    <description>MariaDB</description>
    <classname>org.pentaho.di.core.database.MariaDBDatabaseMeta</classname>
  </database-type>

  <database-type id="MONETDB">
    <description>MonetDB</description>
    <classname>org.pentaho.di.core.database.MonetDBDatabaseMeta</classname>
  </database-type>                                                         

  <database-type id="MSACCESS">
    <description>MS Access</description>
    <classname>org.pentaho.di.core.database.MSAccessDatabaseMeta</classname>
  </database-type>                                                          

  <database-type id="MSSQLNATIVE">
    <description>MS SQL Server (Native)</description>
    <classname>org.pentaho.di.core.database.MSSQLServerNativeDatabaseMeta</classname>
  </database-type>                                                             

  <database-type id="MSSQL">
    <description>MS SQL Server</description>
    <classname>org.pentaho.di.core.database.MSSQLServerDatabaseMeta</classname>
  </database-type>                                                             

  <database-type id="MYSQL">
    <description>MySQL</description>
    <classname>org.pentaho.di.core.database.MySQLDatabaseMeta</classname>
  </database-type>                                                       

  <database-type id="MONDRIAN">
    <description>Native Mondrian</description>
    <classname>org.pentaho.di.core.database.MondrianNativeDatabaseMeta</classname>
  </database-type>                                                       

  <database-type id="NEOVIEW">
    <description>Neoview</description>
    <classname>org.pentaho.di.core.database.NeoviewDatabaseMeta</classname>
  </database-type>                                                         

  <database-type id="NETEZZA">
    <description>Netezza</description>
    <classname>org.pentaho.di.core.database.NetezzaDatabaseMeta</classname>
  </database-type>                                                         

  <database-type id="ORACLE">
    <description>Oracle</description>
    <classname>org.pentaho.di.core.database.OracleDatabaseMeta</classname>
  </database-type>                                                        

  <database-type id="ORACLERDB">
    <description>Oracle RDB</description>
    <classname>org.pentaho.di.core.database.OracleRDBDatabaseMeta</classname>
  </database-type>                                                           

  <database-type id="POSTGRESQL">
    <description>PostgreSQL</description>
    <classname>org.pentaho.di.core.database.PostgreSQLDatabaseMeta</classname>
  </database-type>

  <database-type id="REDSHIFT">
    <description>Redshift</description>
    <classname>org.pentaho.di.core.database.RedshiftDatabaseMeta</classname>
  </database-type>

  <database-type id="REMEDY-AR-SYSTEM">
    <description>Remedy Action Request System</description>
    <classname>org.pentaho.di.core.database.RemedyActionRequestSystemDatabaseMeta</classname>                                                                                           
  </database-type>                                                                          

  <database-type id="SAPDB">
    <description>MaxDB (SAP DB)</description>
    <classname>org.pentaho.di.core.database.SAPDBDatabaseMeta</classname>
  </database-type>                                                       

  <database-type id="SQLITE">
    <description>SQLite</description>
    <classname>org.pentaho.di.core.database.SQLiteDatabaseMeta</classname>
  </database-type>

  <database-type id="SYBASE">
    <description>Sybase</description>
    <classname>org.pentaho.di.core.database.SybaseDatabaseMeta</classname>
  </database-type>

  <database-type id="SYBASEIQ">
    <description>SybaseIQ</description>
    <classname>org.pentaho.di.core.database.SybaseIQDatabaseMeta</classname>
  </database-type>

  <database-type id="TERADATA">
    <description>Teradata</description>
    <classname>org.pentaho.di.core.database.TeradataDatabaseMeta</classname>
  </database-type>

  <database-type id="UNIVERSE">
    <description>UniVerse database</description>
    <classname>org.pentaho.di.core.database.UniVerseDatabaseMeta</classname>
  </database-type>

  <database-type id="VERTICA">
    <description>Vertica</description>
    <classname>org.pentaho.di.core.database.VerticaDatabaseMeta</classname>
  </database-type>
  
  <database-type id="VERTICA5">
    <description>Vertica 5+</description>
    <classname>org.pentaho.di.core.database.Vertica5DatabaseMeta</classname>
  </database-type>
 
 </database-types>

4.报错解决：

hive的api的连接方式是通过jdbc连接的。因此kettle连接hive的思路与连接postgresql的思路相同。因此，仿照postgresql的连接方法自定义一个database-type为hive2。第一步，在kettle-core-7.1.0.0-12.jar的kettle-database-types.xml文件中添加下面内容

   <database-type id="HIVE2">
     <description>HIVE2</description>
     <classname>org.pentaho.di.core.database.Hive2SQLDatabaseMeta</classname>
  </database-type>

注意：其中会有对应的实现类org.pentaho.di.core.database.Hive2SQLDatabaseMeta，而这个实现类是不存在的，需要自己手动创建，如何创建呢？请看下一步。

第二步：仿照postgresql的实现类org.pentaho.di.core.database.PostgreSQLDatabaseMeta来写即可，只需将连接的url以及jdbc driver名称更换为hive相关。具体操作：PostgreSQLDatabaseMeta复制一份重命名为Hive2SQLDatabaseMeta，修改其中的其中的getDriverClass 方法的返回值为hive的jdbc Driver和getDriverClass方法的返回值为hive的url。

将org.pentaho.di.core.database.Hive2SQLDatabaseMeta类编译为.class文件（kettle的源码github上面有，自行下载，下载完成以后就可以进行第2步的复制类修改类然后反编译成class文件）放进kettle-core-7.1.0.0-12.jar的org\pentaho\di\core\database路径下

6.将修改后kettle-core-7.1.0.0-12.jar放入执行项目中，执行ktr：

执行成功：结果：

{/hivetorizhi (master)=1cc5f5bb-b0c4-4938-96f7-85363f88564d, /hivetorizhi (local:slave1-8081)=09ea3633-f5bd-4e43-9352-62b958255469, /hivetorizhi (local:slave2-8082)=35e00dc2-f570-4f26-8540-156fb84d9524}
/hivetorizhi (master)
/hivetorizhi (local:slave2-8082)
写日志

over

附加：

适配hive后的kettle-core-7.1.0.0-12.jar 下载链接：https://download.csdn.net/download/m0_37618809/11081777

lisery_nj

关注

8
点赞
踩
10

收藏

觉得还不错? 一键收藏
11
评论
kettle-java代码执行hive相关ktr时报错： database type with plugin id [HIVE2] couldn‘t be found!

1.在java项目中执行hive相关的ktr: KettleEnvironment.init(); TransMeta transMeta = new TransMeta("C:\\Users\\lixiang\\Desktop\\hivetorizhi.ktr"); // 设置执行模式 TransExecutionConfiguration config = new T...
复制链接

扫一扫