今天在用solr读取数据库连接,然后根据链接通过web服务器取文件,本以为很简单的问题,弄了一天。
主要参考了这几篇文章:https://issues.apache.org/jira/browse/SOLR-4184
http://blog.sina.com.cn/s/blog_986e726b0101f7lx.html
<script><![CDATA[
id = 1;
function GenerateId(row) {
row.put('id', (id ++).toFixed());
var url = row.get('FJLJ');
var index = url.lastIndexOf("/");
var fileName = encodeURI(url.substring(index+1));//这儿主要是最后要取得文件含中文时,在第二个entity执行之前转码
url = url.substring(0,index+1)+fileName;
row.put('FJLJ', url);
return row;
}
]]>
</script>
这是第二个entity:
<entity processor="TikaEntityProcessor" name="onlineTxtFile" onError="skip" //主要是onerror 当下载文件出错时跳过此错误
url="http://192.168.15.40:8080/zdfzxxglpt/servlets/solrServlet?url=${fjb1.FJLJ}" dataSource="binURLDataSource" format="text">
<field column="Author" name="author" />
<field column="title" name="title" />
<field column="text" name="text"/>
</entity>
还有查询txt时,当中文很少时可以读出乱码,中文很多时直接读不出来
try {
bis = new BufferedInputStream(new FileInputStream(new File(dirPath)));
bos = new BufferedOutputStream(response.getOutputStream());
byte[] b = new byte[1024];
int length;
while ((length = bis.read(b)) > 0) {
if(url.endsWith("txt")){
String block = new String(b,"gbk"); //当为txt时重新设置编码,虽然感觉这样写很不好,但终归是出效果了
b = block.getBytes("utf-8");
}
bos.write(b, 0, length);
}
bos.flush();
}
还有一些oracle字段查出来大写等等
<field column="FJID" name="id"/>
这是一个完整的entity,注意pk值大写:
<entity name="fjb" query="select t1.fjid,t1.fjgs,t1.uptime,t1.fjlj from DFZ_FJB t1 join dfz_zlsbb t2 on t1.ywid=t2.zlid where t2.zt!='1' and t1.zt is null" pk="ID" transformer="DateFormatTransformer,ClobTransformer,script:GenerateId" dataSource="oracle"
deltaQuery="select t1.fjid as id from DFZ_FJB t1 join dfz_zlsbb t2 on t1.ywid=t2.zlid where t2.zt!='1' and t1.zt is null and t1.uptime > '${dataimporter.fjb.last_index_time}'"
deletedPkQuery="select t1.fjid as id FROM DFZ_FJB t1 WHERE t1.zt='2' and t1.uptime > '${dataimporter.fjb.last_index_time}'"
deltaImportQuery="select t1.fjid,t1.fjgs,t1.uptime,t1.fjlj from DFZ_FJB t1 where t1.fjid='${dataimporter.delta.ID}'">
<field column="FJID" name="id"/>
<field column="fjgs" />
<field column="uptime" />
<field column="fjlj" />
<entity processor="TikaEntityProcessor" name="onlineTxtFile" onError="skip" url="http://192.168.15.40:8080/Lzdfzxxglpt/servlets/solrServlet?url=${fjb.FJLJ}" dataSource="binURLDataSource" format="text">
<field column="Author" name="author" meta="true"/>
<field column="title" name="title" meta="true"/>
<field column="text" name="text"/>
</entity>
</entity>