springboot整合kettle, 訪問不到ktr檔案
阿新 • • 發佈：2018-12-16
背景介紹
由於系統需要從sql server 遠端資料庫抓取資料儲存到當前資料庫（mysql）中，所以採用了kettle來進行資料抽取。使用spoon圖形化工具進行ktr檔案的編寫。測試通過後，想整合進專案中，然而遇到以下問題：
- 首先，kettle的jar包引入，在一些共有的maven庫中找不到相關jar包。所以從spoon客戶端中，拷貝出jar包，然後加入本地maven庫。主要包括kettle-core、kettle-engine、metastore、commons-vfs2、commons-lang。
mvn install:install-file -DgroupId=pentaho-kettle -DartifactId=kettle-core -Dversion=7.1.0.0-12 -Dpackaging=jar -Dfile=pentaho-kettle.jar
- 其次，將專案部署到linux伺服器後，發現kettle讀不到ktr檔案，檢視堆疊資訊，發現路徑正常。
org.pentaho.di.core.exception.KettleXMLException: Unable to read file [file:///pws.jar!/BOOT-INF/classes!/kettle/userInfo.ktr] Could not read from "file:///pws.jar!/BOOT-INF/classes!/kettle/userInfo.ktr" because it is a not a file. at org.pentaho.di.core.xml.XMLHandler.loadXMLFile(XMLHandler.java:561) at org.pentaho.di.core.xml.XMLHandler.loadXMLFile(XMLHandler.java:540) at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2742) at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2710) at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2687) at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2667) at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2632) at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2595) at cn.pioneer.pws.service.impl.KettleServiceImpl.synchronizationInfo(KettleServiceImpl.java:50) at cn.pioneer.pws.service.impl.KettleServiceImpl.synchroUserData(KettleServiceImpl.java:29) at cn.pioneer.pws.controller.KettleController.synchroUserData(KettleController.java:21) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:877) at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851) at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at cn.pioneer.pws.util.PageFilter.doFilter(PageFilter.java:46) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.commons.vfs2.FileNotFoundException: Could not read from "file:///pws.jar!/BOOT-INF/classes!/kettle/userInfo.ktr" because it is a not a file. at org.apache.commons.vfs2.provider.AbstractFileObject.getInputStream(AbstractFileObject.java:1315) at org.apache.commons.vfs2.provider.DefaultFileContent.getInputStream(DefaultFileContent.java:396) at org.pentaho.di.core.vfs.KettleVFS.getInputStream(KettleVFS.java:267) at org.pentaho.di.core.xml.XMLHandler.loadXMLFile(XMLHandler.java:559) ... 62 common frames omitted Caused by: java.io.FileNotFoundException: /pws.jar!/BOOT-INF/classes!/kettle/userInfo.ktr (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at org.apache.commons.vfs2.provider.local.LocalFile.doGetInputStream(LocalFile.java:209) at org.apache.commons.vfs2.provider.AbstractFileObject.getInputStream(AbstractFileObject.java:1307) ... 65 common frames omitted
問題分析
- 首先，由於在idea中專案能夠正常執行，但在伺服器上面執行失敗。第一反應是window與linux的差異造成的。然後在本機以java -jar xxx 方式執行，同樣報錯。
String filePath = this.getClass().getClassLoader().getResource("kettle/userInfo.ktr").getPath(); KettleEnvironment.init(); TransMeta transMeta = new TransMeta(filePath); Trans trans = new Trans(transMeta); if (!StringUtils.isNullOrEmpty(department)){ trans.setVariable("department", department); } trans.execute(null); // You can pass arguments instead of null. trans.waitUntilFinished();
其中，String filePath = this.getClass().getClassLoader().getResource("kettle/userInfo.ktr").getPath();獲取資原始檔ktr的路徑。
TransMeta transMeta = new TransMeta(filePath);將路徑傳遞給kettle內庫中的處理類，進行ktr檔案的解析及執行。
而問題在於以java -jar 方式執行程式時，由於專案沒有解壓，程式不能訪問未解壓的資原始檔。
解決方案
- 將字串的路徑引數改成流（InputStream）。檢視kettle原始碼發現， TransMeta transMeta = new TransMeta(filePath);內部連續多次呼叫其他帶引數方法，且新增引數都為null，即引數可以忽略。
InputStream inputStream = this.getClass().getClassLoader().getResourceAsStream("/kettle/userInfo.ktr");
KettleEnvironment.init();
TransMeta transMeta = new TransMeta(inputStream, null, true, null,null);
Trans trans = new Trans(transMeta);
if (!StringUtils.isNullOrEmpty(department)){
trans.setVariable("department", department);
}
trans.execute(null); // You can pass arguments instead of null.
trans.waitUntilFinished();