spark on yarn 異常記錄二
阿新 • • 發佈:2019-01-27
通過spark-submit 向叢集提交任務時,–num-executors 數量大於1即會出現異常,異常如下:
17/06/04 11:47:03 WARN DefaultChannelPipeline: An exception 'java.lang.NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
java.lang.NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;
at org.apache.spark.rpc.netty.NettyRpcHandler.channelInactive(NettyRpcEnv.scala:621)
at org.apache.spark.network.server.TransportRequestHandler.channelInactive(TransportRequestHandler.java:99 )
at org.apache.spark.network.server.TransportChannelHandler.channelInactive(TransportChannelHandler.java:103)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:251)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:237 )
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:230)
at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
at org.spark_project.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:257)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:251)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:237)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:230)
at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:251)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:237)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:230)
at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
at org.apache.spark.network.util.TransportFrameDecoder.channelInactive(TransportFrameDecoder.java:182)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:251)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:237)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:230)
at org.spark_project.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1289)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:251)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:237)
at org.spark_project.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:893)
at org.spark_project.io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:691)
at org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:376)
at org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:680)
at org.spark_project.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465)
at org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
at org.spark_project.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:745)
17/06/04 11:47:03 INFO ShutdownHookManager: Shutdown hook called
google半天,有很多說法,但是依舊沒有解決我的問題
參考 jira中各種方法,問題並沒有得到解決。想著,異常原因是方法找不到,那麼我把這個類單獨提取出來,放到我要提交的jar中,會不會得到解決,畢竟以前遇到過這種情況,就是這麼解決的。試了試,問題並沒有得到解決。暫時先記著,等請教幾個大牛,再來更新……