本地windows跑Scala程式呼叫Spark
阿新 • • 發佈:2019-02-06
show at SparkConnMysql.scala:25 17/11/21 11:44:05 INFO DAGScheduler: Got job 0 (show at SparkConnMysql.scala:25) with 1 output partitions 17/11/21 11:44:05 INFO DAGScheduler: Final stage: ResultStage 0 (show at SparkConnMysql.scala:25) 17/11/21 11:44:05 INFO DAGScheduler: Parents of final stage: List() 17/11/21 11 :44:05 INFO DAGScheduler: Missing parents: List() 17/11/21 11:44:05 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at show at SparkConnMysql.scala:25), which has no missing parents 17/11/21 11:44:06 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 5.2 KB, free 5.2 KB) 17/11/21 11:44:06 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 2.5 KB, free 7.7 KB) 17/11/21 11:44:06 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:55844 (size: 2.5 KB, free: 1129.9 MB) 17/11/21 11:44:06 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 17/11/21 11:44:06 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at show at SparkConnMysql.scala:25) 17/11/21 11:44:06 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks 17/11/21 11:44:06 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 1922 bytes) 17/11/21 11:44:06 INFO Executor: Running task 0.0 in stage 0.0 (TID 0) 17/11/21 11:44:06 INFO JDBCRDD: closed connection 17/11/21 11:44:06 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 3472 bytes result sent to driver 17/11/21 11:44:06 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 224 ms on localhost (1/1) 17/11/21 11:44:06 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 17/11/21 11:44:06 INFO DAGScheduler: ResultStage 0 (show at SparkConnMysql.scala:25) finished in 0.261 s 17/11/21 11:44:06 INFO DAGScheduler: Job 0 finished: show at SparkConnMysql.scala:25, took 1.467252 s +---+----+----+------------+------------------+---------+-------+ | id|name| age| phone| email|startdate|enddate| +---+----+----+------------+------------------+---------+-------+ | 11| 徐心三| 24| 2423424| 2423424@qq.com| null| null| | 33| 徐心七| 23| 23232323| 13131@qe| null| null| | 55| 徐彬| 22| 15262301036|徐彬757661238@ww.com| null| null| | 44| 徐成|3333| 23423424332| 2342423@qq.com| null| null| | 66| 徐心四| 23|242342342423| 徐彬23424@qq.com| null| null| | 11| 徐心三| 24| 2423424| 2423424@qq.com| null| null| | 33| 徐心七| 23| 23232323| 13131@qe| null| null| | 55| 徐彬| 22| 15262301036|徐彬757661238@ww.com| null| null| | 44| 徐成|3333| 23423424332| 2342423@qq.com| null| null| | 66| 徐心四| 23|242342342423| 徐彬23424@qq.com| null| null| | 88| 徐心八| 123| 131231312| 123123@qeqe| null| null| | 99| 徐心二| 23| 13131313| 1313133@qeq.com| null| null| |121| 徐心五| 13| 123131231| 1231312@qq.com| null| null| |143| 徐心九| 23| 234234| 徐彬234@wrwr| null| null| +---+----+----+------------+------------------+---------+-------+ only showing top 14 rows 17/11/21 11:44:06 INFO SparkContext: Invoking stop() from shutdown hook 17/11/21 11:44:06 INFO SparkUI: Stopped Spark web UI at http://192.168.114.67:4040 17/11/21 11:44:06 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 17/11/21 11:44:06 INFO MemoryStore: MemoryStore cleared 17/11/21 11:44:06 INFO BlockManager: BlockManager stopped 17/11/21 11:44:06 INFO BlockManagerMaster: BlockManagerMaster stopped 17/11/21 11:44:06 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 17/11/21 11:44:06 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 17/11/21 11:44:06 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
17/11/21 11:44:06 INFO SparkContext: Successfully stopped SparkContext
17/11/21 11:44:07 INFO ShutdownHookManager: Shutdown hook called
17/11/21 11:44:07 INFO ShutdownHookManager: Deleting directory C:\Users\lenovo\AppData\Local\Temp\spark-7877d903-f8f7-4efb-9e0c-7a11ac147153