2. 程式人生 > >hadoop-yarn叢集中，通過shell指令碼自動化提交spark任務

hadoop-yarn叢集中，通過shell指令碼自動化提交spark任務

阿新 • • 發佈：2018-12-22
Set() 18/02/11 12:07:32 INFO yarn.Client: Submitting application application_1518316627470_0003 to ResourceManager 18/02/11 12:07:32 INFO impl.YarnClientImpl: Submitted application application_1518316627470_0003 18/02/11 12:07:32 INFO cluster.SchedulerExtensionServices: Starting Yarn extension services with app application_1518316627470_0003 and
attemptId None 18/02/11 12:07:33 INFO yarn.Client: Application report for application_1518316627470_0003 (state: ACCEPTED) 18/02/11 12:07:33 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start
time: 1518322052230 final status: UNDEFINED tracking URL: http://hadoop1:8088/proxy/application_1518316627470_0003/ user: elon 18/02/11 12:07:34 INFO yarn.Client: Application report for application_1518316627470_0003 (state: ACCEPTED) 18/02/11 12:07:35 INFO yarn.Client: Application report for application_1518316627470_0003 (state: ACCEPTED) 18
/02/11 12:07:36 INFO yarn.Client: Application report for application_1518316627470_0003 (state: ACCEPTED) 18/02/11 12:07:37 INFO yarn.Client: Application report for application_1518316627470_0003 (state: ACCEPTED) 18/02/11 12:07:38 INFO yarn.Client: Application report for application_1518316627470_0003 (state: ACCEPTED) 18/02/11 12:07:39 INFO yarn.Client: Application report for application_1518316627470_0003 (state: ACCEPTED) 18/02/11 12:07:40 INFO yarn.Client: Application report for application_1518316627470_0003 (state: ACCEPTED) 18/02/11 12:07:41 INFO yarn.Client: Application report for application_1518316627470_0003 (state: ACCEPTED) 18/02/11 12:07:42 INFO yarn.Client: Application report for application_1518316627470_0003 (state: ACCEPTED) 18/02/11 12:07:43 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM) 18/02/11 12:07:43 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> hadoop1, PROXY_URI_BASES -> http://hadoop1:8088/proxy/application_1518316627470_0003), /proxy/application_1518316627470_0003 18/02/11 12:07:43 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter 18/02/11 12:07:43 INFO yarn.Client: Application report for application_1518316627470_0003 (state: ACCEPTED) 18/02/11 12:07:44 INFO yarn.Client: Application report for application_1518316627470_0003 (state: RUNNING) 18/02/11 12:07:44 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: 192.168.1.113 ApplicationMaster RPC port: 0 queue: default start time: 1518322052230 final status: UNDEFINED tracking URL: http://hadoop1:8088/proxy/application_1518316627470_0003/ user: elon 18/02/11 12:07:44 INFO cluster.YarnClientSchedulerBackend: Application application_1518316627470_0003 has started running. 18/02/11 12:07:44 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38368. 18/02/11 12:07:44 INFO netty.NettyBlockTransferService: Server created on 192.168.1.111:38368 18/02/11 12:07:44 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 18/02/11 12:07:44 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.1.111, 38368, None) 18/02/11 12:07:44 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.1.111:38368 with 117.0 MB RAM, BlockManagerId(driver, 192.168.1.111, 38368, None) 18/02/11 12:07:44 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.1.111, 38368, None) 18/02/11 12:07:44 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.1.111, 38368, None) 18/02/11 12:07:45 INFO handler.ContextHandler: Started [email protected]5db3d57c{/metrics/json,null,AVAILABLE,@Spark} 18/02/11 12:07:46 INFO scheduler.EventLoggingListener: Logging events to file:/tmp/spark-events/application_1518316627470_0003 18/02/11 12:07:53 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms) 18/02/11 12:07:54 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 290.1 KB, free 116.7 MB) 18/02/11 12:07:54 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.7 KB, free 116.7 MB) 18/02/11 12:07:54 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.1.111:38368 (size: 23.7 KB, free: 116.9 MB) 18/02/11 12:07:54 INFO spark.SparkContext: Created broadcast 0 from textFile at WordCount.scala:22 18/02/11 12:07:54 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.1.113:35724) with ID 1 18/02/11 12:07:55 INFO storage.BlockManagerMasterEndpoint: Registering block manager hadoop3:33799 with 117.0 MB RAM, BlockManagerId(1, hadoop3, 33799, None) 18/02/11 12:07:55 INFO mapred.FileInputFormat: Total input paths to process : 1 18/02/11 12:07:58 INFO spark.SparkContext: Starting job: take at WordCount.scala:26 18/02/11 12:08:02 INFO scheduler.DAGScheduler: Registering RDD 3 (map at WordCount.scala:24) 18/02/11 12:08:02 INFO scheduler.DAGScheduler: Got job 0 (take at WordCount.scala:26) with 1 output partitions 18/02/11 12:08:02 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (take at WordCount.scala:26) 18/02/11 12:08:02 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 0) 18/02/11 12:08:02 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 0) 18/02/11 12:08:02 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at map at WordCount.scala:24), which has no missing parents 18/02/11 12:08:03 INFO memory.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.8 KB, free 116.7 MB) 18/02/11 12:08:03 INFO memory.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.8 KB, free 116.6 MB) 18/02/11 12:08:03 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.1.111:38368 (size: 2.8 KB, free: 116.9 MB) 18/02/11 12:08:03 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 18/02/11 12:08:04 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at map at WordCount.scala:24) (first 15 tasks are for partitions Vector(0, 1)) 18/02/11 12:08:04 INFO cluster.YarnScheduler: Adding task set 0.0 with 2 tasks 18/02/11 12:08:05 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, hadoop3, executor 1, partition 0, PROCESS_LOCAL, 4856 bytes) 18/02/11 12:08:07 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on hadoop3:33799 (size: 2.8 KB, free: 117.0 MB) 18/02/11 12:08:08 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on hadoop3:33799 (size: 23.7 KB, free: 116.9 MB) 18/02/11 12:08:11 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, hadoop3, executor 1, partition 1, PROCESS_LOCAL, 4856 bytes) 18/02/11 12:08:11 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 7027 ms on hadoop3 (executor 1) (1/2) 18/02/11 12:08:12 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 446 ms on hadoop3 (executor 1) (2/2) 18/02/11 12:08:12 INFO cluster.YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool 18/02/11 12:08:12 INFO scheduler.DAGScheduler: ShuffleMapStage 0 (map at WordCount.scala:24) finished in 7.435 s 18/02/11 12:08:12 INFO scheduler.DAGScheduler: looking for newly runnable stages 18/02/11 12:08:12 INFO scheduler.DAGScheduler: running: Set() 18/02/11 12:08:12 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 1) 18/02/11 12:08:12 INFO scheduler.DAGScheduler: failed: Set() 18/02/11 12:08:12 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (ShuffledRDD[4] at reduceByKey at WordCount.scala:24), which has no missing parents 18/02/11 12:08:12 INFO memory.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 3.2 KB, free 116.6 MB) 18/02/11 12:08:12 INFO memory.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2013.0 B, free 116.6 MB) 18/02/11 12:08:12 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.1.111:38368 (size: 2013.0 B, free: 116.9 MB) 18/02/11 12:08:12 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 18/02/11 12:08:12 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (ShuffledRDD[4] at reduceByKey at WordCount.scala:24) (first 15 tasks are for partitions Vector(0)) 18/02/11 12:08:12 INFO cluster.YarnScheduler: Adding task set 1.0 with 1 tasks 18/02/11 12:08:12 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, hadoop3, executor 1, partition 0, NODE_LOCAL, 4632 bytes) 18/02/11 12:08:12 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on hadoop3:33799 (size: 2013.0 B, free: 116.9 MB) 18/02/11 12:08:12 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.168.1.113:35724 18/02/11 12:08:12 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 150 bytes 18/02/11 12:08:12 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 425 ms on hadoop3 (executor 1) (1/1) 18/02/11 12:08:12 INFO scheduler.DAGScheduler: ResultStage 1 (take at WordCount.scala:26) finished in 0.427 s 18/02/11 12:08:12 INFO cluster.YarnScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool 18/02/11 12:08:12 INFO scheduler.DAGScheduler: Job 0 finished: take at WordCount.scala:26, took 14.817908 s (package,1) (this,1) (Version"](http://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version),1) (Because,1) (Python,2) (page](http://spark.apache.org/documentation.html).,1) (cluster.,1) (its,1) ([run,1) (general,3) (have,1) (pre-built,1) (YARN,,1) (locally,2) (changed,1) (locally.,1) (sc.parallelize(1,1) (only,1) (several,1) (This,2) 18/02/11 12:08:12 INFO spark.SparkContext: Invoking stop() from shutdown hook 18/02/11 12:08:13 INFO server.AbstractConnector: Stopped [email protected]{HTTP/1.1,[http/1.1]}{0.0.0.0:4040} 18/02/11 12:08:13 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.1.111:4040 18/02/11 12:08:13 INFO storage.BlockManagerInfo: Removed broadcast_2_piece0 on hadoop3:33799 in memory (size: 2013.0 B, free: 116.9 MB) 18/02/11 12:08:13 INFO storage.BlockManagerInfo: Removed broadcast_2_piece0 on 192.168.1.111:38368 in memory (size: 2013.0 B, free: 116.9 MB) 18/02/11 12:08:13 INFO cluster.YarnClientSchedulerBackend: Interrupting monitor thread 18/02/11 12:08:13 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors 18/02/11 12:08:13 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down 18/02/11 12:08:13 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices (serviceOption=None, services=List(), started=false) 18/02/11 12:08:13 INFO cluster.YarnClientSchedulerBackend: Stopped 18/02/11 12:08:13 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 18/02/11 12:08:13 INFO memory.MemoryStore: MemoryStore cleared 18/02/11 12:08:13 INFO storage.BlockManager: BlockManager stopped 18/02/11 12:08:13 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 18/02/11 12:08:13 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 18/02/11 12:08:13 INFO spark.SparkContext: Successfully stopped SparkContext 18/02/11 12:08:13 INFO util.ShutdownHookManager: Shutdown hook called 18/02/11 12:08:13 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-3b26c620-946b-4efe-a60b-d101e32ec42a

相關推薦

hadoop-yarn叢集中，通過shell指令碼自動化提交spark任務

Set() 18/02/11 12:07:32 INFO yarn.Client: Submitting application application_1518316627470_0003 to ResourceManager 18/02/11 12:07:32 INFO impl.YarnClientIm

在Oozie 中排程執行shell、hive 指令碼，以及通過shell指令碼執行hive/sqoop/shell指令碼的方法

  最近專案需要用到OOzie工具，可是找了好久，也沒有找到一個完整的、統一的解決和部署方案。經過努力，終於打通了其中的所有環節，解決了各種坑爹的問題。   首先，就專案需求做一個簡單的介紹：      專案需要從mysql中匯入資料到hive進行離線計算後，再導回到mys

通過shell指令碼做定時任務，用sqoop匯出資料

一、shell指令碼#!/bin/bash  #coding=UTF-8  #date_year=$(date +%Y)  #echo $date_year  source /etc/profile  for((i=3;i<=8;i++));  do  /opt/sqo

linux中通過shell指令碼複製檔案，並用時間戳命名

其中/var/log/ha/ha.log該檔案是被複制檔案，var/log/ha/log/$currentTimeStamp時間戳生成的檔案複製的檔案，/var/log/ha/cp_info.log是實行本shell的操作記錄，同時本currentTimeStamp時間戳是長

Linux Centos7通過shell指令碼來監控mysql的執行狀態

vim checkmysql.sh #!/bin/sh #create by mingongge at 2018-10-10 port=`netstat -lnt|grep 3306|wc -l` if  [ $post -ne 1 ] ;then    now

[轉載] Linux export變數的生命週期和shell的生命週期相同，即shell指令碼執行完畢後，相應的export變數便失效了

說說shell指令碼中的export 和 source，bash 小弟剛剛接觸linux，對linux上的很多東西都比較陌生，所以寫一寫部落格，當做自己工作的總結和技術的積累吧，也是鞭策自己不斷努力的去學習。 今天之所以起這個標題，把export，source ，bash這三個命令放在一起講

如何通過shell指令碼操作MongoDB

  通過shell指令碼對MongoDB進行自動化操作 執行寫好的 ./show.sh 指令碼 發現能夠建立mongo連結 #!/bin/sh mongo WordPress --eval "show collections;db.posts.find().limit(10);

通過Shell指令碼生成IPA

通過Shell指令碼生成IPA，本文內容有待實際操作後驗證。 方案一：xcodebuild -archivePath # 清理xcodebuild clean -workspace xgj.xcworkspace -scheme xgj # 構建xcodebuild -archivePath xgj.

在k8s叢集中，利用prometheus的jmx_exporter進行tomcat的JVM效能監控，並用grafana作前端展示

查找了很多文件，沒有完全達到我要求的， 於是，作了一定的調整，成現在這樣。 操作步驟如下： 一，準備好兩個檔案。 jmx_prometheus_javaagent-0.3.1.jar jmx_exporter.yml jmx_exporter.yml作最簡單配置如下： --- rules

Spark：在Spark叢集中，叢集的節點個數、RDD分割槽個數、​cpu核心個數三者與並行度的關係？？

梳理一下Spark中關於併發度涉及的幾個概念File，Block，Split，Task，Partition，RDD以及節點數、Executor數、core數目的關係。 輸入可能以多個檔案的形式儲存在HDFS上，每個File都包含了很多塊，稱為Block。 當Spark

通過shell指令碼呼叫儲存過程

shell指令碼 #/bin/sh # 獲取資料庫密碼，根據實際情況，需要修改 export_pgpasswd() { if [ -f /usr/local/svconfig/server

ossutil命令上傳檔案到阿里雲oss詳解，使用Shell指令碼實現資料的自動上傳下載Bucket物件儲存

安裝ossutil yum -y install wget wget http://docs-aliyun.cn-hangzhou.oss.aliyun-inc.com/assets/attach/50452/cn_zh/1524643963683/ossutil64 mv

Mongodb通過shell指令碼增量備份到HDFS

用Mongodb自帶的Mongoexport，通過shell指令碼的方式進行增量備份到HDFS上 1.思路介紹： Mongodb安裝目錄下bin目錄裡有兩個工具：mongoexport和mongodump這兩個工具可以實現備份mongodb的資料。用法也類似，區別這裡不做說明，自行搜尋。mongoexport

通過shell指令碼呼叫oracle儲存過程並加入定時任務

例項 檔名：add_tab_partitions.sh #!/bin/sh username='waep' password='czty_waep' sqlplus -S ${username}/

Linux shell相關基礎，建立shell指令碼掃描判斷區域網主機作業系統型別

nmap：linux使用nmap 對區域網掃描，檢視arp快取表就可以知道局域內主機ip和mac了，方法如下：nmap引數-sP -PU -sS ，-sS半開放掃描，不用執行完TCP三次握手，執行得很快。nmap -A掃描詳細資訊，速度慢。掃描之後檢視arp快取表獲取區域網主

通過shell指令碼構建某服務 start/stop/restart 命令

對於某些常用操作可以通過 shell 指令碼構建成為一個命令，通過傳入不同的引數實現不同的操作。以下以 python 構建的 web 伺服器的 start/stop/restart 命令為例建立 sh

linux -- shell指令碼呼叫其他指令碼，向shell指令碼傳遞引數

Shell 指令碼也可以看做是一個一個的函式，將命令進行了封裝。 看看自己的例子： outTop.sh 是最頂層的。【負責呼叫其他的shell 指令碼】out1.sh 是第2層。【被outTop.sh呼叫】outInsert是第2層。【被outTop.sh呼叫】檔案 out

通過shell指令碼 動態批量匯出mysql 資料

一般mysql匯出資料會使用select into outfile  匯出，這種方式只有在命令列模式下才能把檔案生成 到本地，如果通過儲存過程執行，檔案會生成到mysql資料庫所在的主機上。 這個時候如果你需要動態配置查詢條件來匯出檔案，就會變得很麻煩。這種情況下，可以使用

linux 通過shell指令碼在本地執行遠端機器命令

在本機上想通過shell指令碼執行遠端機器上的命令，其實很簡單，只要在ssh 的後面加上<< remotessh      【命令】 remotessh 就可以了，下面貼出簡單的例子 #!/bin/bash ssh [email protected]

簡單的通過shell指令碼接收引數進行資料庫操作

建表sql語句： SET FOREIGN_KEY_CHECKS=0; -- ---------------------------- -- Table structure for `cent_su