SPARK 原始碼分析技術分享 (帶bilibili視訊)

【本站點正在持續更新中…2018-12-05…】

• SPARK 1.6.0-cdh5.15.0
• Hadoop 2.6.0-cdh5.15.0
• spark-scala-maven
• 微信(技術交流) : thinktothings
• SPARK 原始碼分析技術分享(視訊彙總套裝視訊): https://www.bilibili.com/video/av37442139/
• SPARK 原始碼分析技術分享(視訊彙總線上看):https://blog.csdn.net/thinktothings/article/details/84726769
• SPARK 原始碼分析技術分享 (github) : https://github.com/opensourceteams/spark-scala-maven

RDD依賴 Dependency

NarrowDependency

• OneToOneDependency

  • bilibili 視訊: https://www.bilibili.com/video/av37442139/?p=1
  • 詳細說明文件： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/OneToOneDenpendency.md

• RangeDependency

  • bilibili 視訊: https://www.bilibili.com/video/av37442139/?p=2
  • 詳細說明文件： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/RangeDependency.md

• PruneDependency filter

  • bilibili 視訊: https://www.bilibili.com/video/av37442139/?p=3
  • 詳細說明文件： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/PruneDependency.md

• PruneDependency RangePartitioner

  • bilibili 視訊: https://www.bilibili.com/video/av37442139/?p=4
  • 詳細說明文件： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/PruneDependencyRangePartitioner.md

ShuffleDependency

• bilibili 視訊: https://www.bilibili.com/video/av37442139/?p=5
• 詳細說明文件：https://github.com/opensourceteams/spark-scala-maven/blob/master/md/ShuffleDependency.md

SparkContext 分析

• SparkContext原理分析(Youtube視訊): https://youtu.be/euIuutjAB4I
• SparkContext原始碼分析(Youtube視訊): https://youtu.be/tUH7QnCcwgg
• SparkContext原始碼分析(文件詳解）： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/SparkContext.md

Spark 通訊原理分析

• Spark通訊原理分析(Youtube視訊): https://youtu.be/3vUVwbEGf1E
• Spark通訊原理分析（文件詳解）： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/OutBoxAndInBox.md
• bilibili : https://www.bilibili.com/video/av37442199/

Spark Master啟動原始碼分析

• Spark Master啟動原始碼分析(Youtube視訊): https://youtu.be/74q1nddoaiY​
• Spark Master啟動原始碼分析(Bilibili視訊): https://www.bilibili.com/video/av37442271/
• Master啟動原始碼分析詳細說明文件： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/StartMaster.md

• Spark Master資源排程–worker向master註冊(Youtube視訊): https://youtu.be/74q1nddoaiY​
• Spark Master資源排程–worker向master註冊(Bilibili視訊): https://www.bilibili.com/video/av37442280/
• Spark Master資源除錯–worker向master註冊(文件詳解)： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/MasterScheduler_workerRegisterMaster.md

• Spark Master資源排程–SparkContext向所有master註冊(Youtube視訊): https://youtu.be/AXxCnCc5Mh0​
• Spark Master資源排程–SparkContext向所有master註冊(Bilibili視訊): https://www.bilibili.com/video/av37442295/
• Spark Master資源排程–SparkContext向所有master註冊(文件詳解): https://github.com/opensourceteams/spark-scala-maven/blob/master/md/MasterScheduler_SparkContextRegisterMaster.md

Spark Worker啟動原始碼分析

• Spark Worker 啟動原始碼分析(Youtube視訊): https://youtu.be/ll_Ae6rP7II​​
• Spark Worker 啟動原始碼分析(Bilibili視訊): https://www.bilibili.com/video/av37442247/
• Spark Worker 啟動原始碼分析(文件詳解): https://github.com/opensourceteams/spark-scala-maven/blob/master/md/StartWorker.md

Spark Executor啟動原始碼分析

• Spark Executor啟動原始碼分析(Youtube視訊): https://youtu.be/1qg4UMPV3pQ
• Spark Executor啟動原始碼分析(Bilibili視訊): https://www.bilibili.com/video/av37442311/
• Spark Executor啟動原始碼分析(文件詳解)： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/CoarseGrainedExecutorBackend_start.md

Spark 觸發Job提交

• Spark 觸發Job提交(youtube視訊) : https://youtu.be/X49RIqz2AjM
• Spark 觸發Job提交(bilibili視訊) : https://www.bilibili.com/video/av37445008/
• Spark 觸發Job提交(文件詳解)：https://github.com/opensourceteams/spark-scala-maven/blob/master/md/jobSubmitTrigger.md

Spark DAG排程器事件迴圈處理器

• Spark DAG排程器事件迴圈處理器(Youtube視訊) : https://youtu.be/fT-dpf0KFOA
• Spark DAG排程器事件迴圈處理器(Bilibili視訊) : https://www.bilibili.com/video/av37445034/
• Spark DAG排程器事件迴圈處理器(文件詳解)：https://github.com/opensourceteams/spark-scala-maven/blob/master/md/DAGSchedulerEventProcessLoop.md

Spark DAGScheduler FinalStage處理(Stage劃分)

• Spark FinalStage處理(Stage劃分)(Youtube視訊) : https://youtu.be/yFJugOV0Fak
• Spark FinalStage處理(Stage劃分)(Bilibili視訊) : https://www.bilibili.com/video/av37445057/
• Spark FinalStage處理(Stage劃分)(文件詳解)：https://github.com/opensourceteams/spark-scala-maven/blob/master/md/FinalStage.md

Spark DAGScheduler Stage提交

• Spark Stage提交(Youtube視訊) : https://youtu.be/NI8-_X6mbl4
• Spark Stage提交(Bilibili視訊) : https://www.bilibili.com/video/av37445077/
• Spark Stage提交(文件詳解)：https://github.com/opensourceteams/spark-scala-maven/blob/master/md/SubmitStage.md

Spark DAGScheduler把stage轉換成TaskSet的過程

• Spark DAGScheduler把stage轉換成TaskSet的過程(Bilibili視訊) https://www.bilibili.com/video/av37442139/?p=18
• Spark DAGScheduler把stage轉換成TaskSet的過程(文件詳解)：https://github.com/opensourceteams/spark-scala-maven/blob/master/md/DAGScheduler_stageToTaskSet.md

======================================================================================

Youtub 視訊

• HadoopRdd原始碼分析-讀取本地檔案需求分析-01
• HadoopRDD原始碼分析-檔案拆分partition劃分-02
• HadoopRdd原始碼分析 本地檔案讀取原始碼分析 03
• HadoopRdd原始碼分析 本地檔案讀取原始碼分析 04

Spark 遠端除錯

• 詳細說明文件： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/SparkRemoteDebug.md

相關連結

• HadoopRDD原始碼分析說明文件詳細