技術標籤：Flinkflink編譯器

一、1.11.0版本及以前

以前的方式是先編譯flink-shaded-hadoop這個包，將hadoop和hive指定你對應生產的版本編譯出flink-shaded-hadoop-2-uber_xxx包，然後將這個包放在lib的目錄下，flink啟動任務的時候去lib載入。

想用這種方式可以參考兩個連結：

• https://blog.csdn.net/weixin_44628586/article/details/107106547

• https://blog.csdn.net/guiyifei/article/details/109325980#comments_14400773

• https://www.cnblogs.com/Springmoon-venn/p/11512136.html

• 1.11官網：https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/hadoop.html#using-flink-shaded-hadoop-2-uber-jar-for-resolving-dependency-conflicts-legacy

flink-shade官網原始碼地址：https://github.com/apache/flink-shaded

二、1.11.0版本以後

Flink官方為了讓Flink變得Hadoop Free，現在能支援hadoop2和hadoop3，同時可以指定不同的Hadoop環境。

為了達到這一目標，通過設定export HADOOP_CLASSPATH=**hadoop classpath**即可，不用編譯flink-shaded包。
重點編譯好的Flink的jar裡面是沒有包含Hadoop和Hive的程式碼。當Flink任務啟動的時候，JM和TM都是通過HADOOP_CLASSPATH環境變數獲取Hadoop的相關變數。

剛開始小菜雞以為是**hadoop classpath只是某個隨便寫的某個路徑，後面多虧了渣渣瑞普及小白知識，``裡面是命令，之前記得後面給忘了，所以hadoop classpath**是個命令，執行完之後會看到hadoop所依賴的環境變數：

[

[email protected] ~]$ hadoop classpath
/etc/hadoop/conf:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hadoop/libexec/…/…/hadoop/lib/:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hadoop/libexec/…/…/hadoop/.//:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hadoop/libexec/…/…/hadoop-hdfs/./:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hadoop/libexec/…/…/hadoop-hdfs/lib/:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hadoop/libexec/…/…/hadoop-hdfs/.//:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hadoop/libexec/…/…/hadoop-yarn/lib/:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hadoop/libexec/…/…/hadoop-yarn/.//*

具體來處可以參考：

• 社群的信箱：http://apache-flink.147419.n8.nabble.com/flink-shaded-hadoop-2-uber-td9345.html

• 1.12官網的依賴Yarn的準備以及Hadoop的版本支援都有提及：
  https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/yarn.html#preparation
  https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/yarn.html#supported-hadoop-versions

• JIRA：https://issues.apache.org/jira/browse/FLINK-11086

• 1.11版本官網Hadoop整合：
  https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/hadoop.html#providing-hadoop-classes

這個連結裡面有這麼一段話就可以說明

Flink will use the environment variable HADOOP_CLASSPATH to augment the classpath that is used when starting Flink components such as the Client, JobManager, or TaskManager. Most Hadoop distributions and cloud environments will not set this variable by default so if the Hadoop classpath should be picked up by Flink the environment variable must be exported on all machines that are running Flink components.
Flink將使用環境變數HADOOP CLASSPATH來擴充套件啟動Flink元件(如客戶機、JobManager或TaskManager)時使用的類路徑。大多數Hadoop發行版和雲環境在預設情況下不會設定這個變數，因此，如果應該由Flink獲取Hadoop類路徑，則必須在執行Flink元件的所有機器上匯出環境變數。