技術標籤：FlinkFlink 1.11

背景：

為了更好的使用flink sql&table，系統決定從flink 從1.10 升級到 1.11，然後在測試flink example的命令時出現了這個bug

（不懂怎麼安裝的直接參考官網連結：https://ci.apache.org/projects/flink/flink-docs-release-1.11/try-flink/local_installation.html）

java.lang.IllegalStateException: No Executor found. Please make sure to export the HADOOP_CLASSPATH environment variable or have hadoop in your classpath.
 For more information refer to the "Deployment & Operations" section of the official 
 Apache Flink documentation. at org.apache.flink.yarn.cli.FallbackYarnSessionCli.isActive(FallbackYarnSessionCli.java:59) at 
 org.apache.flink.client.cli.CliFrontend.validateAndGetActiveCommandLine(CliFrontend.java:1090) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:218) at 
 org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:916) at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:992) at 
 org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) at 
 org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:992)
 

原因：

產生這個問題的原因這個，具體的意思就是從Flink 1.11開始，flink-shaded-hadoop-2-uberFlink專案不再正式支援使用發行版。建議使用者通過提供Hadoop依賴關係HADOOP_CLASSPATH（請參見上文），官方(https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/hadoop.html)針對這個問題有詳細的說明見如下：

Starting from Flink 1.11, usingflink-shaded-hadoop-2-uberreleases is not officially supported by the Flink project anymore. Users are advised to provide Hadoop dependencies throughHADOOP_CLASSPATH

(see above).

Hadoop Integration

• Providing Hadoop classes
• Running a job locally
• Usingflink-shaded-hadoop-2-uberjar for resolving dependency conflicts (legacy)

Providing Hadoop classes

In order to use Hadoop features (e.g., YARN, HDFS) it is necessary to provide Flink with the required Hadoop classes, as these are not bundled by default.

The recommended approach is adding the Hadoop classpath to Flink through theHADOOP_CLASSPATHenvironment variable.

Flink will use the environment variableHADOOP_CLASSPATHto augment the classpath that is used when starting Flink components such as the Client, JobManager, or TaskManager. Most Hadoop distributions and cloud environments will not set this variable by default so if the Hadoop classpath should be picked up by Flink the environment variable must be exported on all machines that are running Flink components.

When running on YARN, this is usually not a problem because the components running inside YARN will be started with the Hadoop classpaths, but it can happen that the Hadoop dependencies must be in the classpath when submitting a job to YARN. For this, it’s usually enough to run

export HADOOP_CLASSPATH=`hadoop classpath`

in the shell. Note thathadoopis the hadoop binary and thatclasspathis an argument that will make it print the configured Hadoop classpath. The classpath returned byhadoop classpathalso includes the Hadoop configuration directories.

If you are manually assembling theHADOOP_CLASSPATHvariable, we recommend adding the Hadoop configuration directories as well.

Running a job locally

To run a job locally as one JVM process using the mini cluster, the required hadoop dependencies have to be explicitly added to the classpath of the started JVM process.

To run an application using Maven (also from IDE as a Maven project), the required Hadoop dependencies can be added as provided to the pom.xml, e.g.:

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.8.3</version>
    <scope>provided</scope>
</dependency>

This way it should work both in local and cluster mode where the provided dependencies are added elsewhere as described before.

To run or debug an application in IntelliJ Idea the provided dependencies can be included to the class path in the “Run|Edit Configurations” window.

Usingflink-shaded-hadoop-2-uberjar for resolving dependency conflicts (legacy)

Warning:Starting from Flink 1.11, usingflink-shaded-hadoop-2-uberreleases is not officially supported by the Flink project anymore. Users are advised to provide Hadoop dependencies throughHADOOP_CLASSPATH(see above).

The Flink project used to (until Flink 1.10) release Hadoop distributions for specific versions, that relocate or exclude several dependencies to reduce the risk of dependency clashes. These can be found in theAdditional Componentssection of the download page. For these versions it is sufficient to download the correspondingPre-bundled Hadoopcomponent and putting it into the/libdirectory of the Flink distribution.

If the used Hadoop version is not listed on the download page (possibly due to being a Vendor-specific version), then it is necessary to buildflink-shadedagainst this version. You can find the source code for this project in theAdditional Componentssection of the download page.

NoteIf you want to buildflink-shadedagainst a vendor specific Hadoop version, you first have to configure the vendor-specific maven repository in your local maven setup as describedhere.

Run the following command to build and installflink-shadedagainst your desired Hadoop version (e.g., for version2.6.5-custom):

mvn clean install -Dhadoop.version=2.6.5-custom

After this step is complete, put theflink-shaded-hadoop-2-uberjar into the/libdirectory of the Flink distribution.

解決思路：

參考下面連結：https://issues.apache.org/jira/browse/FLINK-19126

注意啦注意啦，在這裡我做了個ZZ的騷操作！！！！

在這裡別自己想太多，一開始我理解這個命令是 export HADOOP_CLASSPATH=hadoop的檔案路徑，就這樣運行了

export HADOOP_CLASSPATH=/usr/local/hadoop

發現一點作用的都沒起，其實直接執行這個命令就可以了不需要改任何東西,馬上就不報錯了

 export HADOOP_CLASSPATH=`hadoop classpath`

後續：

其實hadoop classpath 命令就是把有關於hadoop的環境全部加載出來

hadoop classpath