編譯 flink 1.12.0
阿新 • • 發佈：2020-12-22
參考文件： 編譯Flink 1.9.0 https://www.cnblogs.com/Springmoon-venn/p/11512136.html
編譯環境： ubuntu 18.05
Java : 1.8.0_271
官網下載原始碼包：https://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-1.12.0/flink-1.12.0-src.tgz
解壓
tar -zxvf flink-1.12.0-src.tgz ls flink-1.12.0* flink-1.12.0-src.tgz flink-1.12.0: build-target flink-java flink-state-backends docs flink-jepsen flink-streaming-java flink-annotations flink-kubernetes flink-streaming-scala flink-clients flink-libraries flink-table flink-connectors flink-mesos flink-tests flink-container flink-metrics flink-test-utils-parent flink-contrib flink-ml-parent flink-walkthroughs flink-core flink-optimizer flink-yarn flink-dist flink-parent.iml flink-yarn-tests flink-docs flink-python LICENSE flink-end-to-end-tests flink-queryable-state licenses flink-examples flink-quickstart NOTICE flink-external-resources flink-runtime pom.xml flink-filesystems flink-runtime-web README.md flink-formats flink-scala target flink-fs-tests flink-scala-shell tools
進入目錄，編譯
mvn clean package -DskipTests
注：flink 1.11 後就不再與 hadoop 版本繫結，不需要編譯特定 hadoop 版本 的 shaded 包
flink 1.11.0 release notes : https://flink.apache.org/news/2020/07/06/release-1.11.0.html
[FLINK-11086] Flink now supports Hadoop versions aboveHadoop 3.0.0. Note that the Flink project does not provide any updated “flink-shaded-hadoop-*” jars.
Users need to provide Hadoop dependencies through theHADOOP_CLASSPATHenvironment variable (recommended) or the lib/ folder.
編譯結果：
[INFO] Reactor Summary: [INFO] [INFO] Flink : Tools : Force Shading ...................... SUCCESS [ 13.192 s] [INFO] Flink : ............................................ SUCCESS [ 6.506 s] [INFO] Flink : Annotations ................................ SUCCESS [ 2.445 s] [INFO] Flink : Test utils : ............................... SUCCESS [ 0.126 s] [INFO] Flink : Test utils : Junit ......................... SUCCESS [ 1.486 s] [INFO] Flink : Metrics : .................................. SUCCESS [ 0.087 s] [INFO] Flink : Metrics : Core ............................. SUCCESS [ 0.725 s] [INFO] Flink : Core ....................................... SUCCESS [ 26.911 s] [INFO] Flink : Java ....................................... SUCCESS [ 5.101 s] [INFO] Flink : Queryable state : .......................... SUCCESS [ 0.067 s] [INFO] Flink : Queryable state : Client Java .............. SUCCESS [ 0.830 s] [INFO] Flink : FileSystems : .............................. SUCCESS [ 0.063 s] [INFO] Flink : FileSystems : Hadoop FS .................... SUCCESS [ 8.505 s] [INFO] Flink : Runtime .................................... SUCCESS [01:33 min] [INFO] Flink : Scala ...................................... SUCCESS [ 27.175 s] [INFO] Flink : FileSystems : Mapr FS ...................... SUCCESS [ 0.895 s] [INFO] Flink : FileSystems : Hadoop FS shaded ............. SUCCESS [ 3.908 s] [INFO] Flink : FileSystems : S3 FS Base ................... SUCCESS [ 1.645 s] [INFO] Flink : FileSystems : S3 FS Hadoop ................. SUCCESS [ 5.757 s] [INFO] Flink : FileSystems : S3 FS Presto ................. SUCCESS [ 6.342 s] [INFO] Flink : FileSystems : Swift FS Hadoop .............. SUCCESS [ 17.588 s] [INFO] Flink : FileSystems : OSS FS ....................... SUCCESS [ 5.269 s] [INFO] Flink : FileSystems : Azure FS Hadoop .............. SUCCESS [ 9.854 s] [INFO] Flink : Optimizer .................................. SUCCESS [ 11.023 s] [INFO] Flink : Connectors : ............................... SUCCESS [ 0.088 s] [INFO] Flink : Connectors : File Sink Common .............. SUCCESS [ 0.340 s] [INFO] Flink : Streaming Java ............................. SUCCESS [ 12.088 s] [INFO] Flink : Clients .................................... SUCCESS [ 2.695 s] [INFO] Flink : Test utils : Utils ......................... SUCCESS [ 1.408 s] [INFO] Flink : Runtime web ................................ SUCCESS [11:09 min] [INFO] Flink : Examples : ................................. SUCCESS [ 0.713 s] [INFO] Flink : Examples : Batch ........................... SUCCESS [ 16.444 s] [INFO] Flink : Connectors : Hadoop compatibility .......... SUCCESS [ 12.183 s] [INFO] Flink : State backends : ........................... SUCCESS [ 0.089 s] [INFO] Flink : State backends : RocksDB ................... SUCCESS [ 2.446 s] [INFO] Flink : Tests ...................................... SUCCESS [ 30.561 s] [INFO] Flink : Streaming Scala ............................ SUCCESS [ 21.328 s] [INFO] Flink : Connectors : HCatalog ...................... SUCCESS [ 6.485 s] [INFO] Flink : Test utils : Connectors .................... SUCCESS [ 0.452 s] [INFO] Flink : Connectors : Base .......................... SUCCESS [ 0.623 s] [INFO] Flink : Connectors : Files ......................... SUCCESS [ 1.767 s] [INFO] Flink : Table : .................................... SUCCESS [ 0.070 s] [INFO] Flink : Table : Common ............................. SUCCESS [ 5.397 s] [INFO] Flink : Table : API Java ........................... SUCCESS [ 2.381 s] [INFO] Flink : Table : API Java bridge .................... SUCCESS [ 1.176 s] [INFO] Flink : Table : API Scala .......................... SUCCESS [ 7.939 s] [INFO] Flink : Table : API Scala bridge ................... SUCCESS [ 7.045 s] [INFO] Flink : Table : SQL Parser ......................... SUCCESS [ 26.022 s] [INFO] Flink : Libraries : ................................ SUCCESS [ 0.106 s] [INFO] Flink : Libraries : CEP ............................ SUCCESS [ 3.086 s] [INFO] Flink : Table : Planner ............................ SUCCESS [01:33 min] [INFO] Flink : Table : SQL Parser Hive .................... SUCCESS [ 6.621 s] [INFO] Flink : Table : Runtime Blink ...................... SUCCESS [ 11.644 s] [INFO] Flink : Table : Planner Blink ...................... SUCCESS [02:01 min] [INFO] Flink : Formats : .................................. SUCCESS [ 0.236 s] [INFO] Flink : Formats : Json ............................. SUCCESS [ 1.569 s] [INFO] Flink : Connectors : Elasticsearch base ............ SUCCESS [ 8.103 s] [INFO] Flink : Connectors : Elasticsearch 5 ............... SUCCESS [ 13.526 s] [INFO] Flink : Connectors : Elasticsearch 6 ............... SUCCESS [ 4.407 s] [INFO] Flink : Connectors : Elasticsearch 7 ............... SUCCESS [ 3.241 s] [INFO] Flink : Connectors : HBase base .................... SUCCESS [ 22.812 s] [INFO] Flink : Connectors : HBase 1.4 ..................... SUCCESS [ 4.806 s] [INFO] Flink : Connectors : HBase 2.2 ..................... SUCCESS [ 7.163 s] [INFO] Flink : Formats : Hadoop bulk ...................... SUCCESS [ 0.864 s] [INFO] Flink : Formats : Orc .............................. SUCCESS [ 1.793 s] [INFO] Flink : Formats : Orc nohive ....................... SUCCESS [ 1.579 s] [INFO] Flink : Formats : Avro ............................. SUCCESS [ 5.951 s] [INFO] Flink : Formats : Parquet .......................... SUCCESS [ 14.026 s] [INFO] Flink : Formats : Csv .............................. SUCCESS [ 1.059 s] [INFO] Flink : Connectors : Hive .......................... SUCCESS [ 12.772 s] [INFO] Flink : Connectors : JDBC .......................... SUCCESS [ 2.882 s] [INFO] Flink : Connectors : RabbitMQ ...................... SUCCESS [ 1.091 s] [INFO] Flink : Connectors : Twitter ....................... SUCCESS [ 2.206 s] [INFO] Flink : Connectors : Nifi .......................... SUCCESS [ 1.343 s] [INFO] Flink : Connectors : Cassandra ..................... SUCCESS [ 4.109 s] [INFO] Flink : Metrics : JMX .............................. SUCCESS [ 1.350 s] [INFO] Flink : Connectors : Kafka ......................... SUCCESS [ 5.280 s] [INFO] Flink : Connectors : Google PubSub ................. SUCCESS [ 3.571 s] [INFO] Flink : Connectors : Kinesis ....................... SUCCESS [ 13.685 s] [INFO] Flink : Connectors : SQL : Elasticsearch 6 ......... SUCCESS [ 5.171 s] [INFO] Flink : Connectors : SQL : Elasticsearch 7 ......... SUCCESS [ 6.483 s] [INFO] Flink : Connectors : SQL : HBase 1.4 ............... SUCCESS [ 6.591 s] [INFO] Flink : Connectors : SQL : HBase 2.2 ............... SUCCESS [ 11.668 s] [INFO] Flink : Connectors : SQL : Hive 1.2.2 .............. SUCCESS [ 5.588 s] [INFO] Flink : Connectors : SQL : Hive 2.2.0 .............. SUCCESS [ 5.818 s] [INFO] Flink : Connectors : SQL : Hive 2.3.6 .............. SUCCESS [ 5.212 s] [INFO] Flink : Connectors : SQL : Hive 3.1.2 .............. SUCCESS [ 7.460 s] [INFO] Flink : Connectors : SQL : Kafka ................... SUCCESS [ 0.957 s] [INFO] Flink : Connectors : SQL : Kinesis ................. SUCCESS [ 6.103 s] [INFO] Flink : Formats : Avro confluent registry .......... SUCCESS [ 2.766 s] [INFO] Flink : Formats : Sequence file .................... SUCCESS [ 0.298 s] [INFO] Flink : Formats : Compress ......................... SUCCESS [ 0.356 s] [INFO] Flink : Formats : SQL Orc .......................... SUCCESS [ 0.581 s] [INFO] Flink : Formats : SQL Parquet ...................... SUCCESS [ 0.633 s] [INFO] Flink : Formats : SQL Avro ......................... SUCCESS [ 1.828 s] [INFO] Flink : Formats : SQL Avro Confluent Registry ...... SUCCESS [ 3.294 s] [INFO] Flink : Examples : Streaming ....................... SUCCESS [ 10.463 s] [INFO] Flink : Examples : Table ........................... SUCCESS [ 6.642 s] [INFO] Flink : Examples : Build Helper : .................. SUCCESS [ 0.119 s] [INFO] Flink : Examples : Build Helper : Streaming Twitter SUCCESS [ 0.670 s] [INFO] Flink : Examples : Build Helper : Streaming State machine SUCCESS [ 0.526 s] [INFO] Flink : Examples : Build Helper : Streaming Google PubSub SUCCESS [ 4.241 s] [INFO] Flink : Container .................................. SUCCESS [ 0.315 s] [INFO] Flink : Queryable state : Runtime .................. SUCCESS [ 1.000 s] [INFO] Flink : Mesos ...................................... SUCCESS [ 19.073 s] [INFO] Flink : Kubernetes ................................. SUCCESS [ 6.304 s] [INFO] Flink : Yarn ....................................... SUCCESS [ 1.964 s] [INFO] Flink : Libraries : Gelly .......................... SUCCESS [ 4.599 s] [INFO] Flink : Libraries : Gelly scala .................... SUCCESS [ 12.661 s] [INFO] Flink : Libraries : Gelly Examples ................. SUCCESS [ 7.576 s] [INFO] Flink : External resources : ....................... SUCCESS [ 0.052 s] [INFO] Flink : External resources : GPU ................... SUCCESS [ 0.245 s] [INFO] Flink : Metrics : Dropwizard ....................... SUCCESS [ 0.333 s] [INFO] Flink : Metrics : Graphite ......................... SUCCESS [ 0.264 s] [INFO] Flink : Metrics : InfluxDB ......................... SUCCESS [ 1.697 s] [INFO] Flink : Metrics : Prometheus ....................... SUCCESS [ 1.057 s] [INFO] Flink : Metrics : StatsD ........................... SUCCESS [ 0.222 s] [INFO] Flink : Metrics : Datadog .......................... SUCCESS [ 0.332 s] [INFO] Flink : Metrics : Slf4j ............................ SUCCESS [ 0.183 s] [INFO] Flink : Libraries : CEP Scala ...................... SUCCESS [ 7.980 s] [INFO] Flink : Table : Uber ............................... SUCCESS [ 6.009 s] [INFO] Flink : Table : Uber Blink ......................... SUCCESS [ 6.234 s] [INFO] Flink : Python ..................................... SUCCESS [ 17.610 s] [INFO] Flink : Table : SQL Client ......................... SUCCESS [ 3.707 s] [INFO] Flink : Libraries : State processor API ............ SUCCESS [ 1.865 s] [INFO] Flink : ML : ....................................... SUCCESS [ 0.074 s] [INFO] Flink : ML : API ................................... SUCCESS [ 0.388 s] [INFO] Flink : ML : Lib ................................... SUCCESS [ 1.066 s] [INFO] Flink : ML : Uber .................................. SUCCESS [ 0.429 s] [INFO] Flink : Scala shell ................................ SUCCESS [ 8.262 s] [INFO] Flink : Dist ....................................... SUCCESS [ 41.965 s] [INFO] Flink : Yarn Tests ................................. SUCCESS [ 6.501 s] [INFO] Flink : E2E Tests : ................................ SUCCESS [ 4.415 s] [INFO] Flink : E2E Tests : CLI ............................ SUCCESS [ 0.327 s] [INFO] Flink : E2E Tests : Parent Child classloading program SUCCESS [ 0.298 s] [INFO] Flink : E2E Tests : Parent Child classloading lib-package SUCCESS [ 0.256 s] [INFO] Flink : E2E Tests : Dataset allround ............... SUCCESS [ 0.220 s] [INFO] Flink : E2E Tests : Dataset Fine-grained recovery .. SUCCESS [ 0.307 s] [INFO] Flink : E2E Tests : Datastream allround ............ SUCCESS [ 1.116 s] [INFO] Flink : E2E Tests : Batch SQL ...................... SUCCESS [ 0.258 s] [INFO] Flink : E2E Tests : Stream SQL ..................... SUCCESS [ 0.272 s] [INFO] Flink : E2E Tests : Distributed cache via blob ..... SUCCESS [ 0.161 s] [INFO] Flink : E2E Tests : High parallelism iterations .... SUCCESS [ 7.459 s] [INFO] Flink : E2E Tests : Stream stateful job upgrade .... SUCCESS [ 1.085 s] [INFO] Flink : E2E Tests : Queryable state ................ SUCCESS [ 2.125 s] [INFO] Flink : E2E Tests : Local recovery and allocation .. SUCCESS [ 0.308 s] [INFO] Flink : E2E Tests : Elasticsearch 5 ................ SUCCESS [ 6.400 s] [INFO] Flink : E2E Tests : Elasticsearch 6 ................ SUCCESS [ 4.810 s] [INFO] Flink : Quickstart : ............................... SUCCESS [ 6.150 s] [INFO] Flink : Quickstart : Java .......................... SUCCESS [ 2.474 s] [INFO] Flink : Quickstart : Scala ......................... SUCCESS [ 0.400 s] [INFO] Flink : E2E Tests : Quickstart ..................... SUCCESS [ 0.350 s] [INFO] Flink : E2E Tests : Confluent schema registry ...... SUCCESS [ 2.201 s] [INFO] Flink : E2E Tests : Stream state TTL ............... SUCCESS [ 2.968 s] [INFO] Flink : E2E Tests : SQL client ..................... SUCCESS [ 3.233 s] [INFO] Flink : E2E Tests : File sink ...................... SUCCESS [ 0.905 s] [INFO] Flink : E2E Tests : State evolution ................ SUCCESS [ 0.586 s] [INFO] Flink : E2E Tests : RocksDB state memory control ... SUCCESS [ 0.497 s] [INFO] Flink : E2E Tests : Common ......................... SUCCESS [ 1.826 s] [INFO] Flink : E2E Tests : Metrics availability ........... SUCCESS [ 0.214 s] [INFO] Flink : E2E Tests : Metrics reporter prometheus .... SUCCESS [ 0.226 s] [INFO] Flink : E2E Tests : Heavy deployment ............... SUCCESS [ 6.590 s] [INFO] Flink : E2E Tests : Connectors : Google PubSub ..... SUCCESS [ 5.169 s] [INFO] Flink : E2E Tests : Streaming Kafka base ........... SUCCESS [ 0.359 s] [INFO] Flink : E2E Tests : Streaming Kafka ................ SUCCESS [ 5.392 s] [INFO] Flink : E2E Tests : Plugins : ...................... SUCCESS [ 0.457 s] [INFO] Flink : E2E Tests : Plugins : Dummy fs ............. SUCCESS [ 0.420 s] [INFO] Flink : E2E Tests : Plugins : Another dummy fs ..... SUCCESS [ 0.277 s] [INFO] Flink : E2E Tests : TPCH ........................... SUCCESS [ 1.295 s] [INFO] Flink : E2E Tests : Streaming Kinesis .............. SUCCESS [ 11.156 s] [INFO] Flink : E2E Tests : Elasticsearch 7 ................ SUCCESS [ 4.509 s] [INFO] Flink : E2E Tests : Common Kafka ................... SUCCESS [ 1.620 s] [INFO] Flink : E2E Tests : TPCDS .......................... SUCCESS [ 4.335 s] [INFO] Flink : E2E Tests : Netty shuffle memory control ... SUCCESS [ 0.262 s] [INFO] Flink : E2E Tests : Python ......................... SUCCESS [ 6.582 s] [INFO] Flink : E2E Tests : HBase .......................... SUCCESS [ 4.213 s] [INFO] Flink : State backends : Heap spillable ............ SUCCESS [ 1.006 s] [INFO] Flink : Contrib : .................................. SUCCESS [ 0.524 s] [INFO] Flink : Contrib : Connectors : Wikiedits ........... SUCCESS [ 0.822 s] [INFO] Flink : FileSystems : Tests ........................ SUCCESS [ 1.202 s] [INFO] Flink : Docs ....................................... SUCCESS [ 2.157 s] [INFO] Flink : Walkthrough : .............................. SUCCESS [ 0.122 s] [INFO] Flink : Walkthrough : Common ....................... SUCCESS [ 1.583 s] [INFO] Flink : Walkthrough : Datastream Java .............. SUCCESS [ 0.134 s] [INFO] Flink : Walkthrough : Datastream Scala ............. SUCCESS [ 0.154 s] [INFO] Flink : Tools : CI : Java .......................... SUCCESS [ 0.694 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 30:34 min [INFO] Finished at: 2020-12-22T18:18:39+08:00 [INFO] Final Memory: 738M/2315M [INFO] ------------------------------------------------------------------------
和之前一樣 Flink : Runtime web 包是編譯的難點，因為要線上下載一些包，基本都在國外，下載慢或者根本訪問不了，這次還算比較順利，只遇到下面兩個問題
nodejs 包下載不下來
Downloading https://nodejs.org/dist/v10.9.0/node-v10.9.0-linux-x64.tar.gz to /home/venn/.m2/repository/com/github/eirslett/node/10.9.0/node-10.9.0-linux-x64.tar.gz
直接訪問 URL 下載，放到 mvn 目錄中
$ pwd /home/venn/.m2/repository/com/github/eirslett/node/10.9.0 $ ls node-10.9.0-linux-x64.tar.gz
npm 執行不動
Running 'npm ci --cache-max=0 --no-save' in /home/venn/git/flink-1.12.0/flink-runtime-web/web-dashboard
直接安裝 npm
apt-get install -y npm
然後重新編譯，就通過了，bin 包在路徑flink-1.12.0/flink-dist/target/flink-1.12.0-bin 下面
啟動個 demo 程式
歡迎關注Flink菜鳥公眾號，會不定期更新Flink（開發技術）相關的推文