1.安裝jkd1.8(這裏不再描述)

2.直接在終端輸入pip install pyspark(官網提供的最簡單的一種安裝方式)

過程如下:

Collecting pyspark
  Downloading https://files.pythonhosted.org/packages/ee/2f/709df6e8dc00624689aa0a11c7a4c06061a7d00037e370584b9f011df44c/pyspark-2.3.1.tar.gz (211.9MB)
    100% |████████████████████████████████| 211.9MB 8.3kB/s 
Requirement already satisfied: py4j
 
==0.10.7 in ./anaconda3/lib/python3.6/site-packages (from pyspark)
Building wheels for collected packages: pyspark
  Running setup.py bdist_wheel for pyspark ... done
  Stored in directory: /home/tan/.cache/pip/wheels/37/48/54/f1b63f0dbb729e20c92f1bbcf1c53c03b300e0b93ca1781526
Successfully built pyspark
Installing collected packages: pyspark
Successfully installed pyspark
 
-2.3.1

安裝完成後, 終端輸入pyspark,啟動pyspark出錯......

tan@tan-Precision-Tower-3620:~$ pyspark
JAVA_HOME is not set

解決方法:

找到pyspark的安裝路徑

tan@tan-Precision-Tower-3620:~$ pip install pyspark
Requirement already satisfied: pyspark in ./anaconda3/lib/python3.6/site-packages
Requirement already satisfied: py4j==0.10.7 in
 
 ./anaconda3/lib/python3.6/site-packages (from pyspark)

找到路徑後,在load-spark-env.sh文件中加上jdk的安裝路徑即可

export JAVA_HOME=/home/tan/jdk1.8.0_181

保存之後, 再次在終端輸入pyspark, 成功啟動pyspark

tan@tan-Precision-Tower-3620:~$ pyspark
Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 18:10:19) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
2018-07-29 12:37:48 WARN  Utils:66 - Your hostname, tan-Precision-Tower-3620 resolves to a loopback address: 127.0.1.1; using 192.168.0.100 instead (on interface enp0s31f6)
2018-07-29 12:37:48 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2018-07-29 12:37:48 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  ‘_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.3.1
      /_/

Using Python version 3.6.4 (default, Jan 16 2018 18:10:19)
SparkSession available as ‘spark‘.
>>> 

完結

ubuntu下pyspark的安裝