Spark standlone driver on cluster 使用者許可權問題
1) 原因
Spark standlone 模式下面driver 提交到cluster,application 使用者變成了spark 部署環境使用者,這樣會帶來兩個問題:
a) spark master web無法知道那個使用者提交的任務;
b) spark許可權問題無法管理,比如讀取Hive、HBase、HDFS資料的許可權問題;
本文提出一種簡單的修改方法,讓Spark standlone driver on cluster 模型下面讀取資料許可權問題可以解決。
2) 程式碼流程
3) 修改方法
原理上面需要從client.main agrs獲取driver提交端的user name;然後經過driverdesciption傳送給master和worker,需要注意的是driverdesciption裡面只能給command envi新增欄位;經過worker傳送給driver ,但是其中經過了陣列轉換,轉換過程比較複雜,需要修改的程式碼較多。
本文直接在sparkconf裡面設定driverproxyuser屬性:
conf.set("driverproxyuser",user.name) |
需要修改的程式碼:
a) sparkcontext 的sparkuser變數; web顯示的使用者;
b) SparkDeploySchedulerBackend 的ApplicationDescription的system user name,該app user name可以用於獲取hive、hbase等表的許可權;
4) 修改程式碼
sparkcontext.scala
diff /SparkCode/spark-1.4.0/core/src/main/scala/org/apache/spark/SparkContext.scala core/src/main/scala/org/apache/spark/SparkContext.scala 301,315c301 < // val sparkUser =Utils.getCurrentUserName() < //val sparkUser= config.get("driverproxyuser",Utils.getCurrentUserName()) < val sparkUser = { < val conf=config.clone < < if(conf.contains("driverproxyuser")) { < logInfo("conf contains driverproxyuser configered by user ?" + conf.contains("driverproxyuser")) < val proxyuser=conf.get("driverproxyuser") < System.setProperty("HADOOP_USER_NAME", proxyuser ) < // System.setProperty("user.name", proxyuser) < proxyuser < }else{ < Utils.getCurrentUserName() < } < } --- > val sparkUser = Utils.getCurrentUserName() 495c481 < executorEnvs("SPARK_USER") = sparkUser //sp Added JARarkUser --- > executorEnvs("SPARK_USER") = sparkUser 521d506 < |
SparkDeploySchedulerBackend.scala
diff /SparkCode/spark-1.4.0/./core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala ./core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala 86,94d85 < < //add by Ricky < val username=System.getProperty("user.name") < if(conf.contains("driverproxyuser")){ < val proxyuser=conf.get("driverproxyuser") < logInfo("set ApplicationDescription user is : "+proxyuser) < System.setProperty("HADOOP_USER_NAME",proxyuser) < System.setProperty("user.name",proxyuser) < } 97,101d87 < < < < logInfo("ApplicationDescription user is :"+appDesc.user) < System.setProperty("user.name",username) // added by Ricky |