0031-如何在CDH啟用Kerberos的情況下安裝及使用Sentry(一)
溫馨提示:要看高清無碼套圖,請使用手機打開並單擊圖片放大查看。
1.文檔編寫目的
本文檔主要講述如何在啟用Kerberos的CDH集群中安裝配置及使用Sentry。
- 內容概述
1.如何安裝Sentry服務
2.Hive/Impala/Hue/HDFS服務如何與Sentry集成
3.Sentry測試
- 測試環境
1.操作系統為CentOS6.5
2.CM和CDH版本為5.11.1
3.采用root用戶操作
- 前置條件
1.CDH集群運行正常
2.集群已啟用Kerberos且正常使用
2.Sentry安裝
1.在MySQL中創建sentry數據庫
建表語句:
create database sentry default character set utf8;
CREATE USER ‘sentry‘@‘%‘ IDENTIFIED BY ‘password‘;
GRANT ALL PRIVILEGES ON sentry. * TO ‘sentry‘@‘%‘;
FLUSH PRIVILEGES;
命令行操作:
[root@ip-172-31-6-148 527-hive-HIVEMETASTORE]# mysql -uroot -p Enter password: ... mysql> create database sentry default character set utf8; Query OK, 1 row affected (0.00 sec) mysql> CREATE USER ‘sentry‘@‘%‘ IDENTIFIED BY ‘password‘; Query OK, 0 rows affected (0.00 sec) mysql> GRANT ALL PRIVILEGES ON sentry. * TO ‘sentry‘@‘%‘; Query OK, 0 rows affected (0.00 sec) mysql> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) mysql>
2.進入Cloudera Manager控制臺點擊“添加服務”
3.進入服務添加界面,選擇Sentry服務,點擊“繼續”
4.選擇Sentry Server及Gateway的安裝節點,點擊“繼續”
5.輸入Sentry服務的數據庫信息,點擊測試,測試通過,點擊“繼續”
6.等待服務安裝成功,點擊“繼續”
7.點擊“完成”,Sentry服務至此安裝完成。
3.Sentry配置
3.1Hive配置
1.配置Hive使用Sentry服務
2.關閉Hive的用戶模擬功能
3.2Impala配置
配置Impala使用Sentry
3.3Hue配置
配置Hue使用Sentry
3.4HDFS配置
配置HDFS開啟ACLs與Sentry權限同步
完成以上配置後,回到Cloudera Manager主頁,部署客戶端配置並重啟相關服務。
4.Sentry測試
4.1創建hive超級用戶
使用hive用戶登錄Kerberos,操作如下
[root@ip-172-31-6-148 196-hive-HIVEMETASTORE]# kinit -kt hive.keytab hive/[email protected]
[root@ip-172-31-6-148 196-hive-HIVEMETASTORE]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hive/[email protected]
Valid starting Expires Service principal
09/07/17 02:26:04 09/08/17 02:26:04 krbtgt/[email protected]
renew until 09/12/17 02:26:04
[root@ip-172-31-6-148 196-hive-HIVEMETASTORE]#
1.使用beeline連接HiveServer2
[root@ip-172-31-6-148 196-hive-HIVEMETASTORE]# beeline
Beeline version 1.1.0-cdh5.12.1 by Apache Hive
beeline> !connect jdbc:hive2://localhost:10000/;principal=hive/[email protected]
scan complete in 3ms
Connecting to jdbc:hive2://localhost:10000/;principal=hive/[email protected]
Connected to: Apache Hive (version 1.1.0-cdh5.12.1)
Driver: Hive JDBC (version 1.1.0-cdh5.12.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://localhost:10000/>
2.創建admin角色
0: jdbc:hive2://localhost:10000/> create role admin;
...
INFO : OK
No rows affected (0.37 seconds)
0: jdbc:hive2://localhost:10000/>
3.為admin角色賦予管理員權限
0: jdbc:hive2://localhost:10000> grant all on server server1 to role admin;
...
INFO : OK
No rows affected (0.221 seconds)
0: jdbc:hive2://localhost:10000>
4.將admin角色授權給hive用戶組
0: jdbc:hive2://localhost:10000> grant role admin to group hive;
...
INFO : OK
No rows affected (0.162 seconds)
0: jdbc:hive2://localhost:10000>
以上操作創建了一個admin角色:
admin : 具有管理員權限,可以讀寫所有數據庫,並授權給hive組(對應操作系統的組)
4.2創建test表
使用hive用戶登錄Kerberos,通過beeline登錄HiveServer2,創建test表,並插入測試數據
0: jdbc:hive2://localhost:10000> create tabletest (s1 string, s2 string) row format delimited fields terminated by ‘,‘;
...
INFO : OK
No rows affected (0.592 seconds)
0: jdbc:hive2://localhost:10000> insert into test values(‘a‘,‘b‘),(‘1‘,‘2‘);
...
INFO : OK
No rows affected (20.123 seconds)
0: jdbc:hive2://localhost:10000>
4.3創建測試角色並將角色授權給用戶組
創建兩個角色:
read:只能讀default庫test表,並授權給fayson用戶組
write:只能寫default庫test表,並授權給user_w用戶組
註意:集群所有節點必須存在fayson和user_w用戶,用戶默認用戶組與用戶名一致,賦權是針對用戶組而不是針對用戶。
[root@ip-172-31-6-148 cdh-shell-master]# id fayson
uid=501(fayson) gid=501(fayson) groups=501(fayson)
[root@ip-172-31-6-148 cdh-shell-master]# useradd user_w
[root@ip-172-31-6-148 cdh-shell-master]# id user_w
uid=502(user_w) gid=502(user_w) groups=502(user_w)
[root@ip-172-31-6-148 cdh-shell-master]#
1.使用hive用戶創建read和write角色,並授權read角色對test表的select權限,write角色對test表的insert權限
0: jdbc:hive2://localhost:10000> create role read;
...
INFO : OK
No rows affected (0.094 seconds)
0: jdbc:hive2://localhost:10000> grant select on table test torole read;
...
INFO : OK
No rows affected (0.1 seconds)
0: jdbc:hive2://localhost:10000> create role write;
...
INFO : OK
No rows affected (0.105 seconds)
0: jdbc:hive2://localhost:10000> grant insert on table test to role write;
...
INFO : OK
No rows affected (0.112 seconds)
0: jdbc:hive2://localhost:10000>
2.將read角色授權給fayson用戶組,write角色授權給user_w用戶組
0: jdbc:hive2://localhost:10000> grant role read to group fayson;
...
INFO : OK
No rows affected (0.187 seconds)
0: jdbc:hive2://localhost:10000> grant role write to group user_w;
...
INFO : OK
No rows affected (0.101 seconds)
0: jdbc:hive2://localhost:10000>
3.使用kadmin創建fayson和user_w用戶
[root@ip-172-31-6-148 ~]# kadmin.local
Authenticating as principal hive/[email protected] with password.
kadmin.local: addprinc [email protected]
WARNING: no policy specified for [email protected]; defaulting to no policy
Enter password for principal "[email protected]":
Re-enter password for principal "[email protected]":
Principal "[email protected]" created.
kadmin.local: addprinc [email protected]
WARNING: no policy specified for [email protected]; defaulting to no policy
Enter password for principal "[email protected]":
Re-enter password for principal "[email protected]":
Principal "[email protected]" created.
kadmin.local:
4.4beeline驗證
1.使用fayson用戶登錄Kerberos
[root@ip-172-31-6-148 ~]# kdestroy
[root@ip-172-31-6-148 ~]# kinit fayson
Password for [email protected]:
[root@ip-172-31-6-148 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: [email protected]
Valid starting Expires Service principal
09/07/17 02:48:35 09/08/17 02:48:35 krbtgt/[email protected]
renew until 09/14/17 02:48:35
[root@ip-172-31-6-148 ~]#
通過beeline連接HiveServer2進行驗證
[root@ip-172-31-6-148 ~]# beeline
Beeline version 1.1.0-cdh5.12.1 by Apache Hive
beeline> !connect jdbc:hive2://localhost:10000/;principal=hive/[email protected]
...
0: jdbc:hive2://localhost:10000/> show tables;
...
INFO : OK
+-----------+--+
| tab_name |
+-----------+--+
| test |
+-----------+--+
1 row selected (0.403 seconds)
0: jdbc:hive2://localhost:10000/> select * from test;
...
INFO : OK
+----------+----------+--+
| test.s1 | test.s2 |
+----------+----------+--+
| a | b |
| 1 | 2 |
| 111 | 222 |
| a | b |
| 1 | 2 |
| 333 | 5555 |
| eeee | dddd |
+----------+----------+--+
7 rows selected (0.282 seconds)
0: jdbc:hive2://localhost:10000/> insert into test values("2", "222");
Error: Error while compiling statement: FAILED: SemanticException No valid privileges
User fayson does not have privileges for QUERY
The required privileges: Server=server1->Db=default->Table=test->action=insert; (state=42000,code=40000)
0: jdbc:hive2://localhost:10000/>
執行Hive的MapReduce任務
0: jdbc:hive2://localhost:10000/> select count(*) from test;
...
INFO : OK
+------+--+
| _c0 |
+------+--+
| 7 |
+------+--+
1 row selected (30.688 seconds)
0: jdbc:hive2://localhost:10000/>
2.使用user_w用戶登錄Kerberos
[root@ip-172-31-6-148 ~]# kdestroy
[root@ip-172-31-6-148 ~]# kinit user_w
Password for [email protected]:
[root@ip-172-31-6-148 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: [email protected]
Valid starting Expires Service principal
09/07/17 03:01:56 09/08/17 03:01:56 krbtgt/[email protected]
renew until 09/14/17 03:01:56
[root@ip-172-31-6-148 ~]#
使用beeline登錄HiveServer2驗證
[root@ip-172-31-6-148 ~]# beeline
Beeline version 1.1.0-cdh5.12.1 by Apache Hive
beeline> !connect jdbc:hive2://localhost:10000/;principal=hive/[email protected]
...
0: jdbc:hive2://localhost:10000/> show tables;
...
INFO : OK
+-----------+--+
| tab_name |
+-----------+--+
| test |
+-----------+--+
1 row selected (0.343 seconds)
0: jdbc:hive2://localhost:10000/> select * from test;
Error: Error while compiling statement: FAILED: SemanticException No valid privileges
User user_w does not have privileges for QUERY
The required privileges: Server=server1->Db=default->Table=test->Column=s1->action=select; (state=42000,code=40000)
0: jdbc:hive2://localhost:10000/> insert into test values("2", "333");
...
INFO : OK
No rows affected (19.379 seconds)
0: jdbc:hive2://localhost:10000/>
驗證總結:
fayson用戶所屬組為fayson擁有test表讀權限,只能對test表進行select和count操作不能進行insert操作;
user_w用戶所屬組為user_w擁有test表寫權限,只能對test表進行insert操作不能進行select操作;
4.5HDFS驗證
1.使用fayson用戶登錄Kerberos,進行如下操作
使用HDFS命令對/user/hive/warehouse/test進行cat、ls、put等操作
[root@ip-172-31-6-148 ~]# hadoop fs -ls /user/hive/warehouse
ls: Permission denied: user=fayson, access=READ_EXECUTE, inode="/user/hive/warehouse":hive:hive:drwxrwx--x
[root@ip-172-31-6-148 ~]# hadoop fs -ls /user/hive/warehouse/test
Found 5 items
-rwxrwx--x+ 3 hive hive 8 2017-09-05 12:52 /user/hive/warehouse/test/000000_0
-rwxrwx--x+ 3 hive hive 8 2017-09-05 13:44 /user/hive/warehouse/test/000000_0_copy_1
-rwxrwx--x+ 3 hive hive 8 2017-09-07 02:36 /user/hive/warehouse/test/000000_0_copy_2
-rwxrwx--x+ 3 hive hive 6 2017-09-07 03:04 /user/hive/warehouse/test/000000_0_copy_3
-rwxrwx--x+ 3 hive hive 19 2017-09-05 13:01 /user/hive/warehouse/test/test.txt
[root@ip-172-31-6-148 ~]# hadoop fs -cat /user/hive/warehouse/test/test.txt
333,5555
eeee,dddd
[root@ip-172-31-6-148 ~]# hadoop fs -rm /user/hive/warehouse/test/test.txt
rm: Failed to move to trash: hdfs://ip-172-31-6-148.fayson.com:8020/user/hive/warehouse/test/test.txt: Permission denied: user=fayson, access=WRITE, inode="/user/hive/warehouse/test":hive:hive:drwxrwx--x
[root@ip-172-31-6-148 ~]# hadoop fs -put a.txt /user/hive/warehouse/test/
put: Permission denied: user=fayson, access=WRITE, inode="/user/hive/warehouse/test":hive:hive:drwxrwx--x
[root@ip-172-31-6-148 ~]#
2.使用user_w用戶登錄Kerberos,進行如下操作
[root@ip-172-31-6-148 ~]# kdestroy
[root@ip-172-31-6-148 ~]# kinit user_w
Password for [email protected]:
[root@ip-172-31-6-148 ~]# hadoop fs -ls /user/hive/warehouse
ls: Permission denied: user=user_w, access=READ_EXECUTE, inode="/user/hive/warehouse":hive:hive:drwxrwx--x
[root@ip-172-31-6-148 ~]# hadoop fs -ls /user/hive/warehouse/test
ls: Permission denied: user=user_w, access=READ_EXECUTE, inode="/user/hive/warehouse/test":hive:hive:drwxrwx--x
[root@ip-172-31-6-148 ~]# hadoop fs -cat /user/hive/warehouse/test/test.txt
cat: Permission denied: user=user_w, access=READ, inode="/user/hive/warehouse/test/test.txt":hive:hive:-rwxrwx--x
[root@ip-172-31-6-148 ~]# hadoop fs -rm /user/hive/warehouse/test/test.txt
17/09/07 03:21:21 INFO fs.TrashPolicyDefault: Moved: ‘hdfs://ip-172-31-6-148.fayson.com:8020/user/hive/warehouse/test/test.txt‘ to trash at: hdfs://ip-172-31-6-148.fayson.com:8020/user/user_w/.Trash/Current/user/hive/warehouse/test/test.txt
[root@ip-172-31-6-148 ~]# hadoop fs -put a.txt /user/hive/warehouse/test/
[root@ip-172-31-6-148 ~]#
fayson用戶所屬組為fayson,擁有test表的讀權限,可以對test表的數據目錄(/user/hive/warehouse/test)瀏覽及查看目錄下文件內容,不能刪除/test/目錄下文件,也不能向test目錄put文件。
user_w用戶所屬組為user_w,擁有test表的write權限,可以對test表的數據目錄put文件及刪除數據文件操作,但不能瀏覽及查看目錄下的文件內容。
說明Sentry實現了HDFS的ACL同步。
4.6Hue驗證
1.使用Hue的管理員,添加Hue的測試用戶fayson和user_w
2.使用fayson用戶登錄Hue,驗證read權限
可以查看test表數據
可以進行Count操作
不能Insert操作
File Browser瀏覽
不能瀏覽父目錄/user/hive/warehouse
可以瀏覽test表的數據目錄/user/hive/warehouse/test
可以查看/user/hive/warehouse/test目錄下文件內容
不能修改/user/hive/warehouse/test目錄下數據文件
3.使用user_w用戶登錄Hue,驗證write權限
不可以查看test表
不可以Count操作
可以向test表插入數據
FileBrowser操作
不可以瀏覽父目錄/user/hive/warehouse
不可以瀏覽test表的數據目錄/user/hive/warehouse/test
fayson和user_w用戶均能通過hue界面看到test表,擁有read角色的fayson用戶組能對test進行select和count操作,並且能通過File Browser瀏覽和查看test表的數據目錄/user/hive/warehouse/test。擁有write角色的user_w用戶組只能對test表進行insert操作,但不能通過File Browser瀏覽和查看test表的數據目錄/user/hive/warehouse/test。說明Sentry在命令行的操作和授權在Hue中依舊有效。
4.7Impala驗證
1.使用fayson用戶登錄Kerberos
[root@ip-172-31-6-148 ~]# kinit fayson
Password for [email protected]:
[root@ip-172-31-6-148 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: [email protected]
Valid starting Expires Service principal
09/07/17 06:36:05 09/08/17 06:36:05 krbtgt/[email protected]
renew until 09/14/17 06:36:05
[root@ip-172-31-6-148 ~]# impala-shell
Starting Impala Shell without Kerberos authentication
...
Connected to ip-172-31-9-33.fayson.com:21000
Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build 5131a031f4aa38c1e50c430373c55ca53e0517b9)
[ip-172-31-9-33.fayson.com:21000] > show tables;
Query: show tables
+------+
| name |
+------+
| test |
+------+
Fetched 1 row(s) in 0.02s
[ip-172-31-9-33.fayson.com:21000] > select * from test;
...
+--------+----------+
| s1 | s2 |
+--------+----------+
| testaa | testbbb |
| 111 | 222 |
| 222 | 2323 |
| 2 | 333 |
| a | b |
| 1 | 2 |
| 1 | test |
| 2 | fayson |
| 3 | zhangsan |
| a | b |
| 1 | 2 |
+--------+----------+
Fetched 11 row(s) in 0.19s
[ip-172-31-9-33.fayson.com:21000] > select count(*) from test;
...
+----------+
| count(*) |
+----------+
| 11 |
+----------+
Fetched 1 row(s) in 0.14s
[ip-172-31-9-33.fayson.com:21000] > insert into test values(‘test44‘,‘test55‘);
Query: insert into test values(‘test44‘,‘test55‘)
Query submitted at: 2017-09-07 06:37:00 (Coordinator: http://ip-172-31-9-33.fayson.com:25000)
ERROR: AuthorizationException: User ‘[email protected]‘ does not have privileges to execute ‘INSERT‘ on: default.test
[ip-172-31-9-33.fayson.com:21000] >
2.使用user_w用戶登錄Kerberos
登錄Impala-shell進行操作
[root@ip-172-31-6-148 ~]# impala-shell
...
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.9.0-cdh5.12.1 (5131a03) built on Thu Aug 24 09:27:32 PDT 2017)
***********************************************************************************
[Not connected] > connect ip-172-31-9-33.fayson.com:21000;
Connected to ip-172-31-9-33.fayson.com:21000
Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build 5131a031f4aa38c1e50c430373c55ca53e0517b9)
[ip-172-31-9-33.fayson.com:21000] > show tables;
Query: show tables
+------+
| name |
+------+
| test |
+------+
Fetched 1 row(s) in 0.29s
[ip-172-31-9-33.fayson.com:21000] > select * from test;
Query: select * from test
Query submitted at: 2017-09-07 06:31:23 (Coordinator: http://ip-172-31-9-33.fayson.com:25000)
ERROR: AuthorizationException: User ‘[email protected]‘ does not have privileges to execute ‘SELECT‘ on: default.test
[ip-172-31-9-33.fayson.com:21000] > insert into test values(‘222‘,‘2323‘);
Query: insert into test values(‘222‘,‘2323‘)
Query submitted at: 2017-09-07 06:32:07 (Coordinator: http://ip-172-31-9-33.fayson.com:25000)
Query progress can be monitored at: http://ip-172-31-9-33.fayson.com:25000/query_plan?query_id=ec406e621c7534c7:6bcbbd5300000000
Modified 1 row(s) in 0.63s
[ip-172-31-9-33.fayson.com:21000] >
驗證總結:
Impala與Sentry集成後可以使用Sentry來進行權限管理,擁有read角色的fayson用戶組只能對test表進行select和count操作不能插入數據,擁有write角色的user_w
用戶組只能對test表插入數據不能進行select和count操作。說明Sentry實現了Hive權限與Impala的同步。
醉酒鞭名馬,少年多浮誇! 嶺南浣溪沙,嘔吐酒肆下!摯友不肯放,數據玩的花!
溫馨提示:要看高清無碼套圖,請使用手機打開並單擊圖片放大查看。
推薦關註Hadoop實操,第一時間,分享更多Hadoop幹貨,歡迎轉發和分享。
原創文章,歡迎轉載,轉載請註明:轉載自微信公眾號Hadoop實操
0031-如何在CDH啟用Kerberos的情況下安裝及使用Sentry(一)