hive 整合sentry
阿新 • • 發佈：2018-11-12
環境
apache-hive-2.3.3-bin
apache-sentry-2.1.0-bin
sentry是目前最新的版本，支援hive的最高版本為2.3.3，hive版本如果高於2.3.3，會出一些版本相容問題[親測]
hive快速安裝
wget http://mirrors.shu.edu.cn/apache/hive/hive-2.3.3/apache-hive-2.3.3-bin.tar.gz
tar -zxvf apache-hive-2.3.3-bin.tar.gz
配置hive-site.xml
mv hive-default.xml.template hive-site.xml mkdir -p /home/xiaobin/soft/apache-hive-2.3.3-bin/tmpdir vi hive-site.xml <property> <name>system:java.io.tmpdir</name> <value>/home/xiaobin/soft/apache-hive-2.3.3-bin/tmpdir</value> </property> <property> <name>system:user.name</name> <value>master</value> </property> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://192.168.1.115/hive2?createDatabaseIfNotExist=true&useUnicode=true</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>123456</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> </property>
copy mysql-connector驅動
cp mysql-connector-java.jar apache-hive-2.3.3-bin/lib/
建立元資料資料庫
mysql> create database hive2;
Query OK, 1 row affected (0.01 sec)
初始化元資料
schematool -dbType mysql -initSchema
sentry安裝
下載
http://sentry.apache.org/general/downloads.html wget http://apache.01link.hk/sentry/2.1.0/apache-sentry-2.1.0-bin.tar.gz tar -zxvf apache-sentry-2.1.0-bin.tar.gz
config
cp sentry-site.xml.service.example sentry-site.xml vi sentry-site.xml <property> <name>sentry.hive.server</name> <value>server1</value> </property> <property> <name>sentry.verify.schema.version</name> <value>true</value> </property> <property> <name>sentry.service.allow.connect</name> <value>hive,impala,hue,hdfs</value> <description>comma separated list of users - List of users that are allowed to connect to the service (eg Hive, Impala) </description> </property> <property> <name>sentry.store.jdbc.url</name> <value>jdbc:mysql://localhost:3306/sentry</value> <description>JDBC connection URL for the backed DB</description> </property> <property> <name>sentry.store.jdbc.user</name> <value>sentry</value> <description>The username of the user that connects to the Sentry database</description> </property> <property> <name>sentry.store.jdbc.password</name> <value>sentry</value> <description>Sentry password for backend JDBC user </description> </property> <property> <name>sentry.service.server.keytab</name> <value></value> <description>Keytab for service principal</description> </property> <property> <name>sentry.service.server.rpcport</name> <value>8038</value> <description> TCP port number for service</description> </property> <property> <name>sentry.service.server.rpcaddress</name> <value>0.0.0.0</value> <description> TCP interface for service to bind to</description> </property> <property> <name>sentry.store.jdbc.driver</name> <value>com.mysql.jdbc.Driver</value> <description>Backend JDBC driver - org.apache.derby.jdbc.EmbeddedDriver (only when dbtype = derby) JDBC Driver class for the backed DB</description> </property> <property> <name>sentry.service.admin.group</name> <value>hive,impala,hue,hdfs</value> <description>Comma separates list of groups. List of groups allowed to make policy updates</description> </property> <property> <name>sentry.store.group.mapping</name> <value>org.apache.sentry.provider.common.HadoopGroupMappingService</value> <description> Group mapping class for Sentry service. org.apache.sentry.provider.file.LocalGroupMapping service can be used for local group mapping. </description> </property> <property> <name>sentry.store.group.mapping.resource</name> <value> </value> <description> Policy file for group mapping. Policy file path for local group mapping, when sentry.store.group.mapping is set to LocalGroupMapping Service class.</description> </property> <property> <name>sentry.service.security.mode</name> <value>none</value> <description>Options: kerberos, none. Authentication mode for Sentry service. Currently supports Kerberos and trusted mode </description> </property> <property> <name>sentry.service.server.principal</name> <value> </value> <description>Service Kerberos principal</description> </property> <property> <name>sentry.service.web.enable</name> <value>true</value> <description>Enable web service</description> </property> <property> <name>sentry.service.web.authentication.type</name> <value>NONE</value> <description>Options: kerberos, NONE. Authentication mode for Sentry web service.</description> </property> <property> <name>sentry.service.web.authentication.kerberos.keytab</name> <value></value> <description>Keytab for web service principal</description> </property> <property> <name>sentry.service.web.authentication.kerberos.principal</name> <value></value> <description>Web service Kerberos principal</description> </property> <property> <name>sentry.service.web.authentication.allow.connect.users</name> <value></value> <description>comma separated list of users - List of users that are allowed to connect to the web service (eg Hive, Impala) </description> </property>
建立sentry元資料資料庫
Create Database sentry;
Create User sentry Identified By 'sentry';
Grant All On sentry.* To [email protected]'localhost' Identified By 'sentry';
Grant All On sentry.* To [email protected]'%' Identified By 'sentry';
flush privileges;
複製mysql-connector驅動
cp mysql-connector-java.jar apache-sentry-2.1.0-bin/lib/
初始化元資料
sentry --command schema-tool --conffile apache-sentry-2.1.0-bin/conf/sentry-site.xml --dbType mysql --initSchema
啟動service
./sentry --command service --conffile apache-sentry-2.1.0-bin/conf/sentry-site.xml
檢視是否啟動成功
netstat -anpl|grep 8038
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:8038 0.0.0.0:* LISTEN 11950/java
hive整合sentry
copy sentry 客戶端配置檔案
cp apache-sentry-2.1.0-bin/conf/sentry-site.xml.hive-client.example apache-hive-2.3.3-bin/conf/
cd apache-hive-2.3.3-bin/conf/
mv sentry-site.xml.hive-client.example sentry-site.xml
配置$HIVE_HOME/conf/sentry-site.xml
<property>
<name>sentry.service.security.mode</name>
<value>none</value>
<description>Options: kerberos, none. Authentication mode for Sentry service. Currently supports Kerberos and trusted mode </description>
</property>
<property>
<name>sentry.service.client.server.rpc-addresses</name>
<value>localhost</value>
<description> TCP address of the sentry store server</description>
</property>
<property>
<name>sentry.service.client.server.rpc-port</name>
<value>8038</value>
<description>Port # of the sentry store server</description>
</property>
<property>
<name>sentry.service.client.server.rpc-connection-timeout</name>
<value>200000</value>
<description>Client timeout default(200000) RPC connection timeout in milisecs</description>
</property>
<property>
<name>sentry.metastore.service.users</name>
<value>hive</value>
<description>
Comma separated list of users
List of service users (eg hive, impala) to bypass
the Sentry metastore authorization. These
services handle the metadata authorization
on their side.
</description>
</property>
<!--
Some common client properties same as file
based provider
-->
<property>
<name>sentry.hive.provider</name>
<value>org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider</value>
<description> Deprecated name: hive.sentry.provider. Group mapping which should be used at client side</description>
</property>
<property>
<name>sentry.hive.server</name>
<value>server1</value>
<description> Deprecated name: hive.sentry.server. Defaut: HS2. Hive Server2 Server identifier like "server1"</description>
</property>
<property>
<name>sentry.hive.failure.hooks</name>
<value> </value>
<description>Deprecated Name: hive.sentry.failure.hooks</description>
</property>
<property>
<name>sentry.hive.testing.mode</name>
<value>true</value>
</property>
<property>
<name>sentry.hive.provider.backend</name>
<value>org.apache.sentry.provider.db.SimpleDBProviderBackend</value>
<description> Options: {org.apache.sentry.provider.db.SimpleDBProviderBackend, org.apache.sentry.provider.file.SimpleFileProviderBackend}
Privilege provider to be used, we support file based or db based
</description>
</property>
vi hive-site.xml
<property>
<name>hive.metastore.pre.event.listeners</name>
<value>org.apache.sentry.binding.metastore.MetastoreAuthzBinding</value>
</property>
<property>
<name>hive.metastore.event.listeners</name>
<value>org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener</value>
</property>
<property>
<name>hive.server2.enable.impersonation</name>
<value>true</value>
</property>
<property>
<name>hive.security.authorization.task.factory</name>
<value>org.apache.sentry.binding.hive.SentryHiveAuthorizationTaskFactoryImpl</value>
</property>
<property>
<name>hive.server2.session.hook</name>
<value>org.apache.sentry.binding.hive.HiveAuthzBindingSessionHook</value>
</property>
<property>
<name>hive.sentry.conf.url</name>
<value>file:///home/xiaobin/soft/apache-hive-2.3.3-bin/conf/sentry-site.xml</value>
</property>
copy sentry jars
cp apache-sentry-2.1.0-bin/lib/sentry-*.jar apache-hive-2.3.3-bin/lib/
cp apache-sentry-2.1.0-bin/lib/shiro-* apache-hive-2.3.3-bin/lib/
啟動hiveserver2
hiveserver2 --hiveconf hive.root.logger=INFO,console
檢視hiveserver2監聽埠
netstat -anpl|grep 10000
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:10000 0.0.0.0:* LISTEN 12231/java