分布式下Session一致性架構舉例
阿新 • • 發佈：2018-03-11
創建用戶 dep 存儲 download dt.jar jdk1.8 tex service rop
一、問題及方案
見這篇文章：分布式下Session一致性問題
二、分布式環境搭建：
系統環境
[root@centos7 ~]# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
[root@centos7 ~]# 2.1 安裝jdk
# 下載jdk-8u141-linux-x64.tar.gz # 創建目錄 mkdir -p /opt/java # 解壓 tar -xzvf jdk-8u141-linux-x64.tar.gz -C /opt/java # 創建鏈接 ln -s /opt/java/jdk1.8.0_141 /usr/local/jdk # 設置變量 /etc/profile末尾添加 export JAVA_HOME=/usr/local/jdk export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar # 使變量生效 source /etc/profile export PATH=$JAVA_HOME/bin:$PATH # 驗證 java -version
2.2 安裝tomcat
安裝3個tomcat並設置其端口號
# 下載apache-tomcat-8.5.16.tar.gz # 創建目錄 mkdir -p /opt/tomcat # 解壓 tar -xzvf apache-tomcat-8.5.16.tar.gz -C /opt/tomcat # 清空ROOT目錄 rm -rf /opt/tomcat/apache-tomcat-8.5.16/webapps/ROOT/* # 復制3個tomcat cd /opt/tomcat cp -a apache-tomcat-8.5.16 tomcat1 cp -a apache-tomcat-8.5.16 tomcat2 cp -a apache-tomcat-8.5.16 tomcat3 # 修改端口號 sed -i '22s/8005/8'"$i"'05/' $file sed -i '69s/8080/8'"$i"'80/' $file sed -i '116s/8009/8'"$i"'09/' $file
修改如下3個端口號
用如下腳本修改：a.sh
#!/bin/sh for i in {1..3} do file=/opt/tomcat/tomcat"$i"/conf/server.xml sed -i '22s/8005/8'"$i"'05/' $file sed -i '69s/8080/8'"$i"'80/' $file sed -i '116s/8009/8'"$i"'09/' $file #sed -i '148s#appBase=".*"#appBase="/data/webapps"#' $file done
執行腳本修改端口
sh a.sh
執行後，3個tomcat的端口號分別改為
8180,8280,8380啟動並驗證這3個tomcat
# 分別向3個tomcat寫入一個測試文件a.txt
for i in {1..3};do echo 8"$i"80>/opt/tomcat/tomcat"$i"/webapps/ROOT/a.txt;done
# 啟動
for i in {1..3};do /opt/tomcat/tomcat"$i"/bin/startup.sh;done
# 驗證
for i in {1..3};do curl 127.0.0.1:8"$i"80/a.txt;done
2.3 安裝nginx
# 下載nginx-1.13.9.tar.gz
mkdir -p /opt/nginx/nginx-1.3.13.9
# 創建用戶
useradd nginx -s /sbin/nologin -M
# 安裝pcre openssl
yum install pcre pcre-devel openssl openssl-devel -y
# 解壓、編譯安裝
tar -xzvf nginx-1.13.9.tar.gz
cd nginx-1.13.9
./configure --user=nginx --group=nginx --prefix=/opt/nginx/nginx-1.13.9 --with-http_stub_status_module --with-http_ssl_module
make
make install
# 創建鏈接
ln -s /opt/nginx/nginx-1.13.9 /usr/local/nginx
# 設置環境變量
echo 'export PATH=/usr/local/nginx/sbin:$PATH' >> /etc/profile
source /etc/profilenginx常用命令
# 啟動
nginx
# 檢查配置文件
nginx -t
# 平滑啟動
nginx -s reload
# 檢查
netstat -tunlp | grep nginx遇到的問題：
nginx: [emerg] getpwnam("nginx") failed
未創建nginx用戶配置nginx
/usr/local/nginx/conf/nginx.conf
worker_processes 1;
events {
worker_connections 1024;
}
# http最外層模塊
http {
# 全局變量參數
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
upstream web_pool {
server 127.0.0.1:8180 weight=1;
server 127.0.0.1:8280 weight=1;
server 127.0.0.1:8380 weight=2;
}
# server相當於虛擬站點
server {
listen 80;
server_name localhost;
location / {
proxy_pass http://web_pool;
index index.html index.htm;
}
}
}測試一下
for i in {1..9};do curl 127.0.0.1/a.txt;done
2.4 安裝redis
wget http://download.redis.io/releases/redis-4.0.8.tar.gz
mkdir -p /opt/redis
tar -xzvf redis-4.0.8.tar.gz -C /opt/redis/
ln -s /opt/redis/redis-4.0.8 /usr/local/redis
cd /usr/local/redis
make && make install
# 配置讓其他機器也可以訪問
vi /usr/local/redis/redis.conf
#bind 127.0.0.1
bind 0.0.0.0操作redis
# 查看redis版本
redis-cli -v
# 啟動redis
redis-server redis.conf > /tmp/redis.log 2>&1 &
# or
nohup redis-server redis.conf &
# 監控redis
redis-cli -h 192.168.234.130 -p 6379 monitor
# 關閉redis
redis-cli -h 127.0.0.1 -p 6379 shutdown客戶端
# 連接
redis-cli -h 192.168.5.220
# 查看
keys *
# 清空
flushall三、問題重現
3.1 web應用
java-web-login
只實現登錄功能
/login 登錄頁面，已登錄跳轉到/index
/logout 退出
/index 首頁，未登錄跳轉到/login將這個應用部署到3個tomcat上
重啟tomcat
for i in {1..3};do /opt/tomcat/tomcat"$i"/bin/shutdown.sh;done
for i in {1..3};do /opt/tomcat/tomcat"$i"/bin/startup.sh;done
3.2 問題描述
登錄界面是8280端口，輸入用戶名密碼點擊登錄，由於nginx配置的是基於輪詢的算法進行分發，8380端口的服務器處理的登錄，因此session創建於8380端口，而8280處於非登錄狀態。
四、解決實現
4.1 session sticky
修改nginx的分發算法改為基於ip
upstream web_pool {
# 默認是輪詢
ip_hash;
server 127.0.0.1:8180 weight=1;
server 127.0.0.1:8280 weight=1;
server 127.0.0.1:8380 weight=2;
}修改後重啟nginx
nginx -t
nginx -s reload4.2 session replication
http://tomcat.apache.org/tomcat-8.5-doc/cluster-howto.html
tomcat配置server.xml
<!--修改每個tomcat下的conf/server.xml-->
<!--其中Receiver這個結點的端口分別配置為4001,4002,4003-->
<!--channelSendOptions-->
<!--8表示異步發送 2表示確認發送 4表示同步發送 10表示同步+確認-->
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="8">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true" />
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<!-- 228.0.0.4是主播地址-->
<Membership className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.4"
port="45564"
frequency="500"
dropTime="3000" />
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto"
port="4001"
autoBind="100"
selectorTimeout="5000"
maxThreads="6" />
<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender" />
</Sender>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector" />
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor" />
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter="" />
<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve" />
<Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
tempDir="/tmp/war-temp/"
deployDir="/tmp/war-deploy/"
watchDir="/tmp/war-listen/"
watchEnabled="false" />
<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener" />
</Cluster>tomcat的配置web.xml
# 修改每個tomcat下的conf/web.xml不起作用
# 一定要修改每個應用的web.xml，在<web-app>這個節點下添加
<distributable />4.3 session數據集中存儲
數據存儲到redis，使用spring-data-redis
見應用java-web-login
把web_spring-redis.xml重命名為web.xml，部署到tomcat即可，得先啟動redis參照“安裝redis”這一節。
spring-redis.xml
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">
<!-- 引入配置文件 -->
<bean id="propertyConfigurer"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<list>
<value>classpath*:redis.properties</value>
</list>
</property>
</bean>
<bean id="redisHttpSessionConfiguration"
class="org.springframework.session.data.redis.config.annotation.web.http.RedisHttpSessionConfiguration">
<property name="maxInactiveIntervalInSeconds" value="600"/>
</bean>
<bean id="jedisPoolConfig" class="redis.clients.jedis.JedisPoolConfig">
<property name="maxTotal" value="${redis.pool.maxTotal}" />
<property name="maxIdle" value="${redis.pool.maxIdle}" />
</bean>
<bean id="jedisConnectionFactory"
class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory"
destroy-method="destroy">
<property name="hostName" value="${redis.hostname}" />
<property name="port" value="${redis.port}" />
<property name="timeout" value="${redis.timeout}" />
<property name="usePool" value="true" />
<property name="poolConfig" ref="jedisPoolConfig" />
</bean>
</beans>註意：
- Spring版本的選擇 >=4.0.3
pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.yysue</groupId>
<artifactId>java-web-login</artifactId>
<packaging>war</packaging>
<name>java-web-login</name>
<version>0.0.1-SNAPSHOT</version>
<properties>
<java.version>1.8</java.version>
<spring.version>4.0.3.RELEASE</spring.version>
</properties>
<!-- 依賴 -->
<dependencies>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.0.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>javax.servlet.jsp</groupId>
<artifactId>jsp-api</artifactId>
<version>2.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>jstl</artifactId>
<version>1.2</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-web</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-oxm</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-tx</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context-support</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-aop</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-redis</artifactId>
<version>1.5.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.session</groupId>
<artifactId>spring-session-data-redis</artifactId>
<version>1.2.2.RELEASE</version>
</dependency>
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>2.8.1</version>
</dependency>
</dependencies>
<build>
<!-- 插件 -->
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>${java.version}</source>
<target>${java.version}</target>
</configuration>
</plugin>
</plugins>
</build>
</project>分布式下Session一致性架構舉例