ab 灰度上线方案

最新推荐文章于 2023-08-07 17:47:32 发布
最新推荐文章于 2023-08-07 17:47:32 发布
阅读量1w 收藏 8
点赞数 5
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/luwfls/article/details/82350164
版权

后台服务AB部署

标签（空格分隔）： 运维

清单

├── apache-maven-3.5.2-bin.tar.gz
├── cronolog-1.6.2.tar.gz
├── gb-tomcat-cluster
│   ├── bin
│   ├── conf
│   ├── mode.last
│   ├── tomcat-customer-a
│   ├── tomcat-customer-b
│   ├── tomcat-seller-a
│   ├── tomcat-seller-b
│   ├── tomcat-statics-a
│   ├── tomcat-statics-b
│   ├── tomcat-wechat-a
│   └── tomcat-wechat-b
├── jdk-8u144-linux-x64.tar.gz
├── nginx-conf
│   ├── gb-conf
│   └── ngxFlowControl
└── nginx-install.tar.gz

服务器配置

  • 统一JDK版本
  • 统一maven版本

 # 安装JDK
    tar -zxvf jdk-8u144-linux-x64.tar.gz
    mv jdk1.8.0_144 /usr/local/jdk1.8.0_144
 # 安装maven
    tar -zxvf apache-maven-3.5.2-bin.tar.gz
    mv apache-maven-3.5.2 /usr/local/apache-maven-3.5.2
 # 安装cronolog
    tar -zxvf cronolog-1.6.2.tar.gz
    ./configure
    make && make install
 # 配置PATH
    vim /etc/profile
    PATH=$PATH:/usr/loacl/jad:/usr/loacl/jdk1.8.0_144/bin:/usr/local/apache-maven-3.5.2/bin
 # 复制maven本地仓库（加快构建速度，第一次从中央仓库、私服抓取比较慢）
 scp -r -P39876 root@172.16.100.63:/root/.m2/repository ./

Tomcat 相关

清单

├── bin
│   ├── gb-build
│   └── gb-server
├── code
│   └── greatbear-framework
├── conf
│   ├── customer
│   ├── seller
│   ├── statics
│   └── wechat
├── mode.last
├── tomcat-customer-a
│   ├── bin
│   ├── conf
│   ├── lib
│   ├── LICENSE
│   ├── logs
│   ├── NOTICE
│   ├── RELEASE-NOTES
│   ├── RUNNING.txt
│   ├── temp
│   ├── webapps
│   └── work
├── tomcat-customer-b
│   ├── bin
│   ├── conf
│   ├── lib
│   ├── LICENSE
│   ├── logs
│   ├── NOTICE
│   ├── RELEASE-NOTES
│   ├── RUNNING.txt
│   ├── temp
│   ├── webapps
│   └── work
├── tomcat-seller-a
│   ├── bin
│   ├── conf
│   ├── lib
│   ├── LICENSE
│   ├── logs
│   ├── NOTICE
│   ├── RELEASE-NOTES
│   ├── RUNNING.txt
│   ├── temp
│   ├── webapps
│   └── work
├── tomcat-seller-b
│   ├── bin
│   ├── conf
│   ├── lib
│   ├── LICENSE
│   ├── logs
│   ├── NOTICE
│   ├── RELEASE-NOTES
│   ├── RUNNING.txt
│   ├── temp
│   ├── webapps
│   └── work
├── tomcat-statics-a
│   ├── bin
│   ├── conf
│   ├── lib
│   ├── LICENSE
│   ├── logs
│   ├── NOTICE
│   ├── RELEASE-NOTES
│   ├── RUNNING.txt
│   ├── temp
│   ├── webapps
│   └── work
├── tomcat-statics-b
│   ├── bin
│   ├── conf
│   ├── lib
│   ├── LICENSE
│   ├── logs
│   ├── NOTICE
│   ├── RELEASE-NOTES
│   ├── RUNNING.txt
│   ├── temp
│   ├── webapps
│   └── work
├── tomcat-wechat-a
│   ├── bin
│   ├── conf
│   ├── lib
│   ├── LICENSE
│   ├── logs
│   ├── NOTICE
│   ├── RELEASE-NOTES
│   ├── RUNNING.txt
│   ├── temp
│   ├── webapps
│   └── work
├── tomcat-wechat-b
│   ├── bin
│   ├── conf
│   ├── lib
│   ├── LICENSE
│   ├── logs
│   ├── NOTICE
│   ├── RELEASE-NOTES
│   ├── RUNNING.txt
│   ├── temp
│   ├── webapps
│   └── work
├── work-a
└── work-b

server.xml 修改

<?xml version='1.0' encoding='utf-8'?>
<Server port="-1" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JasperListener" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

  <GlobalNamingResources>
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>

  <Service name="Catalina">

<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
            maxThreads="1500" minSpareThreads="30" maxIdleTime="60000"/>
    <Connector  executor="tomcatThreadPool"
                port="8070" protocol="org.apache.coyote.http11.Http11NioProtocol"
                maxHttpHeaderSize="8192"
                acceptCount="800"
                compression="off"
                compressionMinSize="2048"
                enableLookups="false"
                disableUploadTimeout="false"
                connectionTimeout="20000"
                redirectPort="8443"
                URIEncoding="UTF-8"/>

    <Engine name="Catalina" defaultHost="localhost">

      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase"/>
      </Realm>

      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">

        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />
        <Valve className="org.apache.catalina.valves.RemoteIpValve"  
               remoteIpHeader="X-Forwarded-For"  
               protocolHeader="X-Forwarded-Proto"  
               protocolHeaderHttpsValue="https"/>
        <Context path="/greatbear-customer" reloadable="true" docBase="/data/customer/work/greatbear-customer"></Context>
      </Host>
    </Engine>
  </Service>
</Server>
  • 关闭 SHUTDOWN 端口（8005）（避免端口冲突）
  • 关闭AJP端口（apache 使用的协议，我们用不到）（避免端口冲突）
  • 统一Tomcat线程数，队列长度，请求长度限制
  • 可灵活配置的JVM 内存配置
  • 灵活配置 tomcat 数量
  • 统一多tomcat日志到同一个文件，方便查找，方便管理。
  • 共享一个catlina.sh，方便配置修改：比如日志切割等

脚本情况

  • 构建脚本
    • 可选择tag（默认最新的5个tag）
    • 可选择构建环境（dev|test|prod）
    • 可选择部署方案（a|b）
#! /bin/sh
# chkconfig: 2345 96 14
GB_CLUSTER_PATH=/data/gb-tomcat-cluster
GIT_CODE_PATH=$GB_CLUSTER_PATH/code
PROJECT_NAME=greatbear-framework
GIT_REPOSITORY=ssh://git@172.16.117.224:39876/GreatBear/greatbear-framework.git

if [ ! -e "$GIT_CODE_PATH/$PROJECT_NAME/.git" ]
then
    cd $GIT_CODE_PATH
    git clone $GIT_REPOSITORY
fi

cd $GIT_CODE_PATH/$PROJECT_NAME

GIT_TAG_LIST=$(git describe --tags `git rev-list --tags --max-count=5`)
echo "" 
echo "请选择要发布的Tag,输入数字" 
select tag in ${GIT_TAG_LIST[@]}
do
    git fetch --all
    git checkout $tag
    git clean -df
    break
done

ENV_LIST=(
    "prod"
    "test"
    "dev"
)

echo "请选择构建环境"

select env in ${ENV_LIST[@]}
do
    if [ -n "$env" ]; then
            echo 'building $env ...'
        mvn clean package -Penv-$env -Dmaven.test.skip=true
        break
    fi
    echo "请选择正确的环境"
done


echo "构建成功"
echo "请选择要发布的模式, 上一次发布的模式为 : " `cat $GB_CLUSTER_PATH/mode.last`

MODE_LIST=(
    "a"
    "b"
)
select mode in ${MODE_LIST[@]}
do
    if [ -n "$mode" ]; then 
        read -r -p "请确定发布的模式: $mode? [y/n] " input
        if [ $input = "y" ]; then
            rm -fr $GB_CLUSTER_PATH/work-$mode/*
            cp -fr $GIT_CODE_PATH/$PROJECT_NAME/*/target/greatbear*  $GB_CLUSTER_PATH/work-$mode/
            echo "$mode" > $GB_CLUSTER_PATH/mode.last
        fi
            echo '发布成功，请重启服务 参见 service gb-server [start|stop|restart] '
            exit 1
        break
    fi
    echo "请输入数字"
done
  • 服务管理脚本
    • 统一管理所有服务
    • 可选择运行的服务(seller|customer|statics|wechat)
    • 可选择（start|stop|restart）
    • 可选择运行模式（a|b）

#! /bin/sh
#Tomcat init script for linux
#
# chkconfig: 2345 96 14
# description: tomcat server init script
# Source Function Library

CRONO_LOG_PATH=`which cronolog`

if [ !"$CRONO_LOG_PATH" = "/usr/local/sbin/cronolog" ]; then
    echo "请安装日志分割工具 cronolog"
fi

MODE_LIST=(
        "a" 
        "b" 
)

GB_SERVICE=(
    "seller"
    "customer"
    "statics"
    "wechat"
)

JAVA_OPTS='-server -Xms2G -Xmx2G -Xmn512m -Xss256k'
DEPLOY_SERVICE=''
DEPLOY_MODE=''

echo '请选择要发布的项目'
select service in ${GB_SERVICE[@]}
do
    case $service in 
        seller | customer )
            JAVA_OPTS='-server -Xms12G -Xmx12G -Xmn3G -Xss256k'
            DEPLOY_SERVICE=$service
        ;;
        *)
            JAVA_OPTS='-server -Xms2G -Xmx2G -Xmn512m -Xss256k'
            DEPLOY_SERVICE=$service

    esac
    break
done

if [ -z "$DEPLOY_SERVICE" ];then
echo '无效的项目，退出'
exit 1
fi

echo "use JVM config : $JAVA_OPTS"
echo "请选择要使用的模式"
select mode in ${MODE_LIST[@]}
do
    if [ -n "$mode" ];then
        DEPLOY_MODE=$mode
        break
    fi  
    echo '请选择 a | b'
done

JAVA_HOME=/usr/bin/java
CATALINA_HOME=/data/gb-tomcat-cluster/tomcat-$DEPLOY_SERVICE-$DEPLOY_MODE
GB_HOME=/data/gb-tomcat-cluster
export JAVA_HOME CATALINA_HOME JAVA_OPTS

echo "use JVM config : $JAVA_OPTS"
read -r -p "请确认项目 $DEPLOY_SERVICE : 模式 : $DEPLOY_MODE ? [y/n] " input
if [ $input = "y" ]; then
    case $1 in
        start)
                exec find $GB_HOME/conf/$DEPLOY_SERVICE/$mode*.xml -exec $CATALINA_HOME/bin/catalina.sh start -config {} \;
        ;;
        stop)
                exec ps -ef |grep "$GB_HOME/conf/$DEPLOY_SERVICE/$mode.*xml start" |grep -v grep | grep -v python | awk '{print $2}' |xargs kill -9
        ;;
        restart)
                ps -ef |grep "$GB_HOME/conf/$DEPLOY_SERVICE/$mode.*xml start" |grep -v grep | grep -v python | awk '{print $2}' |xargs kill -9
                exec find $GB_HOME/conf/$DEPLOY_SERVICE/$mode*.xml -exec $CATALINA_HOME/bin/catalina.sh start -config {} \;
        ;;
        *)
                echo "USAGE:`basename $0` {start|stop|restart}"
                    exit 1
            ;;
    esac
fi  

echo 'exit'
exit 1

端口分布情况


├── customer
│   ├── a-8070.xml
│   ├── a-8071.xml
│   ├── a-8072.xml
│   ├── a-8073.xml
│   ├── a-8074.xml
│   ├── a-8075.xml
│   ├── a-8076.xml
│   ├── a-8077.xml
│   ├── a-8078.xml
│   ├── a-8079.xml
│   ├── b-8170.xml
│   ├── b-8171.xml
│   ├── b-8172.xml
│   ├── b-8173.xml
│   ├── b-8174.xml
│   ├── b-8175.xml
│   ├── b-8176.xml
│   ├── b-8177.xml
│   ├── b-8178.xml
│   └── b-8179.xml
├── seller
│   ├── a-8080.xml
│   ├── a-8081.xml
│   ├── a-8082.xml
│   ├── a-8083.xml
│   ├── a-8084.xml
│   ├── b-8180.xml
│   ├── b-8181.xml
│   ├── b-8182.xml
│   ├── b-8183.xml
│   └── b-8184.xml
├── statics
│   ├── a-8050.xml
│   └── b-8150.xml
└── wechat
    ├── a-8030.xml
    ├── a-8031.xml
    ├── b-8130.xml
    └── b-8131.xml

nginx 相关

安装步骤

#复制安装包到/usr/local/src/
    cp nginx-install.tar.gz /usr/local/src/
#解压
    tar -zxvf nginx-install.tar.gz
#执行安装
    sh nginx_https_lua_install.sh
#复制配置文件、lua脚本

配置文件及lua脚本清单


├── gb-conf
│   ├── greatbear-customer-a.conf
│   ├── greatbear-customer-b.conf
│   ├── greatbear-seller-a.conf
│   ├── greatbear-seller-b.conf
│   ├── nginx-customer.conf
│   └── nginx-seller.conf
└── ngxFlowControl
    └── ngxlua

ab 配置

  • nginx-customer.conf


user  root;
worker_processes  32;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    use epoll;
    worker_connections  10240;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"'
                      'upstream_response_time $upstream_response_time request_time $request_time ';

    lua_package_path '/usr/local/nginx-customer-ab/ngxFlowControl/?.lua;;';
    lua_package_cpath '/usr/local/nginx-customer-ab/ngxFlowControl/ngxlua/lib/?.so;;';

    # access_log  logs/access.log  main;
    # error_log  logs/error.log;

    sendfile        on;
    #tcp_nopush     on;
    #keepalive_timeout  0;

    proxy_ignore_client_abort on;

    #gzip  on;
    #压缩配置#
    gzip  on;           #打开gzip压缩功能
    gzip_min_length 1k; #压缩阈值
    gzip_buffers 4 16k; #buffer 不用修改
    gzip_comp_level 8;  #压缩级别:1-10，数字越大压缩的越好，时间也越长
    gzip_types text/plain application/javascript text/css application/xml text/javascript application/x-httpd-php image/jpeg image/gif image/png;  #        压缩文件类型
    gzip_vary off;      #跟Squid等缓存服务有关，on的话会在Header里增加 "Vary: Accept-Encoding"
    gzip_disable "MSIE [1-6]\.";  #IE1-6版本不支持gzip压缩



    include /usr/local/nginx-customer-ab/conf/vhost/greatbear-seller-a.conf;
}
  • nginx-seller.conf

user  root;
worker_processes  32;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    use epoll;
    worker_connections  10240;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"'
                      'upstream_addr $upstream_addr '
                      'upstream_response_time $upstream_response_time request_time $request_time ';

    lua_package_path '/usr/local/nginx-seller-ab/ngxFlowControl-ab/?.lua;;';
    lua_package_cpath '/usr/local/nginx-https-ab/ngxFlowControl-ab/ngxlua/lib/?.so;;';


    #access_log  logs/access.log  main;
    #error_log  logs/error.log;

    sendfile        on;
    #tcp_nopush     on;
    #keepalive_timeout  0;
    keepalive_timeout  65;

    proxy_ignore_client_abort on;    

    #gzip  on;
    #压缩配置#
    gzip  on;           #打开gzip压缩功能
    gzip_min_length 1k; #压缩阈值
    gzip_buffers 4 16k; #buffer 不用修改
    gzip_comp_level 8;  #压缩级别:1-10，数字越大压缩的越好，时间也越长
    gzip_types text/plain application/javascript text/css application/xml text/javascript application/x-httpd-php image/jpeg image/gif image/png;  #        压缩文件类型
    gzip_vary off;      #跟Squid等缓存服务有关，on的话会在Header里增加 "Vary: Accept-Encoding"
    gzip_disable "MSIE [1-6]\.";  #IE1-6版本不支持gzip压缩



    include /usr/local/nginx-seller-ab/conf/vhost/greatbear-seller-a.conf;
}
  • greatbear-customer-a.conf

    upstream greatbear-server-http{
        server 172.16.100.51:8080 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8081 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8082 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8083 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8084 max_fails=5 fail_timeout=10s;

        server 172.16.100.52:8080 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8081 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8082 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8083 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8084 max_fails=5 fail_timeout=10s;

        server 172.16.100.62:8080 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8081 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8082 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8083 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8084 max_fails=5 fail_timeout=10s;

        server 172.16.100.21:8080 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8010 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8020 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8030 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8040 max_fails=5 fail_timeout=10s;
    }

    upstream greatbear-customer-http{
        server 172.16.100.63:8070 max_fails=5 fail_timeout=10s;
        server 172.16.100.63:8071 max_fails=5 fail_timeout=10s;
        server 172.16.100.63:8072 max_fails=5 fail_timeout=10s;
        server 172.16.100.63:8073 max_fails=5 fail_timeout=10s;
        server 172.16.100.63:8074 max_fails=5 fail_timeout=10s;
        server 172.16.100.63:8075 max_fails=5 fail_timeout=10s;
        server 172.16.100.63:8076 max_fails=5 fail_timeout=10s;
        server 172.16.100.63:8077 max_fails=5 fail_timeout=10s;
        server 172.16.100.63:8078 max_fails=5 fail_timeout=10s;
        server 172.16.100.63:8079 max_fails=5 fail_timeout=10s;
    }

    upstream greatbear-customer-1 {
        server 127.0.0.1:8070 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8071 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-customer-2 {
        server 127.0.0.1:8071 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8072 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-customer-3 {
        server 127.0.0.1:8072 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8073 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-customer-4 {
        server 127.0.0.1:8073 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8074 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-customer-5 {
        server 127.0.0.1:8074 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8075 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-customer-6 {
        server 127.0.0.1:8075 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8076 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-customer-7 {
        server 127.0.0.1:8076 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8077 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-customer-8 {
        server 127.0.0.1:8077 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8078 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-customer-9 {
        server 127.0.0.1:8078 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8079 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-customer-10 {
        server 127.0.0.1:8079 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8070 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }


    upstream greatbear-wx-kefu-1 {
        server 172.16.100.31:8030 max_fails=10 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-wx-kefu-2 {
        server 172.16.100.31:8031 max_fails=10 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-wx-kefu-default {
        server 172.16.100.31:8030 max_fails=10 fail_timeout=10s;
        keepalive 1000;
    }

    server {
        listen       80;
        listen       443 ssl;
        server_name opp.skynet.shangdejigou.cn greatbearopp.sunlands.com;
        include /usr/local/nginx-customer-ab/conf/vhost/deny.ip;
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto  $scheme; # ssl
        #charset koi8-r;
        large_client_header_buffers 4 16k;
        client_max_body_size 300m;
        client_body_buffer_size 128k;
        proxy_buffer_size 64k;
        proxy_buffers   4 32k;
        proxy_busy_buffers_size 64k;
        proxy_temp_file_write_size 64k;

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        location  / {
             proxy_pass   http://greatbear-server-http/;
             proxy_http_version 1.1;
             proxy_connect_timeout 600s;
             proxy_read_timeout 3600s;
             proxy_send_timeout 600s;
             proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection "Upgrade";
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             #proxy_request_buffering off;
             #proxy_buffering off;

        }


        location  /greatbear-customer {
             proxy_pass   http://greatbear-customer-http;
             proxy_http_version 1.1;
             proxy_connect_timeout 600s;
             proxy_read_timeout 3600s;
             proxy_send_timeout 600s;
             proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection "Upgrade";
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             #proxy_request_buffering off;
             #proxy_buffering off;

        }


        location ~ /greatbear-customer/customer/pu[ls][lh] {
                set $customer_back_num '10';
                set $cb_pname 'uid';
                set $cb_strategy 'balance_hash';
                set_by_lua_file $back_idx /usr/local/nginx-customer-ab/ngxFlowControl/ngxlua/server/customer_balance.lua;
                proxy_pass http://greatbear-customer-$back_idx;
                proxy_http_version 1.1;
                proxy_connect_timeout 600s;
                proxy_read_timeout 86400s;
                proxy_send_timeout 600s;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                keepalive_timeout  300s 300s;
        }
        location  /greatbear-wechat/customer/message {
                set $wx_back_num '2';
                set $back_idx '';
                rewrite_by_lua_file /usr/local/nginx-customer-ab/ngxFlowControl/ngxlua/server/wx_balance.lua;
                proxy_pass http://greatbear-wx-kefu-$back_idx;
                proxy_http_version 1.1;
                proxy_connect_timeout 600s;
                proxy_read_timeout 3600s;
                proxy_send_timeout 600s;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                keepalive_timeout  300s 300s;
                keepalive_requests 10000;

        }

        location /visit {
            root /data/greatbear-static;
            index index-*.html;
        }


        location ~* \.(mp3)$ {
            root /data/work/greatbear-web/assets;
            rewrite /greatbear-war/assets/(.*\.mp3) /$1;
        }

        location /nginx-status
        {
                stub_status on;
                access_log off;
                allow 172.16.102.131;
                allow 172.16.102.230;
                allow 127.0.0.1;
                deny all;
        }

        location /robots.txt {
            root /data;
        }
        location /report.html {
            root /usr/share/nginx/html/site;
            access_log off;
            allow 172.16.102.131;
            allow 172.16.202.10;
            deny all;
        }

        # ssl
        # ssl on;
        ssl_certificate star_sunlands_com.crt;
        ssl_certificate_key star_sunlands_com.key;
        ssl_session_timeout  5m;
        ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers AES128-SHA:AES256-SHA:RC4-SHA:DES-CBC3-SHA:RC4-MD5;

        access_log /usr/local/nginx-customer/logs/greatbear.customer.access.log;
        error_log  /usr/local/nginx-customer/logs/greatbear.customer.error.log;
     }
  • greatbear-customer-b.conf

    upstream greatbear-server-http{
        server 172.16.100.51:8180 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8181 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8182 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8183 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8184 max_fails=5 fail_timeout=10s;

        server 172.16.100.52:8180 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8181 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8182 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8183 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8184 max_fails=5 fail_timeout=10s;

        server 172.16.100.62:8180 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8181 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8182 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8183 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8184 max_fails=5 fail_timeout=10s;

        server 172.16.100.21:8180 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8110 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8120 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8130 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8140 max_fails=5 fail_timeout=10s;
    }

    upstream greatbear-customer-http{
        server 172.16.100.63:8170 max_fails=5 fail_timeout=10s;
        server 172.16.100.63:8171 max_fails=5 fail_timeout=10s;
        server 172.16.100.63:8172 max_fails=5 fail_timeout=10s;
        server 172.16.100.63:8173 max_fails=5 fail_timeout=10s;
        server 172.16.100.63:8174 max_fails=5 fail_timeout=10s;
        server 172.16.100.63:8175 max_fails=5 fail_timeout=10s;
        server 172.16.100.63:8176 max_fails=5 fail_timeout=10s;
        server 172.16.100.63:8177 max_fails=5 fail_timeout=10s;
        server 172.16.100.63:8178 max_fails=5 fail_timeout=10s;
        server 172.16.100.63:8179 max_fails=5 fail_timeout=10s;
    }

    upstream greatbear-customer-1 {
        server 127.0.0.1:8170 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8171 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-customer-2 {
        server 127.0.0.1:8171 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8172 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-customer-3 {
        server 127.0.0.1:8172 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8173 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-customer-4 {
        server 127.0.0.1:8173 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8174 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-customer-5 {
        server 127.0.0.1:8174 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8175 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-customer-6 {
        server 127.0.0.1:8175 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8176 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-customer-7 {
        server 127.0.0.1:8176 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8177 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-customer-8 {
        server 127.0.0.1:8177 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8178 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-customer-9 {
        server 127.0.0.1:8178 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8179 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-customer-10 {
        server 127.0.0.1:8179 max_fails=10 fail_timeout=10s;
        server 127.0.0.1:8170 backup max_fails=5 fail_timeout=10s;
        keepalive 1000;
    }


    upstream greatbear-wx-kefu-1 {
        server 172.16.100.31:8130 max_fails=10 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-wx-kefu-2 {
        server 172.16.100.31:8131 max_fails=10 fail_timeout=10s;
        keepalive 1000;
    }

    upstream greatbear-wx-kefu-default {
        server 172.16.100.31:8130 max_fails=10 fail_timeout=10s;
        keepalive 1000;
    }

    server {
        listen       80;
        listen       443 ssl;
        server_name opp.skynet.shangdejigou.cn greatbearopp.sunlands.com;
        include /usr/local/nginx-customer-ab/conf/vhost/deny.ip;
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto  $scheme; # ssl
        #charset koi8-r;
        large_client_header_buffers 4 16k;
        client_max_body_size 300m;
        client_body_buffer_size 128k;
        proxy_buffer_size 64k;
        proxy_buffers   4 32k;
        proxy_busy_buffers_size 64k;
        proxy_temp_file_write_size 64k;

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        location  / {
             proxy_pass   http://greatbear-server-http/;
             proxy_http_version 1.1;
             proxy_connect_timeout 600s;
             proxy_read_timeout 3600s;
             proxy_send_timeout 600s;
             proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection "Upgrade";
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             #proxy_request_buffering off;
             #proxy_buffering off;

        }


        location  /greatbear-customer {
             proxy_pass   http://greatbear-customer-http;
             proxy_http_version 1.1;
             proxy_connect_timeout 600s;
             proxy_read_timeout 3600s;
             proxy_send_timeout 600s;
             proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection "Upgrade";
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             #proxy_request_buffering off;
             #proxy_buffering off;

        }


        location ~ /greatbear-customer/customer/pu[ls][lh] {
                set $customer_back_num '10';
                set $cb_pname 'uid';
                set $cb_strategy 'balance_hash';
                set_by_lua_file $back_idx /usr/local/nginx-customer-ab/ngxFlowControl/ngxlua/server/customer_balance.lua;
                proxy_pass http://greatbear-customer-$back_idx;
                proxy_http_version 1.1;
                proxy_connect_timeout 600s;
                proxy_read_timeout 86400s;
                proxy_send_timeout 600s;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                keepalive_timeout  300s 300s;
        }
        location  /greatbear-wechat/customer/message {
                set $wx_back_num '2';
                set $back_idx '';
                rewrite_by_lua_file /usr/local/nginx-customer-ab/ngxFlowControl/ngxlua/server/wx_balance.lua;
                proxy_pass http://greatbear-wx-kefu-$back_idx;
                proxy_http_version 1.1;
                proxy_connect_timeout 600s;
                proxy_read_timeout 3600s;
                proxy_send_timeout 600s;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                keepalive_timeout  300s 300s;
                keepalive_requests 10000;

        }

        location /visit {
            root /data/greatbear-static;
            index index-*.html;
        }


        location ~* \.(mp3)$ {
            root /data/work/greatbear-web/assets;
            rewrite /greatbear-war/assets/(.*\.mp3) /$1;
        }

        location /nginx-status
        {
                stub_status on;
                access_log off;
                allow 172.16.102.131;
                allow 172.16.102.230;
                allow 127.0.0.1;
                deny all;
        }

        location /robots.txt {
            root /data;
        }
        location /report.html {
            root /usr/share/nginx/html/site;
            access_log off;
            allow 172.16.102.131;
            allow 172.16.202.10;
            deny all;
        }

        # ssl
        # ssl on;
        ssl_certificate star_sunlands_com.crt;
        ssl_certificate_key star_sunlands_com.key;
        ssl_session_timeout  5m;
        ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers AES128-SHA:AES256-SHA:RC4-SHA:DES-CBC3-SHA:RC4-MD5;

        access_log /usr/local/nginx-customer/logs/greatbear.customer.access.log;
        error_log  /usr/local/nginx-customer/logs/greatbear.customer.error.log;
     }
  • greatbear-seller-a.conf

    upstream greatbear-server-iphash{

        server 172.16.100.62:8080 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8081 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8082 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8083 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8084 max_fails=5 fail_timeout=10s;

        server 172.16.100.21:8080 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8081 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8082 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8083 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8084 max_fails=5 fail_timeout=10s;

        server 172.16.100.51:8080 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8081 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8082 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8083 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8084 max_fails=5 fail_timeout=10s;

        server 172.16.100.52:8080 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8081 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8082 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8083 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8084 max_fails=5 fail_timeout=10s;

    }

     upstream greatbear-server-{

        server 172.16.100.62:8080 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8081 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8082 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8083 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8084 max_fails=5 fail_timeout=10s;

        server 172.16.100.21:8080 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8081 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8082 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8083 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8084 max_fails=5 fail_timeout=10s;

        server 172.16.100.51:8080 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8081 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8082 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8083 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8084 max_fails=5 fail_timeout=10s;

        server 172.16.100.52:8080 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8081 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8082 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8083 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8084 max_fails=5 fail_timeout=10s;

    }


    upstream greatbear-server-1{
        server 172.16.100.62:8080 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8081 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-2{
        server 172.16.100.62:8081 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8082 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-3{
        server 172.16.100.62:8082 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8083 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-4{
        server 172.16.100.62:8083 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8084 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-5{
        server 172.16.100.62:8084 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8080 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-6{
        server 172.16.100.21:8080 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8081 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-7{
        server 172.16.100.21:8081 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8082 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-8{
        server 172.16.100.21:8082 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8083 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-9{
        server 172.16.100.21:8083 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8084 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-10{
        server 172.16.100.21:8084 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8080 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-11{
        server 172.16.100.51:8080 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8081 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-12{
        server 172.16.100.51:8081 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8082 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-13{
        server 172.16.100.51:8082 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8083 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-14{
        server 172.16.100.51:8083 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8084 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-15{
        server 172.16.100.51:8084 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8080 max_fails=5 fail_timeout=10s backup;
    }
    upstream greatbear-server-16{
        server 172.16.100.52:8080 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8081 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-17{
        server 172.16.100.52:8081 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8082 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-18{
        server 172.16.100.52:8082 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8083 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-19{
        server 172.16.100.52:8083 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8084 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-20{
        server 172.16.100.52:8084 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8080 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-blacklist {
        server 172.16.100.31:18050 max_fails=5 fail_timeout=10s;
    }

    upstream greatbear-statistics {
        server 172.16.100.31:8050 max_fails=5 fail_timeout=10s;
    }

    server {
        listen       80;
        listen       443 ssl;
        server_name  customer.skynet.shangdejigou.cn greatbearcst.sunlands.com stres.skynet.shangdejigou.cn;
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto  $scheme;
        #charset koi8-r;
        large_client_header_buffers 4 16k;
        client_max_body_size 600m;
        client_body_buffer_size 128k;
        proxy_buffer_size 64k;
        proxy_buffers   4 32k;
        proxy_busy_buffers_size 64k;
        proxy_temp_file_write_size 64k;

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        location  / {
             proxy_pass   http://greatbear-server-iphash/;
             proxy_http_version 1.1;
             proxy_connect_timeout 600s;
             proxy_read_timeout 3600s;
             proxy_send_timeout 600s;
             proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection "Upgrade";
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             #proxy_request_buffering off;
             #proxy_buffering off;

        }

        location = /greatbear-war/ws/im/seller {
             set $back_num '20';
             set $uid '';
             set $param_name 'seller_id';
             set_by_lua_file $uid /usr/local/nginx-seller-ab/ngxFlowControl/ngxlua/server/counselor_balance.lua;
             proxy_pass http://greatbear-server-$uid;
             proxy_http_version 1.1;
             proxy_connect_timeout 600s;
             proxy_read_timeout 3600s;
             proxy_send_timeout 600s;
             proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection "Upgrade";
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

        location = /greatbear-war/erpCard/addERPCard {
             set $back_num '20';
             set $uid '';
             set $param_name 'seller_id';
             set_by_lua_file $uid /usr/local/nginx-seller-ab/ngxFlowControl/ngxlua/server/counselor_balance.lua;
             proxy_pass http://greatbear-server-$uid;
             proxy_http_version 1.1;
             proxy_connect_timeout 600s;
             proxy_read_timeout 3600s;
             proxy_send_timeout 600s;
             proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection "Upgrade";
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

        location = /greatbear-war/dataMonitor/findTodayData {
             set $back_num '20';
             set $uid '';
             set $param_name 'seller_id';
             set_by_lua_file $uid /usr/local/nginx-seller-ab/ngxFlowControl/ngxlua/server/counselor_balance.lua;
             proxy_pass http://greatbear-server-$uid;
             proxy_http_version 1.1;
             proxy_connect_timeout 600s;
             proxy_read_timeout 3600s;
             proxy_send_timeout 600s;
             proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection "Upgrade";
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

        location /greatbear-blacklist {

             proxy_pass   http://greatbear-blacklist;
             proxy_http_version 1.1;
             proxy_connect_timeout 600s;
             proxy_read_timeout 3600s;
             proxy_send_timeout 600s;
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

        location /greatbear-statistics {

             proxy_pass   http://greatbear-statistics;
             proxy_http_version 1.1;
             proxy_connect_timeout 600s;
             proxy_read_timeout 3600s;
             proxy_send_timeout 600s;
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }


        location /visit {
            proxy_http_version 1.1;
            proxy_set_header Host $proxy_host;
            proxy_read_timeout 3600s;
            proxy_send_timeout 600s;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            set $ab_pname 'siteId';
            set_by_lua_file $ab_tag /usr/local/nginx-seller-ab/ngxFlowControl/ngxlua/server/ab_static.lua;

            #if ( $ab_tag = 'req_cdn' ){
            #     proxy_pass http://stres.skynet.shangdejigou.cn;
            #}

            if ( $ab_tag = 'req_b' ){
                rewrite ^/visit/(.*)$ /visitb/$1 last;
            }

            root /data/greatbear-static;
            index index-*.html;
        }

        location /visitb {
            root /data/greatbear-static-b;
            index index-*.html;
        }

        location ~* \.(mp3)$ {
            root /data/work/greatbear-web/assets;
            rewrite /greatbear-war/assets/(.*\.mp3) /$1;
        }

        location /nginx-status
        {
                stub_status on;
                access_log off;
                allow 172.16.102.131;
                allow 172.16.102.230;
                allow 127.0.0.1;
                deny all;
        }

        location /static {
            root /data/greatbear-static;
            index greatbear.html;
        }

        # 小程序验证
        location /8MvKoFJito.txt {
            root /data/greatbear-static/checkfile/;
        }

        # ssl
        # ssl on;
        ssl_certificate star_sunlands_com.crt;
        ssl_certificate_key star_sunlands_com.key;
        ssl_session_timeout  5m;
        ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers AES128-SHA:AES256-SHA:RC4-SHA:DES-CBC3-SHA:RC4-MD5;

        access_log /usr/local/nginx-seller-ab/logs/greatbear.ssl.access.log main;
        error_log  /usr/local/nginx-seller-ab/logs/greatbear.ssl.error.log;
     }
  • greatbear-seller-b.conf

    upstream greatbear-server-iphash{

        server 172.16.100.62:8180 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8181 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8182 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8183 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8184 max_fails=5 fail_timeout=10s;

        server 172.16.100.21:8180 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8181 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8182 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8183 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8184 max_fails=5 fail_timeout=10s;

        server 172.16.100.51:8180 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8181 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8182 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8183 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8184 max_fails=5 fail_timeout=10s;

        server 172.16.100.52:8180 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8181 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8182 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8183 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8184 max_fails=5 fail_timeout=10s;

    }

     upstream greatbear-server-{

        server 172.16.100.62:8180 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8181 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8182 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8183 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8184 max_fails=5 fail_timeout=10s;

        server 172.16.100.21:8180 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8181 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8182 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8183 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8184 max_fails=5 fail_timeout=10s;

        server 172.16.100.51:8180 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8181 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8182 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8183 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8184 max_fails=5 fail_timeout=10s;

        server 172.16.100.52:8180 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8181 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8182 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8183 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8184 max_fails=5 fail_timeout=10s;

    }


    upstream greatbear-server-1{
        server 172.16.100.62:8180 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8181 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-2{
        server 172.16.100.62:8181 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8182 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-3{
        server 172.16.100.62:8182 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8183 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-4{
        server 172.16.100.62:8183 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8184 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-5{
        server 172.16.100.62:8184 max_fails=5 fail_timeout=10s;
        server 172.16.100.62:8180 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-6{
        server 172.16.100.21:8180 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8181 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-7{
        server 172.16.100.21:8181 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8182 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-8{
        server 172.16.100.21:8182 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8183 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-9{
        server 172.16.100.21:8183 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8184 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-10{
        server 172.16.100.21:8184 max_fails=5 fail_timeout=10s;
        server 172.16.100.21:8180 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-11{
        server 172.16.100.51:8180 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8181 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-12{
        server 172.16.100.51:8181 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8182 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-13{
        server 172.16.100.51:8182 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8183 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-14{
        server 172.16.100.51:8183 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8184 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-15{
        server 172.16.100.51:8184 max_fails=5 fail_timeout=10s;
        server 172.16.100.51:8180 max_fails=5 fail_timeout=10s backup;
    }
    upstream greatbear-server-16{
        server 172.16.100.52:8180 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8181 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-17{
        server 172.16.100.52:8181 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8182 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-18{
        server 172.16.100.52:8182 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8183 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-19{
        server 172.16.100.52:8183 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8184 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-server-20{
        server 172.16.100.52:8184 max_fails=5 fail_timeout=10s;
        server 172.16.100.52:8180 max_fails=5 fail_timeout=10s backup;
    }

    upstream greatbear-blacklist {
        server 172.16.100.31:18050 max_fails=5 fail_timeout=10s;
    }

    upstream greatbear-statistics {
        server 172.16.100.31:8150 max_fails=5 fail_timeout=10s;
    }

    server {
        listen       80;
        listen       443 ssl;
        server_name  customer.skynet.shangdejigou.cn greatbearcst.sunlands.com stres.skynet.shangdejigou.cn;
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto  $scheme;
        #charset koi8-r;
        large_client_header_buffers 4 16k;
        client_max_body_size 600m;
        client_body_buffer_size 128k;
        proxy_buffer_size 64k;
        proxy_buffers   4 32k;
        proxy_busy_buffers_size 64k;
        proxy_temp_file_write_size 64k;

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        location  / {
             proxy_pass   http://greatbear-server-iphash/;
             proxy_http_version 1.1;
             proxy_connect_timeout 600s;
             proxy_read_timeout 3600s;
             proxy_send_timeout 600s;
             proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection "Upgrade";
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             #proxy_request_buffering off;
             #proxy_buffering off;

        }

        location = /greatbear-war/ws/im/seller {
             set $back_num '20';
             set $uid '';
             set $param_name 'seller_id';
             set_by_lua_file $uid /usr/local/nginx-seller-ab/ngxFlowControl/ngxlua/server/counselor_balance.lua;
             proxy_pass http://greatbear-server-$uid;
             proxy_http_version 1.1;
             proxy_connect_timeout 600s;
             proxy_read_timeout 3600s;
             proxy_send_timeout 600s;
             proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection "Upgrade";
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

        location = /greatbear-war/erpCard/addERPCard {
             set $back_num '20';
             set $uid '';
             set $param_name 'seller_id';
             set_by_lua_file $uid /usr/local/nginx-seller-ab/ngxFlowControl/ngxlua/server/counselor_balance.lua;
             proxy_pass http://greatbear-server-$uid;
             proxy_http_version 1.1;
             proxy_connect_timeout 600s;
             proxy_read_timeout 3600s;
             proxy_send_timeout 600s;
             proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection "Upgrade";
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

        location = /greatbear-war/dataMonitor/findTodayData {
             set $back_num '20';
             set $uid '';
             set $param_name 'seller_id';
             set_by_lua_file $uid /usr/local/nginx-seller-ab/ngxFlowControl/ngxlua/server/counselor_balance.lua;
             proxy_pass http://greatbear-server-$uid;
             proxy_http_version 1.1;
             proxy_connect_timeout 600s;
             proxy_read_timeout 3600s;
             proxy_send_timeout 600s;
             proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection "Upgrade";
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

        location /greatbear-blacklist {

             proxy_pass   http://greatbear-blacklist;
             proxy_http_version 1.1;
             proxy_connect_timeout 600s;
             proxy_read_timeout 3600s;
             proxy_send_timeout 600s;
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

        location /greatbear-statistics {

             proxy_pass   http://greatbear-statistics;
             proxy_http_version 1.1;
             proxy_connect_timeout 600s;
             proxy_read_timeout 3600s;
             proxy_send_timeout 600s;
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }


        location /visit {
            proxy_http_version 1.1;
            proxy_set_header Host $proxy_host;
            proxy_read_timeout 3600s;
            proxy_send_timeout 600s;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            set $ab_pname 'siteId';
            set_by_lua_file $ab_tag /usr/local/nginx-seller-ab/ngxFlowControl/ngxlua/server/ab_static.lua;

            #if ( $ab_tag = 'req_cdn' ){
            #     proxy_pass http://stres.skynet.shangdejigou.cn;
            #}

            if ( $ab_tag = 'req_b' ){
                rewrite ^/visit/(.*)$ /visitb/$1 last;
            }

            root /data/greatbear-static;
            index index-*.html;
        }

        location /visitb {
            root /data/greatbear-static-b;
            index index-*.html;
        }

        location ~* \.(mp3)$ {
            root /data/work/greatbear-web/assets;
            rewrite /greatbear-war/assets/(.*\.mp3) /$1;
        }

        location /nginx-status
        {
                stub_status on;
                access_log off;
                allow 172.16.102.131;
                allow 172.16.102.230;
                allow 127.0.0.1;
                deny all;
        }

        location /static {
            root /data/greatbear-static;
            index greatbear.html;
        }

        # 小程序验证
        location /8MvKoFJito.txt {
            root /data/greatbear-static/checkfile/;
        }

        # ssl
        # ssl on;
        ssl_certificate star_sunlands_com.crt;
        ssl_certificate_key star_sunlands_com.key;
        ssl_session_timeout  5m;
        ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers AES128-SHA:AES256-SHA:RC4-SHA:DES-CBC3-SHA:RC4-MD5;

        access_log /usr/local/nginx-seller-ab/logs/greatbear.ssl.access.log main;
        error_log  /usr/local/nginx-seller-ab/logs/greatbear.ssl.error.log;
     }

扩展

keepalived 高可用方案

https://blog.csdn.net/tianjun2012/article/details/53437792

luwfls
关注
  • 5
    点赞
  • 踩
  • 8
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
电商网站HTTPS实践之路（四）——灰度上线篇
皖南笑笑生的博客
05-17 2657
对于电商而言，不能因为技术的升级就轻易影响了线上业务，毕竟最宝贵的是用户流量。当我们一鼓作气完成了系统的HTTPS改造和优化后，上线过程则是小心翼翼逐步完成的，毕竟这么大的调整动作牵一发而动全身。从HTTP过渡到全站HTTPS，没有发生一次严重事故，究其原因是我们始终遵守三个原则，即降级原则、灰度原则和开闭原则。 1. 灰度发布 2. 开关控制 3. 性能监控 1. 灰度发布 ...
nginx+lua+redis实现灰度发布 | 京东云技术团队
最新发布
京东科技开发者
12-18 1788
授人以鱼不如授人以渔.先学会用，在学原理，在学创造，可能一辈子用不到这种能力，但是不能不具备这种能力。这篇文章主要是沉淀使用nginx+lua+redis实现灰度，当我们具备了这种能力，随时可以基于这种能力和思想调整实现方案：比如nginx+lua+(其他数据源)、nginx+(其他脚本语言)
灰度发布（上线）和AB Test
lmm0513的博客
12-09 889
原文地址：https://blog.csdn.net/maxmao1024/article/details/84305609 一、灰度发布 1. 简介 灰度发布，是指在黑与白之间，能够平滑过渡的一种发布方式。 通过不同策略对用户进行分流，不同的用户组使用不同的应用版本。 2. 优点 互联网服务变动频繁，发布周期短。速度与质量总是难以双全。灰度发布有以下优点： 降低...
灰度上线
weixin_33923762的博客
02-09 170
　　传统的产品研发模式大致可以分为：产品调研-架构评估-产品启动-需求分析-产品设计-产品开发-产品发布七大阶 段。本人在公司经历过大大小小的项目数以百计，发觉这些阶段一直以来都是以一条直线的形式串行着：从产品调研到产品发布，总是一拖到底。这样的做法对于范 围比较大，周期比较长的项目，尤其是用户体验类项目而言，存在较大的弊端：我们很可能在没有足够清楚用户需求的情况下，定制了过多的辅助功能，这样即拉...
服务灰度发布方案.docx
04-29
OpenResty 的 AB Testing Gateway 是基于 OpenResty 的灰度发布解决方案，提供了简单的配置方式和测试方式。AB Testing Gateway 通过在 OpenResty 中添加灰度发布的规则，以控制流量的分配，并确保整体系统的稳定。...
AB技术方案.pdf
08-28
AB技术方案.pdf AB技术方案.pdf是关于PLC（Programmable Logic Controller，程序逻辑控制器）系统的技术协议，主要介绍了COMPACT LOGIX 1769 PLC系统的技术特点和应用。 1. COMPACT LOGIX 1769 PLC系统概述 ...
AB与西门子通讯方案
12-25
rockwell ab 与 siemens 通讯 sst pb3
基于Airoha AB1601的矫正书写姿势智能笔方案-电路方案
04-20
Airoha 推出AB1601+Bosch BMA253智能笔方案可有效解决青少年因书写姿势不当造成的近视问题。智能笔内置了四个感应器，分别是距离感应器、光线感应器、触摸感应器和角度感应器，全方位对孩子的坐姿、握姿和环境光进行...
AB PLC程序下载步骤
01-27
AB PLC 程序下载步骤 AB PLC 程序下载是一个复杂的过程，它涉及到多个步骤和设置。本文将详细介绍 AB PLC 程序下载的步骤，包括配置 IP 地址、打开 BOOTP、网线连接、修改 IP 地址、enable 设置、修改子网掩码等。 ...
如何设计可靠的灰度方案
阿里云云栖号
08-23 2740
简介：一个较大的业务或系统改动，往往会影响整个产品的用户体验或操作流程。为了控制影响面，可以选取一批特定用户、流程、单据等，只允许这一部分用户或数据按照变更后的新逻辑在系统中流转，而另一部分用户仍然执行变更前的老逻辑。这一步是线上系统灰度方案的起点。 作者 | 既同 来源 | 阿里技术公众号 一 灰度的基本概念 1 一个典型的灰度方案 一个较大的业务或系统改动，往往会影响整个产品的用户体验或操作流程。为了控制影响面，可以选取一批特定用户、流程、单据等，只允许这一部分用户或数据按照变更后的新逻辑.
AB测试-最佳方案测试
纸上得来终觉浅，绝知此事要躬行
04-03 3518
目录 实验设计 流量分配 实验效果 实验结束 多个活动交集量化的实验设计 业务迭代的同时，如何与自身的过去比较 更为复杂的实验设计 AB测试的目标是选出不同参考方案的优点进行选择，包括：用户体验，用户流量，性能等等，每个产品测试的点不一样AB测试的目标就会不同。 百度上的解释更加偏向于开发web、APP：https://baike.baidu.com/item/AB%E6%B5%8B%E8%AF%95/9231223?fr=aladdin 产品经理关注点在于用户体验和接受度：http:/
【干货】基于 Nginx 实现一个灰度上线系统
小生方勤
08-07 490
软件开发一般不会上来就是最终版本，而是会一个版本一个版本的迭代。新版本上线前都会经过测试，但就算这样，也不能保证上线了不出问题。所以，在公司里上线新版本代码一般都是通过灰度系统。灰度系统可以把流量划分成多份，一份走新版本代码，一份走老版本代码。而且灰度系统支持设置流量的比例，比如可以把走新版本代码的流量设置为 5%，没啥问题再放到 10%，50%，最后放到 100% 全量。这样可以把出现问题的影响...
基于 Nginx 实现一个灰度上线系统
奇舞周刊
07-03 412
软件开发一般不会上来就是最终版本，而是会一个版本一个版本的迭代。新版本上线前都会经过测试，但就算这样，也不能保证上线了不出问题。所以，在公司里上线新版本代码一般都是通过灰度系统。灰度系统可以把流量划分成多份，一份走新版本代码，一份走老版本代码。而且灰度系统支持设置流量的比例，比如可以把走新版本代码的流量设置为 5%，没啥问题再放到 10%，50%，最后放到 100% 全量。这样可以把出现问题的影响...
教你如何对产品做AB测试（abtest）
热门推荐
Weiguang_123的专栏
10-17 12万+
我们经常会面临多个设计方案的选择，比如app或pc端某个界面的某个按钮是用红色还是用蓝色，是放左边还是放右边。传统的解决方法通常是集体讨论表决，或者由某位专 家或领导或文青来拍板，实在决定不了时也有随机选一个上线的。虽然传统解决办法多数情况下也是有效的，但A/B 测试（A/B Testing）可能是解决这类问题的一个更好的方法。 所谓 A/B 测试，简单来说，就是为同一个目标制定两个方案（比如
灰度发布和AB test
黄河敏捷开发
05-05 2万+
 马化腾曾在一场创业演讲中，讲述了微信的诞生史：微信在诞生之前，在腾讯内部有三个团队在同时做微信，主要竞争者为张小龙的e-mail团队和手机QQ团队。做这个产品之前，腾讯内部并没有给这个产品定一个完整的基调，而是让公司内部形成一个激烈的竞争，通过观察用户对产品的喜好程度和产品的实际完成情况决定上线结果，这就是腾讯的灰度。 很多时候我们都发现过这种现象，有时相同的手机App里展示的页面会和别人...
AB 压测的部署与使用指南，还有出现的问题与解决方案
weixin_33961829的博客
05-22 1365
为什么80%的码农都做不了架构师？>>> ...
蓝绿部署、金丝雀发布（灰度发布）、AB测试……
方志朋的博客
05-07 510
点击关注公众号，Java干货及时送达来源 |https://www.jianshu.com/p/0df88fe4a1e3随着微服务架构的普及，线上服务越来越多，随之而来的就是部署越来越频...
灰度发布和AB Test
maxmao1024的博客
11-20 1448
文章目录一、灰度发布1. 简介2. 优点3. 缺点二、AB Test1. 简介2. 优缺点3. 核心思想4. 实现步骤三、实现方案1. nginx控制2. 后台代码控制3. 前端代码控制四、第三方框架五、参考资料 一、灰度发布 1. 简介 灰度发布，是指在黑与白之间，能够平滑过渡的一种发布方式。 通过不同策略对用户进行分流，不同的用户组使用不同的应用版本。 2. 优点 互联网服务变动频繁，发布周期...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值