Solaris10 SPARC操作系统中使用Nginx+Tomcat实现负载均衡，并实现websocket代理（二）

2 实施步骤

上一节我们简单介绍了负载均衡的知识，下面我们就来看看具体如何实现负载均衡。

2.1 准备资料

Nginx依赖很多其他的东西，因此我们需要提前准备好。

2.1.1 Nginx软件

首先是实现负载均衡的代理服务器nginx，下载的地址是：
http://nginx.org/en/download.html

下载的时候一定要区分好所下载的是windows版本的还是Linux/Unix版本的。最好的判断方式就是下载完成之后，打开压缩文件查看下，如果存在nginx.exe这个可执行文件，则为windous版本的，否则为非windows版本的：
Windows版本的示意图如下：

Linux/Unix版本的示意图如下：

2.1.2 GCC的下载

在对nginx进行安装之前，我们需要先对nginx进行编译，这个编译的过程需要依赖GCC环境，否则编译不过去。
下载地址为：http://download.csdn.net/detail/freellf/3329820
libgcc-3.4.6-sol9-sparc-local.gz
(网上好多地方说是可以在www.sunfreeware.com 上下载，但是我硬是没找到在哪里下载，所以只好自己搜索了)

2.1.3 依赖的模块

2.1.3.1 PCRE

这个是必须的，nginx的rewrite模块就是要靠这个才能跑起来
pcre-8.32.tar.gz ：
下载地址为：http://download.csdn.net/detail/wangyaodong915/4934067

2.1.3.2 ZLIB

Zlib是一个压缩和解压模块，在配置文件中的gzip功能需要该模块
zlib-1.2.8.tar.gz：
下载地址：http://www.zlib.net/

2.1.3.3 nginx_tcp_proxy_module

Nginx开源软件默认没有提供TCP协议的负载均衡，所以当我们需要tcp协议的负载均衡的话，我们就需要这个模块。
nginx_tcp_proxy_module-master.zip：
下载地址：https://github.com/yaoweibin/nginx_tcp_proxy_module

2.1.3.4 openssl

nginx_tcp_proxy_module会用到openssl(http://www.openssl.org)的支持，所以需要这个包。
openssl-1.0.2a.tar.gz：
下载地址：https://www.openssl.org/source/

2.2 安装部署

2.2.1 GCC的安装步骤

2.2.1.1 上传GCC

安装之前我们需要知道我们的操作系统的版本，查看方式如下：

bash-3.00#cd ./etc
bash-3.00#more release

通过查看，我的系统的版本是Solaris 10 specs。
然后根据版本下载对应的GCC，下载完成之后放到一个目录：
在/usr/local下面新建一个空文件夹src，然后切换目录到这里：

bash-3.00#cd ./usr/local/src/

上传libgcc-3.4.6-sol9-sparc-local.gz到/usr/local/src/：

2.2.1.2 安装gcc

bash-3.00#gunzip libgcc-3.4.6-sol9-sparc-local.gz
bash-3.00#pkgadd -d libgcc-3.4.6-sol9-sparc-local

2.2.1.3 设置全局环境变量

设置全局环境变量：/etc/profile是一个全局的环境变量设置，只要登陆系统的用户都会执行里面的ENV环境变量设置(etc/profile)

PATH=$PATH:/usr/local/bin
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export PATH 
export LD_LIBRARY_PATH

注意：修改的时候可以先把profile下载到本地，然后再进行修改，最后覆盖系统里的，不要直接在系统中进行修改

2.2.1.4 检验安装

bash-3.00#gcc –v

如果正常显示类似下面的信息，则说明安装成功：

Reading specs from /usr/local/lib/gcc/sparc-sun-solaris2.10/3.4.6/specs Configured with: ../configure --with-as=/usr/ccs/bin/as --with-ld=/usr/ccs/bin/ld --enable-shared --enable-languages=c,c++,f77 Thread model: posix gcc version 3.4.6

2.2.2 安装Nginx

2.2.2.1 上传并解压nginx

首先创建空文件夹server：

bash-3.00# cd ./opt/tchzt/
bash-3.00# mkdir server

上传nginx到指定目录：/opt/tchzt/server/

解压：

bash-3.00# tar -xzvf nginx-1.8.0.tar.gz

会报如下错误：

tar: z: unknown function modifier
Usage: tar {c|r|t|u|x}[BDeEFhilmnopPqTvw@[0-7]][bfk][X...] [blocksize] [tarfile] [size] [exclude-file...] {file | -I include-file | -C directory file}...

因此我们需要换一种方式进行解压(先去掉.gz,然后再用tar进行解压)：

bash-3.00# gunzip  nginx-1.8.0.tar.gz
bash-3.00# tar  –xvf  nginx-1.8.0.tar

2.2.2.2 上传并解压nginx依赖的模块

首先切换目录：

bash-3.00# cd ./opt/tchzt/server/nginx-1.8.0/

上传依赖库到/opt/tchzt/server/ nginx-1.8.0/：

1. pcre-8.32.tar.gz
2. zlib-1.2.8.tar.gz
3. nginx_tcp_proxy_module-master.zip
4. openssl-1.0.2a.tar.gz

使用以下命令依次解压每个*.tar.gz压缩文件到当前文件夹：

bash-3.00# gunzip *.tar.gz
bash-3.00# tar -xvf *.tar

使用以下命令解压*.zip：

unzip nginx_tcp_proxy_module-master.zip

2.2.2.3 把tcp模块以补丁形式加到nginx

bash-3.00# gpatch -p1 < nginx_tcp_proxy_module-master/tcp.patch

注意：有的地方说是需patch -p1<nginx_tcp_proxy_module-master/tcp.patch我这里使用patch会报错：

Looks like a unified context diff.
File to patch:

2.2.2.4 配置nginx

bash-3.00# ./configure --add-module=nginx_tcp_proxy_module-master  --with-pcre=pcre-8.32 --with-openssl=openssl-1.0.2a --with-zlib=zlib-1.2.8 --with-poll_module --prefix=/opt/tchzt/server/nginx-1.8.0/run

会报如下错误

checking for OS
         + SunOS 5.10 sun4u
        checking for C compiler ... not found
        ./configure: error: C compiler cc is not found

解决方案如下：在设置全局环境变量的文件里添加CC这个新的变量：

PATH=$PATH:/usr/local/bin
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
CC=gcc

export PATH 
export LD_LIBRARY_PATH
export CC

再次运行配置命令，正常运行。

2.2.2.5 编译nginx

bash-3.00# make

注意：编译需要的时间比较长，请耐心等待编译完成

2.2.2.6 安装nginx

bash-3.00# make install

完成之后，会生成一个run目录：/opt/tchzt/server/nginx-1.8.0/run/
这里的所有内容就是我们真正使用的。

这个run目录使我们在

bash-3.00# ./configure --add-module=nginx_tcp_proxy_module-master  --with-pcre=pcre-8.32 --with-openssl=openssl-1.0.2a --with-zlib=zlib-1.2.8 --with-poll_module --prefix=/opt/tchzt/server/nginx-1.8.0/run

的时候设置的，当然也可以设置成其他的。

2.2.2.7 启动nginx

检查配置文件是否正确：

bash-3.00# cd ./opt/tchzt/server/nginx-1.8.0/run/sbin/
bash-3.00# ./nginx –t

如果出现以下信息，说明/opt/tchzt/server/nginx-1.8.0/run/conf/nginx.conf没有问题：

如果报错，则需要查看并修改配置文件。

启动nginx：

bash-3.00# ./nginx

查看/opt/tchzt/server/nginx-1.8.0/run/logs/nginx.pid，如果这个文件存在，则说明正常启动。

关闭nginx：

bash-3.00# ./nginx -s stop

查看/opt/tchzt/server/nginx-1.8.0/run/logs/nginx.pid，如果这个文件不存在，则说明正常关闭。

至此，nginx已经完全安装完毕。下面我们需要做的就是负载均衡的各种配置。

2.2.3 配置nginx

2.2.3.1 Nginx.config配置如下：

#nginx所用用户和组，window下不指定  
user  root;
#工作的子进程数量（通常等于CPU数量或者2倍于CPU）
worker_processes  3;
#错误日志存放路径 
error_log  logs/error.log  info;
#指定pid存放文件
pid        logs/nginx.pid;

events {
    #使用网络IO模型linux建议epoll，FreeBSD建议采用kqueue，window下不指定。  
    #use epoll; 
    #允许最大连接数 
    worker_connections  2048;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    #定义日志格式  
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  logs/access.log  main;

    sendfile        on;
    tcp_nopush      on;
    tcp_nodelay     on;

    keepalive_timeout  65;

    #导入解压缩命令的配置文件
    include    gzip.conf;  

    upstream localhost {  
        #根据ip计算将请求分配各那个后端tomcat，许多人误认为可以解决session问题，其实并不能。  
        #同一机器在多网情况下，路由切换，ip可能不同  
        #ip_hash;   
        server localhost:9088;  
        server localhost:9089;  
    }

    server {
        listen       8080;
        server_name  localhost;

        location / {
            proxy_connect_timeout   3;  
            proxy_send_timeout      30;  
            proxy_read_timeout      30;  
            proxy_pass http://localhost;  
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

tcp {
    upstream websocket {
        server localhost:28390;  
        server localhost:9082;  
        check interval=3000 rise=2 fall=5 timeout=1000;
    }

    server {
        listen 9080;
        proxy_pass websocket;
    }
}

2.2.3.2 gzip.conf

创建并配置解压缩命令的配置文件如下：

gzip              on;  
gzip_min_length   1000;  
gzip_types        text/plain text/css application/x-javascript;

2.2.3.3 Proxy.conf

创建并配置代理命令的配置文件如下：

proxy_redirect          off;  
proxy_set_header        Host $host;  
proxy_set_header        X-Real-IP $remote_addr;  
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;  
client_max_body_size    10m;  
client_body_buffer_size 128k;  
proxy_connect_timeout   300;  
proxy_send_timeout      300;  
proxy_read_timeout      300;  
proxy_buffer_size       4k;  
proxy_buffers           4 32k;  
proxy_busy_buffers_size 64k;  
proxy_temp_file_write_size 64k; 

2.2.4 配置tomcat

只需要修改server.xml文件即可，端口的修改要保证两个tomcat不一样。可以采取一个tomcat采用默认的端口，另一个tomcat采用修改的端口。

2.2.4.1 第一处需要修改的地方：

修改端口号：

<Server port="8005" shutdown="SHUTDOWN">

2.2.4.2 第二处需要修改的地方：

修改端口号，同时添加

               maxThreads="1500"
               maxSpareThreads="500" 
               minSpareThreads="100"
               acceptCount="1000"
               enableLookups="false"

添加之后如下：

<Connector port="8080" protocol="HTTP/1.1" 
               maxThreads="1500"
               maxSpareThreads="500" 
               minSpareThreads="100"
               acceptCount="1000"
               enableLookups="false"
              connectionTimeout="20000" 
              redirectPort="8443" />

2.2.4.3 第三处需要修改的地方：

修改端口号：

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

2.2.4.4 第四处需要修改的地方：

在engine中添加jvmRoute=”node”：

<Engine name="Catalina" defaultHost="localhost" jvmRoute="node">

2.2.4.5 第五处需要修改的地方：

在配置文件内添加：

<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
            <Manager className="org.apache.catalina.ha.session.DeltaManager" expireSessionsOnShutdown="false" notifyListenersOnReplication="true"/>
            <Channel className="org.apache.catalina.tribes.group.GroupChannel"> 
                <!--此处可以不做修改，Tomcat集群的默认组播地址就是228.0.0.4，默认端口为：45564-->  
                <Membership className="org.apache.catalina.tribes.membership.McastService" address="228.0.0.4" port="45564" frequency="500" dropTime="3000" mcastBindAddress="127.0.0.1" /> 
                <!--此处需要根据需要进行修改，address为Tomcat所在主机的IP地址，port为接收组播消息的端口，默认为4000到5000,可以在该范围内随意设置。确保集群内端口号唯一即可-->    
                <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver" address="127.0.0.1" port="4000" autoBind="100" selectorTimeout="5000" maxThreads="6"/>
                <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter"> 
                    <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender" /> 
                </Sender> 
                <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/> 
                <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/> 
                <Interceptor className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"/> 
            </Channel>
            <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/> 
            <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> 
            <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer" tempDir="/tmp/war-temp/" deployDir="/tmp/war-deploy/" watchDir="/tmp/war-listen/" watchEnabled="false"/> 
            <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/> 
            <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/> 
        </Cluster>

最终的tomcat的配置文件server.xml样式如下所示：

<?xml version='1.0' encoding='utf-8'?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<!-- Note:  A "Server" is not itself a "Container", so you may not
     define subcomponents such as "Valves" at this level.
     Documentation at /docs/config/server.html
 -->
<Server port="8005" shutdown="SHUTDOWN">

  <!--APR library loader. Documentation at /docs/apr.html -->
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <!--Initialize Jasper prior to webapps are loaded. Documentation at /docs/jasper-howto.html -->
  <Listener className="org.apache.catalina.core.JasperListener" />
  <!-- Prevent memory leaks due to use of particular java/javax APIs-->
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <!-- JMX Support for the Tomcat server. Documentation at /docs/non-existent.html -->
  <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />

  <!-- Global JNDI resources
       Documentation at /docs/jndi-resources-howto.html
  -->
  <GlobalNamingResources>
    <!-- Editable user database that can also be used by
         UserDatabaseRealm to authenticate users
    -->
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>

  <!-- A "Service" is a collection of one or more "Connectors" that share
       a single "Container" Note:  A "Service" is not itself a "Container", 
       so you may not define subcomponents such as "Valves" at this level.
       Documentation at /docs/config/service.html
   -->
  <Service name="Catalina">

    <!--The connectors can use a shared executor, you can define one or more named thread pools-->
    <!--
    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-" 
        maxThreads="150" minSpareThreads="4"/>
    -->


    <!-- A "Connector" represents an endpoint by which requests are received
         and responses are returned. Documentation at :
         Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
         Java AJP  Connector: /docs/config/ajp.html
         APR (HTTP/AJP) Connector: /docs/apr.html
         Define a non-SSL HTTP/1.1 Connector on port 8080
    -->
    <Connector port="8080" protocol="HTTP/1.1" 
               maxThreads="1500"
               maxSpareThreads="500" 
               minSpareThreads="100"
               acceptCount="1000"
               enableLookups="false" 
               connectionTimeout="20000" 
               redirectPort="8443" />
    <!-- A "Connector" using the shared thread pool-->
    <!--
    <Connector executor="tomcatThreadPool"
               port="8080" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443" />
    -->           
    <!-- Define a SSL HTTP/1.1 Connector on port 8443
         This connector uses the JSSE configuration, when using APR, the 
         connector should be using the OpenSSL style configuration
         described in the APR documentation -->
    <!--
    <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               clientAuth="false" sslProtocol="TLS" />
    -->

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />


    <!-- An Engine represents the entry point (within Catalina) that processes
         every request.  The Engine implementation for Tomcat stand alone
         analyzes the HTTP headers included with the request, and passes them
         on to the appropriate Host (virtual host).
         Documentation at /docs/config/engine.html -->

    <!-- You should set jvmRoute to support load-balancing via AJP ie :
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">         
    --> 
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="node">

      <!--For clustering, please take a look at documentation at:
          /docs/cluster-howto.html  (simple how to)
          /docs/config/cluster.html (reference documentation) -->
      <!--
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
      -->  
        <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
            <Manager className="org.apache.catalina.ha.session.DeltaManager" expireSessionsOnShutdown="false" notifyListenersOnReplication="true"/>
            <Channel className="org.apache.catalina.tribes.group.GroupChannel"> 
                <!--此处可以不做修改，Tomcat集群的默认组播地址就是228.0.0.4，默认端口为：45564-->  
                <Membership className="org.apache.catalina.tribes.membership.McastService" address="228.0.0.4" port="45564" frequency="500" dropTime="3000" mcastBindAddress="127.0.0.1" /> 
                <!--此处需要根据需要进行修改，address为Tomcat所在主机的IP地址，port为接收组播消息的端口，默认为4000到5000,可以在该范围内随意设置。确保集群内端口号唯一即可-->    
                <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver" address="192.168.50.130" port="4000" autoBind="100" selectorTimeout="5000" maxThreads="6"/>
                <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter"> 
                    <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender" /> 
                </Sender> 
                <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/> 
                <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/> 
                <Interceptor className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"/> 
            </Channel>
            <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/> 
            <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> 
            <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer" tempDir="/tmp/war-temp/" deployDir="/tmp/war-deploy/" watchDir="/tmp/war-listen/" watchEnabled="false"/> 
            <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/> 
            <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/> 
        </Cluster>
      <!-- The request dumper valve dumps useful debugging information about
           the request and response data received and sent by Tomcat.
           Documentation at: /docs/config/valve.html -->
      <!--
      <Valve className="org.apache.catalina.valves.RequestDumperValve"/>
      -->

      <!-- This Realm uses the UserDatabase configured in the global JNDI
           resources under the key "UserDatabase".  Any edits
           that are performed against this UserDatabase are immediately
           available for use by the Realm.  -->
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
             resourceName="UserDatabase"/>

      <!-- Define the default virtual host
           Note: XML Schema validation will not work with Xerces 2.2.
       -->
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">

        <!-- SingleSignOn valve, share authentication between web applications
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
        -->

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"  
               prefix="localhost_access_log." suffix=".txt" pattern="common" resolveHosts="false"/>
        -->

      </Host>
    </Engine>
  </Service>
</Server>

2.2.5 配置项目

根据你的项目具体情况，可能需要修改一些端口。

至此，所有的部署均部署完毕了。