Installation of Mongodb Cluster in Replica Mode[HA]

Before You Install

Concept

Replica sets are a processes of synchronizing data on multiple machines, they provide data redundancy and expands data availability. Storing data on multiple servers can avoid data loss caused by single server. Which can also be freed from hardware failures or service interruptions, and can be dedicated to disaster recovery or backup from a single machine with additional data copies.
A mongodb replica set is a set of mongodb instances with the same data. The primary node receives all write operations, and all other instances can receive the operation same as the primary to keep data synchronized.
Among the whole mongodb cluster only one primary instance exists and it can receive the clients’ write operation, the other secondary instances can only provide read operation.

Structure and Principle

The replica set of MongoDB is different from the previous master-slave mode.
When cluster Primary fails, the replica set can vote automatically, elect a new Primary, and guide the remaining Secondary servers to connect to the new Primary, which is transparent to the application. It means the replica set is a master-slave replication with failover function.
Official recommendation for MongoDB replica nodes is at least 3. It is recommended that the number of replica be odd, up to 12 replica nodes and up to 7 nodes participate in the election. Limitation of the number of replica nodes, mainly because too many replica nodes in a cluster increase the cost of replication, and drag down the cluster.
Overall performance. Too many replica nodes participate in the election, which will also increase the election time.

Configure Network Names

Configure each host in the cluster as follows to ensure that all members can communicate with each other:

1. Set the hostname to a unique name (not localhost).

2. Edit /etc/hosts with the IP address and fully qualified domain name (FQDN) of each host in the cluster and you must make the IP of each host in cluster STATIC in /etc/sysconfig/network-scripts/ifcfg-eth0.

   172.18.2.1**    node0(Primary)
   172.18.2.1**    node1(Secondary)
   172.18.2.1**    node2(Secondary)
   

   Note: But format like ‘node0.tocoding.com’ will be the best

3. Edit /etc/sysconfig/network with the FQDN of this host only:

   HOSTNAME=node0
   

4. Verify that each host consistently identifies to the network:
   a. Run uname -a and check that the hostname matches the output of the hostname command.
   b. Run /sbin/ifconfig and note the value of inet addr in the eth0 entry, for example:

   eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
         inet 172.18.2.1**  netmask 255.255.255.0  broadcast 172.18.2.255
   

   c. Run host -v -t A $(hostname) and verify that the output matches the hostname command.
   The IP address should be the same as reported by ifconfig for eth0.

Disabled the firewall and selinux

Disabling the Firewall

1. For iptables, save the existing rule set:

   $ sudo iptables-save > ~/firewall.rules
   

2. Disable the firewall:

   $ sudo systemctl disable firewalld
   $ sudo systemctl stop firewalld
   

Setting SELinux mode

Security-Enhanced Linux (SELinux) allows you to set access control through policies.
To set the SELinux mode, perform the following steps on each host.

1. Check the SELinux state:

   $ getenforce
   

2. If the output is either Permissive or Disabled, you can skip this task and continue on to Disabling the Firewall. If the output is enforcing, continue to the next step.
3. Open the /etc/selinux/config file (in some systems, the /etc/sysconfig/selinux file).
4. Change the line SELINUX=enforcing to SELINUX=permissive.

   $ sudo systemctl disable firewalld
   $ sudo systemctl stop firewalld
   

5. Save and close the file.
6. Restart your system or run the following command to disable SELinux immediately:

   $ setenforce 0
   

You can reenable SELinux by changing SELINUX=permissive back to SELINUX=enforcing in /etc/selinux/config (or /etc/sysconfig/selinux), and then running the following command to immediately switch to enforcing mode:

$ setenforce 1

Installation of Replica Set Mongodb

Install the mongod service

1. Configuring the repository:

   cat >> /etc/yum.repos.d/mongodb-org-3.6.repo << EOF
   [mongodb-org-3.4]
   name=MongoDB Repository
   baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.4/x86_64/
   gpgcheck=1
   enabled=1
   gpgkey=https://www.mongodb.org/static/pgp/server-3.4.asc
   EOF
   

2. Installed by yum command

   $ yum install -y mongodb-org
   

3. Verify the installation

   $ rpm -qa |grep mongodb 
   mongodb-org-3.2.10-1.el7.x86_64 
   mongodb-org-mongos-3.2.10-1.el7.x86_64 
   mongodb-org-shell-3.2.10-1.el7.x86_64 
   mongodb-org-server-3.2.10-1.el7.x86_64 
   mongodb-org-tools-3.2.10-1.el7.x86_64 
   $ rpm -ql mongodb-org-server 
   /etc/init.d/mongod 
   /etc/mongod.conf 
   /etc/sysconfig/mongod 
   /usr/bin/mongod 
   /usr/share/doc/mongodb-org-server-3.2.10 
   /usr/share/doc/mongodb-org-server-3.2.10/GNU-AGPL-3.0 
   /usr/share/doc/mongodb-org-server-3.2.10/MPL-2 
   /usr/share/doc/mongodb-org-server-3.2.10/README 
   /usr/share/doc/mongodb-org-server-3.2.10/THIRD-PARTY-NOTICES 
   /usr/share/man/man1/mongod.1 
   /var/lib/mongo /var/log/mongodb 
   /var/log/mongodb/mongod.log 
   /var/run/mongodb
   

4. Edit the configuration file

   $ vim /etc/mongod.conf
   

   # network interfaces
   net:
     port: 27017
     bindIp: 0.0.0.0  # Listen to local interface only, comment to listen on all interfaces.
   

5. Start the mongod service

   $ service mongod start
   

   [root@node0 ~]# netstat -nltp|grep mongo 
   tcp  0  0 0.0.0.0:27017     0.0.0.0:*    LISTEN      3989/mongod  
   

The installation directories are described as below:

• MongoDB data directory: /usr/local/mongodb/data
• MongoDB log directory: /usr/local/mongodb/log/mongo.log
• MongoDB configuration directory: /usr/local/mongodb/mongodb.conf

Configure the configuration file

1. The configuration file shown as below:

   Important: If you use a certain client software to connect a mongodb cluster in replica set mode the replSet name is REQUIRED.

   $ vim /usr/local/mongodb/mongodb.conf
   

   port=27017
   bind_ip=172.18.2.1**  //The certain IP for the host will be the best.           
   dbpath=/usr/local/mongodb/data
   logpath=/usr/local/mongodb/log/mongo.log
   pidfilepath=/usr/local/mongodb/mongo.pid
   fork=true
   logappend=true
   shardsvr=true
   directoryperdb=true
   #auth=true
   #keyFile =/usr/local/mongodb/keyfile
   replSet=hqmongodb
   

2. The control script of mongodb cluster describe as follows:

   $ vim /etc/init.d/mongodb
   

   #!/bin/sh
   # chkconfig: - 64 36
   # description:mongod
   case $1 in
   start)
   /usr/local/mongodb/bin/mongod --maxConns 20000 --config /usr/local/mongodb/mongodb.conf
   ;;
   stop)
   /usr/local/mongodb/bin/mongo 172.18.2.1**:27017/admin --eval "db.shutdownServer()"
   #/usr/local/mongodb/bin/mongo 172.18.2.1**:27017/admin --eval "db.auth('system', '123456');db.shutdownServer()"
   ;;
   status)
   /usr/local/mongodb/bin/mongo 172.18.2.1**:27017/admin --eval "db.stats()"
   #/usr/local/mongodb/bin/mongo 172.18.2.1**:27017/admin --eval "db.auth('system', '123456');db.stats()"
   ;; esac
   

3. Start the mongodb on each host

   $ /etc/init.d/mongodb start
   about to fork child process, waiting until server is ready for connections.
   forked process: 28211
   child process started successfully, parent exiting
   

• Normal exception:

  about to fork child process, waiting until server is ready for connections.
  forked process: 14229
  ERROR: child process failed, exited with error number 100
  

This is a common mistake in Mongod startup. When the mongod.lock file in /usr/local/mongodb/data is closed illegally, the mongod.lock file is not destroyed. When the lock file is checked at the second startup, the error is reported.
Solution: Delete the mongod.lock file in the /usr/local/mongodb/data directory, and then execute ./mongod -- repair:

$ rm -rf /usr/local/mongodb/data/mongod.lock 
$ mongod --repair

Configure the primary node

1. Connect to primary node by execute command:

   $ mongo 172.18.2.1**:27017
   

2. Initialize the primary node

   > rs.initiate()
   {
     "info2" : "no configuration specified. Using a default configuration for the set",
     "me" : "172.18.2.1**:27017",
     "ok" : 1
   }
   hqmongodb:OTHER> rs.conf()
   {
     "_id" : "hqmongodb",
     "version" : 1,
     "protocolVersion" : NumberLong(1),
     "members" : [
       {
         "_id" : 0,
         "host" : "172.18.2.1**:27017",
         "arbiterOnly" : false,
         "buildIndexes" : true,
         "hidden" : false,
         "priority" : 1,
         "tags" : {
            
         },
         "slaveDelay" : NumberLong(0),
         "votes" : 1
       }
     ],
     "settings" : {
       "chainingAllowed" : true,
       "heartbeatIntervalMillis" : 2000,
       "heartbeatTimeoutSecs" : 10,
       "electionTimeoutMillis" : 10000,
       "catchUpTimeoutMillis" : 2000,
       "getLastErrorModes" : {
          
       },
       "getLastErrorDefaults" : {
         "w" : 1,
         "wtimeout" : 0
       },
       "replicaSetId" : ObjectId("5932f142a55dc83eca86ea86")
     }
   }
   

3. Add replication node to the cluster

   hqmongodb:PRIMARY> rs.add("172.18.2.1**:27017")
   { "ok" : 1 }
   hqmongodb:PRIMARY> rs.add("172.18.2.1**:27017")
   { "ok" : 1 }
   

4. Set priority of the cluster nodes

   hqmongodb:PRIMARY> cfg = rs.conf()      
   {
     "_id" : "hqmongodb",
     "version" : 3,
     "protocolVersion" : NumberLong(1),
     "members" : [
       {
         "_id" : 0,
         "host" : "172.18.2.1**:27017",
         "arbiterOnly" : false,
         "buildIndexes" : true,
         "hidden" : false,
         "priority" : 1,
         "tags" : {
            
         },
         "slaveDelay" : NumberLong(0),
         "votes" : 1
       },
       {
         "_id" : 1,
         "host" : "172.18.2.1**:27017",
         "arbiterOnly" : false,
         "buildIndexes" : true,
         "hidden" : false,
         "priority" : 1,
         "tags" : {
            
         },
         "slaveDelay" : NumberLong(0),
         "votes" : 1
       },
       {
         "_id" : 2,
         "host" : "172.18.2.1**:27017",
         "arbiterOnly" : false,
         "buildIndexes" : true,
         "hidden" : false,
         "priority" : 1,
         "tags" : {
            
         },
         "slaveDelay" : NumberLong(0),
         "votes" : 1
       }
     ],
     "settings" : {
       "chainingAllowed" : true,
       "heartbeatIntervalMillis" : 2000,
       "heartbeatTimeoutSecs" : 10,
       "electionTimeoutMillis" : 10000,
       "catchUpTimeoutMillis" : 2000,
       "getLastErrorModes" : {
          
       },
       "getLastErrorDefaults" : {
         "w" : 1,
         "wtimeout" : 0
       },
       "replicaSetId" : ObjectId("5932f142a55dc83eca86ea86")
     }
   }
   hqmongodb:PRIMARY> cfg.members[0].priority = 1
   1
   hqmongodb:PRIMARY> cfg.members[1].priority = 1
   1
   hqmongodb:PRIMARY> cfg.members[2].priority = 2      //Set node which the id number is 2 as standby primary to execute the failover function.
   2
   hqmongodb:PRIMARY> rs.reconfig(cfg)                 //Confirm the configuration
   { "ok" : 1 }
   

Configure the secondary nodes

1. Connect to the host:

   [root@node1 ~]# mongo 172.18.2.1**:27017
   hqmongodb:SECONDARY> db.getMongo().setSlaveOk()   //Set the secondary node as readOnly. Only set the SECONDARY node.
   

Enable Login Authentication

1. Connect to the host:

   [root@node0 ~]# mongo 172.18.2.1**:27017
   

• Normal exception:"errmsg" : "not master and slaveOk=false"
  Solution: Execute command > rs.slaveOk();

2. Add administration account:

   hqmongodb:PRIMARY> use admin     
   switched to db admin   
   hqmongodb:PRIMARY> db.createUser({user:"system",pwd:"123456",roles:[{role:"root",db:"admin"}]})
   Successfully added user: {
     "user" : "system",
     "roles" : [
       {
         "role" : "root",
         "db" : "admin"
       }
     ]
   }
   hqmongodb:PRIMARY> db.auth('system','123456')  
   1
   hqmongodb:PRIMARY> db.createUser({user:'administrator', pwd:'123456', roles:[{ role: "userAdminAnyDatabase", db: "admin"}]});
   Successfully added user: {
     "user" : "administrator",
     "roles" : [
       {
         "role" : "userAdminAnyDatabase",
         "db" : "admin"
       }
     ]
   }
   hqmongodb:PRIMARY> db.auth('administrator','123456')      
   1
   hqmongodb:PRIMARY> show collections
   system.users
   system.version
   hqmongodb:PRIMARY> db.system.users.find()
   { "_id" : "admin.system", "user" : "system", "db" : "admin", "credentials" : { "SCRAM-SHA-1" : { "iterationCount" : 10000, "salt" : "uTGH9NI6fVUFXd2u7vu3Pw==", "storedKey" : "qJBR7dlqj3IgnWpVbbqBsqo6ECs=", "serverKey" : "pTQhfZohNh760BED7Zn1Vbety4k=" } }, "roles" : [ { "role" : "root", "db" : "admin" } ] }
   { "_id" : "admin.administrator", "user" : "administrator", "db" : "admin", "credentials" : { "SCRAM-SHA-1" : { "iterationCount" : 10000, "salt" : "zJ3IIgYCe4IjZm0twWnK2Q==", "storedKey" : "2UCFc7KK1k5e4BgWbkTKGeuOVB4=", "serverKey" : "eYHK/pBpf8ntrER1A8fiI+GikBY=" } }, "roles" : [ { "role" : "userAdminAnyDatabase", "db" : "admin" } ] }
   

   [root@node0 ～]# mongo 182.48.115.236:27017 -u system -p 123456 --authenticationDatabase admin
   [root@node0 ～]# mongo 182.48.115.236:27017 -u administrator -p 123456  --authenticationDatabase admin
   

3. Enable Authentication

   [root@node0 ~]# cd /usr/local/mongodb/            
   [root@node0 mongodb]# openssl rand -base64 21 > keyfile  
   [root@node0 mongodb]# chmod 600 /usr/local/mongodb/keyfile
   [root@node0 mongodb]# scp /usr/local/mongodb/keyfile node*:`pwd` 
   [root@node* ~]# vim /usr/local/mongodb/mongodb.conf    
   ......
   auth=true
   keyFile =/usr/local/mongodb/keyfile
   

4. Edit the script /etc/init.d/mongodb

   #!/bin/sh
   # chkconfig: - 64 36
   # description:mongod
   case $1 in
   start)
   /usr/local/mongodb/bin/mongod --maxConns 20000 --config /usr/local/mongodb/mongodb.conf
   ;;
   stop)
   #/usr/local/mongodb/bin/mongo 172.18.2.1**:27017/admin --eval "db.shutdownServer()"
   /usr/local/mongodb/bin/mongo 172.18.2.1**:27017/admin --eval "db.auth('system', '123456');db.shutdownServer()"
   ;;
   status)
   #/usr/local/mongodb/bin/mongo 172.18.2.1**:27017/admin --eval "db.stats()"
   /usr/local/mongodb/bin/mongo 172.18.2.1**:27017/admin --eval "db.auth('system', '123456');db.stats()"
   ;; esac
   

Verify the configuration

1. Connect to the host:

   [root@node0 ~]# mongo 172.18.2.1**:27017 -u system -p 123456 --authenticationDatabase admin
   

2. Check the status of the replica set

   hqmongodb:PRIMARY> rs.status()           
   {
     "set" : "hqmongodb",
     "date" : ISODate("2017-06-03T19:06:59.708Z"),
     "myState" : 1,
     "term" : NumberLong(2),
     "heartbeatIntervalMillis" : NumberLong(2000),
     "optimes" : {
       "lastCommittedOpTime" : {
         "ts" : Timestamp(1496516810, 1),
         "t" : NumberLong(2)
       },
       "appliedOpTime" : {
         "ts" : Timestamp(1496516810, 1),
         "t" : NumberLong(2)
       },
       "durableOpTime" : {
         "ts" : Timestamp(1496516810, 1),
         "t" : NumberLong(2)
       }
     },
     "members" : [
       {
         "_id" : 0,
         "name" : "172.18.2.1**:27017",
         "health" : 1,
         "state" : 1,                         //state=1 means the primary
         "stateStr" : "PRIMARY",              
         "uptime" : 138,
         "optime" : {
           "ts" : Timestamp(1496516810, 1),
           "t" : NumberLong(2)
         },
         "optimeDate" : ISODate("2017-06-03T19:06:50Z"),
         "infoMessage" : "could not find member to sync from",
         "electionTime" : Timestamp(1496516709, 1),
         "electionDate" : ISODate("2017-06-03T19:05:09Z"),
         "configVersion" : 4,
         "self" : true
       },
       {
         "_id" : 1,
         "name" : "172.18.2.1**:27017",
         "health" : 1,
         "state" : 2,
         "stateStr" : "SECONDARY",
         "uptime" : 116,
         "optime" : {
           "ts" : Timestamp(1496516810, 1),
           "t" : NumberLong(2)
         },
         "optimeDurable" : {
           "ts" : Timestamp(1496516810, 1),
           "t" : NumberLong(2)
         },
         "optimeDate" : ISODate("2017-06-03T19:06:50Z"),
         "optimeDurableDate" : ISODate("2017-06-03T19:06:50Z"),
         "lastHeartbeat" : ISODate("2017-06-03T19:06:59.533Z"),
         "lastHeartbeatRecv" : ISODate("2017-06-03T19:06:59.013Z"),
         "pingMs" : NumberLong(0),
         "syncingTo" : "172.18.2.1**:27017",
         "configVersion" : 4
       },
       {
         "_id" : 2,
         "name" : "172.18.2.1**:27017",
         "health" : 1,
         "state" : 2,
         "stateStr" : "SECONDARY",
         "uptime" : 189,
         "optime" : {
           "ts" : Timestamp(1496516810, 1),
           "t" : NumberLong(2)
         },
         "optimeDurable" : {
           "ts" : Timestamp(1496516810, 1),
           "t" : NumberLong(2)
         },
         "optimeDate" : ISODate("2017-06-03T19:06:50Z"),
         "optimeDurableDate" : ISODate("2017-06-03T19:06:50Z"),
         "lastHeartbeat" : ISODate("2017-06-03T19:06:59.533Z"),
         "lastHeartbeatRecv" : ISODate("2017-06-03T19:06:59.013Z"),
         "pingMs" : NumberLong(0),
         "syncingTo" : "172.18.2.1**:27017",
         "configVersion" : 4
       },
     ],
     "ok" : 1
   }
   

Restart the Host Member of Cluster

[root@node* ~]# /etc/init.d/mongodb stop
[root@node* ~]# /etc/init.d/mongodb start