Solr(9)Solr Index Replication on Ubuntu and Scala Client

Solr(9)Solr Index Replication on Ubuntu and Scala Client

1. Create One More Core
Go to the example directory, copy the core directory
> pwd
/opt/solr/example
> cp -r collection1 jobs
> rm -fr collection1/

Go to the example directory, start the server
> java -jar start.jar

Go to the console web UI
http://ubuntu-master:8983/solr/#/

Add Core -
name: jobs
instanceDir: /opt/solr/example/solr/jobs
dataDir: /opt/solr/example/solr/jobs/data
config: /opt/solr/example/solr/jobs/conf/solrconfig.xml
schema: /opt/solr/example/solr/jobs/conf/schema.xml

Add one Record to the Solr System
Jobs —> Documents —> Request-Handler —>Document Type (Solr Command Raw XML or JSON)
<add>
<doc>
<field name=“id”>1</field>
<field name=“title”>senior software engineer</field>
</doc>
</add>

It is not working, maybe because of the commit issue. I tried with JSON, it works.
{
“id”:”1”,
“title”:”software engineer"
}

Click on the “Query” Tab, you will get all your data from there.

2. Set up the Replicate Server
> scp -r ubuntu-master:/opt/solr/example/solr/jobs ./

Check the master configuration, search for “/replication”, adding these configuration
<lst name="master">
<str name="enable">${master.enable:false}</str>
<str name="replicateAfter">commit</str>
<str name="replicateAfter">startup</str>
<str name="confFiles">schema.xml,stopwords.txt</str>
</lst>
<lst name="slave">
<str name="enable">${slave.enable:false}</str>
<str name="masterUrl">${master.url:http://ubuntu-master:8983/solr/jobs}</str>
<str name="pollInterval">00:00:60</str>
<str name="httpConnTimeout">5000</str>
<str name="httpReadTimeout">10000</str>
</lst>

Also, change the auto commit time in the configuration.
<autoCommit>
<maxDocs>300000</maxDocs>
<!-- 5 minutes -->
<maxTime>300000</maxTime>
<openSearcher>true</openSearcher>
</autoCommit>

I will do the same thing on other solr configuration files on slaves. I have 2 slaves, ubuntu-dev1, ubuntu-dev2

Start the master with this command with the master option enabled
> java -Dmaster.enable=true -jar start.jar

Start the slaves on my slave servers.
> java -Dslave.enable=true -jar start.jar

From the Server Web UI Console, I can only see the replication is enabled.
http://ubuntu-master:8983/solr/#/jobs/replication

We can go to the slave console to check
http://ubuntu-dev1:8983/solr/#/jobs/query

Right now, I can add one more data on master and check if it gets indexed on the slaves.
Some console logging on slaves
529867 [snapPuller-10-thread-1] INFO org.apache.solr.handler.SnapPuller – Slave in sync with master.
589867 [snapPuller-10-thread-1] INFO org.apache.solr.handler.SnapPuller – Master's generation: 8
589868 [snapPuller-10-thread-1] INFO org.apache.solr.handler.SnapPuller – Slave's generation: 7
589869 [snapPuller-10-thread-1] INFO org.apache.solr.handler.SnapPuller – Starting replication process
589883 [snapPuller-10-thread-1] INFO org.apache.solr.handler.SnapPuller – Number of files in latest index in master: 52

After the process, we can search any latest data on slaves and masters.

3. Set up the Load Balance
I am running HA PROXY with the SOLR master, so I need to choose another port number, the configuration will be as follow:
listen solr_cluster 0.0.0.0:8984
acl master_methods method POST DELETE PUT
use_backend solr_master_backend if master_methods
default_backend solr_read_backends

backend solr_master_backend
server solr-master ubuntu-master:8983 check inter 5000 rise 2 fall 2

backend solr_read_backends
balance roundrobin
server solr-slave1 ubuntu-dev1:8983 check inter 5000 rise 2 fall 2
server solr-slave2 ubuntu-dev2:8983 check inter 5000 rise 2 fall 2

It is working well, we can check from here
http://ubuntu-master/haproxy-status

4. Build a Simple Client

https://github.com/takezoe/solr-scala-client
This class helps a lot. CaseClassMapper

package com.sillycat.jobsconsumer.persistence

import com.sillycat.jobsconsumer.models.Job
import com.sillycat.jobsconsumer.utilities.{IncludeConfig, IncludeLogger}
import jp.sf.amateras.solr.scala.SolrClient
import jp.sf.amateras.solr.scala.sample.Param


/**
* Created by carl on 8/6/15.
*/
object SolrClientDAO extends IncludeLogger with IncludeConfig{

private val solrClient = {
try {
logger.info("Init the SOLR Client ---------------")
val solrURL = config.getString(envStr("solr.url.jobs"))
logger.info("SOLR URL = " + solrURL)
val client = new SolrClient(solrURL)
client
} catch {
case x: Throwable =>
logger.error("Couldn't connect to SOLR: " + x)
null
}
}

def releaseResource = {
if(solrClient != null){
solrClient.shutdown()
}
}


def addJob(job:Job): Unit ={
//logger.debug("Adding job (" + job + ") to solr")
solrClient.add(job)
}

def query(query:String):Seq[Job] = {
logger.debug("Fetching the job results with query = " + query)
val result = solrClient.query(query).getResultAs[Job]()
result.documents
}

def commit = {
solrClient.commit()
}

}

The dependency will be as follow:
//for solr scala driver
resolvers += "amateras-repo" at "http://amateras.sourceforge.jp/mvn/"

"jp.sf.amateras.solr.scala" %% "solr-scala-client" % "0.0.12",

And the Test Class is as follow:
package com.sillycat.jobsconsumer.persistence

import com.sillycat.jobsconsumer.models.Job
import com.sillycat.jobsconsumer.utilities.IncludeConfig
import org.scalatest.{BeforeAndAfterAll, Matchers, FunSpec}
import redis.embedded.RedisServer

/**
* Created by carl on 8/7/15.
*/
class SolrDAOSpec extends FunSpec with Matchers with BeforeAndAfterAll with IncludeConfig{

override def beforeAll() {
if(config.getString("build.env").equals("test")){

}
}

override def afterAll() {

}

describe("SolrDAO") {
describe("#add and query"){
it("Add one single job to Solr") {
val expect = Job("id1","title1","desc1","industry1")

val num = 10000
val start = System.currentTimeMillis()
for ( i<- 1 to num){
val job = (Job("id" + i, "title" + i, "desc" + i, "industry" + i))
SolrClientDAO.addJob(job)
}

val end = System.currentTimeMillis()

println("total time for " + num + " is " + (end-start))
println("it is " + num / ((end-start)/1000) + " jobs/second")


// SolrDAO.commit
// val result = SolrDAO.query("title:title1")
// result should not be (null)
// result.size > 0 should be (true)
// result.foreach { item =>
// println(item.toString + "\n")
// }
}
}
}

}


Clean all the data during testing
http://ubuntu-master:8983/solr/jobs/update?stream.body=%3Cdelete%3E%3Cquery%3E*:*%3C/query%3E%3C/delete%3E&commit=true

Actually the data schema is stored and defined in conf/schema.xml, I should update as follow:
<field name="title" type="text_general" indexed="true" stored="true" multiValued="false"/>
<field name="desc" type="text_general" indexed="true" stored="true" multiValued="false"/>
<field name="industry" type="text_general" indexed="true" stored="true" multiValued="false"/>


add single job at one time
total time for 10000 is 180096
it is 55 jobs/second

Find the log4j.properties here and change the log level
/opt/solr/example/resources/log4j.properties

I turned off the logging and used 2 threads on the clients, I get performance about below on each.
total time for 10000 is 51688
it is 196 jobs/second

The performance is as follow for single threads
total time for 10000 is 28398
it is 357 jobs/second


References:
Setup Scaling Servers
http://blog.csdn.net/thundersssss/article/details/5385699
http://lutaf.com/197.htm
http://blog.warningrc.com/2013/06/10/Solr-data-backup.html

Single mode on Jetty
http://sillycat.iteye.com/blog/2227398

load balance on the slaves
http://davehall.com.au/blog/dave/2010/03/13/solr-replication-load-balancing-haproxy-and-drupal
https://gist.github.com/feniix/1974460
http://stackoverflow.com/questions/10090386/how-to-check-solr-healthy-using-haproxy

solr clients
https://github.com/takezoe/solr-scala-client
https://wiki.apache.org/solr/Solrj