Configuring a Liferay cluster

From:http://blogs.aca-it.be/blogs/-/blogs/configuring-a-liferay-cluster

Introduction

Configuring a Liferay cluster is part experience and part black magic. There is some information that you can find online, there's some information you can only find out while working on it and then there are some things like how to configure ehcache to use unicast that you can only discover through blood, sweat and tears. This post will first describe how to set up a Liferay 6.1 cluster with Ehcache in multicast and in unicast mode.

To get clustering to work in Liferay you need to make sure that all of the subsystems below are configured correctly:

• Database

• Indexing

• Media gallery

• Quartz

• Cluster Link

• Ehcache

 

Database

The first subsystem that needs to be configured for clustering, the database, is also one of the easiest to configure correctly. You just need to point each node in the cluster to the same database, either by using the same JNDI datasource

jdbc.default.jndi.name=jdbc /liferay

or by using the same JDBC configuration directly in your portal-ext.properties on each node

jdbc.default.driverClassName=com.mysql.jdbc.Driver jdbc.default.url=jdbc:mysql: //dbserver :3306 /liferay_test ?useUnicode= true &characterEncoding=UTF-8&useFastDateParsing= false &autoReconnectForPools= true jdbc.default.username=dbuser jdbc.default.password=Y@r3FiL

 

Indexing

For Liferay 6.1 the only reliable way to cluster the indexing is to use SOLR. For this you'll need to do 2 things: set up a separate SOLR server (or use an existing one) and deploy a correctly configured solr-web.war from the Marketplace to all cluster nodes.

Depending on which Liferay flavor you're using, CE or EE, this will be an easy process or a little bit more difficult. If you're running Liferay EE, the process is pretty straightforward as for that version there are solr-web versions available for SOLR 3 and 4. For Liferay CE it's a bit more complicated as there's only a relatively old WAR available for SOLR 1.4, which you'll need to upgrade yourself if you want to use newer SOLR versions with Liferay CE.

For this blog we're assuming that a dedicated Liferay SOLR instance will be used (but an additional core in an existing SOLR will also work). To set up a SOLR, you can just follow the instructions on their site: https://wiki.apache.org/solr/SolrInstall. Once you have a default SOLR up and running, you'll need to add some configuration to it so Liferay can use it for indexing. This configuration is done by replacing the existing schema.xml with the Liferay SOLR schema.xml that you can find in the WEB-INF/conf directory of the solr-web.war you downloaded.

If you're running on Liferay 6.1 CE and want to use a newer SOLR version than 1.4, you'll also need to change the schema.xml and possibly also the solr-spring.xml a bit to get it working. The version of the schema.xml that worked for us is:

<? xml version = "1.0" ?> < schema name = "liferay" version = "1.1" >      < types >          < fieldType name = "string" class = "solr.StrField" sortMissingLast = "true" omitNorms = "true" />          < fieldType name = "boolean" class = "solr.BoolField" sortMissingLast = "true" omitNorms = "true" />          < fieldType name = "integer" class = "solr.IntField" omitNorms = "true" />          < fieldType name = "long" class = "solr.LongField" omitNorms = "true" />          < fieldType name = "float" class = "solr.FloatField" omitNorms = "true" />          < fieldType name = "double" class = "solr.DoubleField" omitNorms = "true" />          < fieldType name = "sint" class = "solr.SortableIntField" sortMissingLast = "true" omitNorms = "true" />          < fieldType name = "slong" class = "solr.SortableLongField" sortMissingLast = "true" omitNorms = "true" />          < fieldType name = "sfloat" class = "solr.SortableFloatField" sortMissingLast = "true" omitNorms = "true" />          < fieldType name = "sdouble" class = "solr.SortableDoubleField" sortMissingLast = "true" omitNorms = "true" />          < fieldType name = "date" class = "solr.DateField" sortMissingLast = "true" omitNorms = "true" />          < fieldType name = "text_ws" class = "solr.TextField" positionIncrementGap = "100" >              < analyzer >                  < tokenizer class = "solr.WhitespaceTokenizerFactory" />              </ analyzer >          </ fieldType >          < fieldType name = "text" class = "solr.TextField" positionIncrementGap = "100" >              < analyzer type = "index" >                  < tokenizer class = "solr.WhitespaceTokenizerFactory" />                  < filter class = "solr.StopFilterFactory" ignoreCase = "true" words = "stopwords.txt" />                  < filter class = "solr.WordDelimiterFilterFactory" generateWordParts = "1" generateNumberParts = "1" catenateWords = "1" catenateNumbers = "1" catenateAll = "0" />                  < filter class = "solr.LowerCaseFilterFactory" />                  < filter class = "solr.RemoveDuplicatesTokenFilterFactory" />              </ analyzer >              < analyzer type = "query" >                  < tokenizer class = "solr.WhitespaceTokenizerFactory" />                  < filter class = "solr.SynonymFilterFactory" synonyms = "synonyms.txt" ignoreCase = "true" expand = "true" />                  < filter class = "solr.StopFilterFactory" ignoreCase = "true" words = "stopwords.txt" />                  < filter class = "solr.WordDelimiterFilterFactory" generateWordParts = "1" generateNumberParts = "1" catenateWords = "0" catenateNumbers = "0" catenateAll = "0" />                  < filter class = "solr.LowerCaseFilterFactory" />                  < filter class = "solr.RemoveDuplicatesTokenFilterFactory" />              </ analyzer >          </ fieldType >          < fieldType name = "textTight" class = "solr.TextField" positionIncrementGap = "100" >              < analyzer >                  < tokenizer class = "solr.WhitespaceTokenizerFactory" />                  < filter class = "solr.SynonymFilterFactory" synonyms = "synonyms.txt" ignoreCase = "true" expand = "false" />                  < filter class = "solr.StopFilterFactory" ignoreCase = "true" words = "stopwords.txt" />                  < filter class = "solr.WordDelimiterFilterFactory" generateWordParts = "0" generateNumberParts = "0" catenateWords = "1" catenateNumbers = "1" catenateAll = "0" />                  < filter class = "solr.LowerCaseFilterFactory" />                  < filter class = "solr.RemoveDuplicatesTokenFilterFactory" />              </ analyzer >          </ fieldType >          < fieldType name = "alphaOnlySort" class = "solr.TextField" sortMissingLast = "true" omitNorms = "true" >              < analyzer >                  < tokenizer class = "solr.KeywordTokenizerFactory" />                  < filter class = "solr.LowerCaseFilterFactory" />                  < filter class = "solr.TrimFilterFactory" />                  < filter class = "solr.PatternReplaceFilterFactory" pattern = "([^a-z])" replacement = "" replace = "all" />              </ analyzer >          </ fieldType >          < fieldtype name = "ignored" stored = "false" indexed = "false" class = "solr.StrField" />      </ types >      < fields >          <!--              Had to add additional fields, 'dash' fields and 'copyfields' to make              tables and sorting work correctly in certain Control Panel pages:                  first-name, last-name, screen-name, job-title and type              otherwise you'll see the following errors in the SOLR log:                  Feb 15, 2013 11:20:23 AM org.apache.solr.common.SolrException log                  SEVERE: org.apache.solr.common.SolrException: can not sort on multivalued field: job-title                      at org.apache.solr.schema.SchemaField.checkSortability(SchemaField.java:160)              http://www.liferay.com/community/forums/-/message_boards/message/21525098              http://liferay-blogging.blogspot.be/2012/03/liferay-and-solr-solrexception-can-not.html           -->          < field name = "comments" type = "text" indexed = "true" stored = "true" />          < field name = "content" type = "text" indexed = "true" stored = "true" />          < field name = "description" type = "text" indexed = "true" stored = "true" />          < field name = "entryClassPK" type = "text" indexed = "true" stored = "true" />          < field name = "firstName" type = "text" indexed = "true" stored = "true" />          < field name = "first-name" type = "text" indexed = "true" stored = "true" />          < field name = "firstName_sortable" type = "string" indexed = "true" stored = "true" />          < field name = "job-title" type = "text" indexed = "true" stored = "true" />          < field name = "jobTitle_sortable" type = "string" indexed = "true" stored = "true" />          < field name = "lastName" type = "text" indexed = "true" stored = "true" />          < field name = "last-name" type = "text" indexed = "true" stored = "true" />          < field name = "lastName_sortable" type = "string" indexed = "true" stored = "true" />          < field name = "leftOrganizationId" type = "slong" indexed = "true" stored = "true" />          < field name = "name" type = "text" indexed = "true" stored = "true" />          < field name = "name_sortable" type = "string" indexed = "true" stored = "true" />          < field name = "properties" type = "string" indexed = "true" stored = "true" />          < field name = "rightOrganizationId" type = "slong" indexed = "true" stored = "true" />          < field name = "screen-name" type = "text" indexed = "true" stored = "true" />          < field name = "screenName_sortable" type = "string" indexed = "true" stored = "true" />          < field name = "title" type = "text" indexed = "true" stored = "true" />          < field name = "type" type = "text" indexed = "true" stored = "true" />          < field name = "type_sortable" type = "string" indexed = "true" stored = "true" />          < field name = "uid" type = "string" indexed = "true" stored = "true" />          < field name = "url" type = "string" indexed = "true" stored = "true" />          < field name = "userName" type = "string" indexed = "true" stored = "true" />          < field name = "version" type = "string" indexed = "true" stored = "true" />          <!--              http://liferay-blogging.blogspot.be/2012/03/liferay-and-solr-solrexception-can-not.html           -->          < field name = "modified" type = "text" indexed = "true" stored = "true" />          <!--              Added 'omitNorms' attribute on '*' to fix the following error:                       Liferay side:                               12:07:05,844 ERROR [SolrIndexWriterImpl:55] org.apache.solr.common.SolrException: Bad Request                           SOLR side:                               Jul 30, 2012 12:07:05 PM org.apache.solr.common.SolrException log                  SEVERE: org.apache.solr.common.SolrException:                  ERROR: [doc=PluginPackageIndexer_PORTLET_liferay/solr-web/6.1.0/war] cannot set an index-time boost, norms are omitted for field entryClassName: com.liferay.p          -->          < dynamicField name = "*CategoryNames" type = "string" indexed = "true" multiValued = "true" stored = "true" />          < dynamicField name = "*CategoryIds" type = "string" indexed = "true" multiValued = "true" stored = "true" />          < dynamicField name = "expando/*" type = "text" indexed = "true" multiValued = "true" stored = "true" />          < dynamicField name = "web_content/*" type = "text" indexed = "true" stored = "true" />          <!--          This must be the last entry since the fields element is an ordered set.          -->          < dynamicField name = "*" type = "string" indexed = "true" multiValued = "true" stored = "true" omitNorms = "false" />      </ fields >      < copyField source = "firstName" dest = "firstName_sortable" />      < copyField source = "first-name" dest = "firstName_sortable" />      < copyField source = "job-title" dest = "jobTitle_sortable" />      < copyField source = "lastName" dest = "lastName_sortable" />      < copyField source = "last-name" dest = "lastName_sortable" />      < copyField source = "name" dest = "name_sortable" />      < copyField source = "screen-name" dest = "screenName_sortable" />      < copyField source = "type" dest = "type_sortable" />      < uniqueKey >uid</ uniqueKey >      < defaultSearchField >content</ defaultSearchField >      < solrQueryParser defaultOperator = "OR" /> </ schema >

and for solr-spring.xml

<? xml version = "1.0" ?> < beans          default-destroy-method = "destroy"          default-init-method = "afterPropertiesSet"          xmlns = "         xmlns:xsi="         xmlns:jee="         xmlns:util="         xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd          http://www.springframework.org/schema/jee  http://www.springframework.org/schema/jee/spring-jee.xsd          http://www.springframework.org/schema/util  >               <bean class="com.liferay.portal.spring.context.PortletBeanFactoryPostProcessor" />     <!-- Solr search engine -->     <bean id="com.liferay.portal.search.solr.server.BasicAuthSolrServer" class="com.liferay.portal.search.solr.server.BasicAuthSolrServer">         <constructor-arg type="java.lang.String" value=" />     </bean>     <bean id="com.liferay.portal.search.solr.SolrIndexSearcherImpl" class="com.liferay.portal.search.solr.SolrIndexSearcherImpl">         <property name="solrServer" ref="com.liferay.portal.search.solr.server.BasicAuthSolrServer" />         <property name="swallowException" value="true" />     </bean>     <bean id="com.liferay.portal.search.solr.SolrIndexWriterImpl" class="com.liferay.portal.search.solr.SolrIndexWriterImpl">         <property name="commit" value="true" />         <property name="solrServer" ref="com.liferay.portal.search.solr.server.BasicAuthSolrServer" />     </bean>     <bean id="com.liferay.portal.search.solr.SolrSearchEngineImpl" class="com.liferay.portal.kernel.search.BaseSearchEngine">         <property name="clusteredWrite" value="false" />         <property name="indexSearcher" ref="com.liferay.portal.search.solr.SolrIndexSearcherImpl" />         <property name="indexWriter" ref="com.liferay.portal.search.solr.SolrIndexWriterImpl" />         <property name="luceneBased" value="true" />         <property name="vendor" value="SOLR" />     </bean>     <!-- Configurator -->     <bean id="searchEngineConfigurator.solr" class="com.liferay.portal.kernel.search.PluginSearchEngineConfigurator">         <property name="searchEngines">             <util:map>                 <entry key="SYSTEM_ENGINE" value-ref="com.liferay.portal.search.solr.SolrSearchEngineImpl" />             </util:map>         </property>     </bean> </beans>

Once you have SOLR up and running with the new schema, you just need to tweak the solr-web.war a little bit before deploying it on all nodes as it assumes SOLR is running on localhost:8080 which probably isn't the case. You can change this is in the solr-spring.xml file that you can find in the WEB-INF/classes/META-INF directory of the WAR file. Just change the constructor-arg value of the bean with id com.liferay.portal.search.solr.server.BasicAuthSolrServer so it points to the correct server and port.

 

Media gallery

Now that the document library and image gallery have been combined to form the media gallery in newer Liferay versions, the configuration to cluster it has also simplified. To cluster the media gallery you have 2 options: database or file system. There was also an option to useJackrabbit for this purpose, but this has been deprecated in Liferay 6.1.

Using the database is the simplest option as you only need to add one property to your portal-ext.properties file

dl.store.impl=com.liferay.portlet.documentlibrary.store.DBStore

This will automatically use the database that is already configured for Liferay to store al media items. As long a your database supports BLOBs of sufficient size to cover your media needs, this is an easy solution. But if your database has issues with large files, like videos for example, you can best use the second option: a common filesystem.

For this, you only need to configure a different store, usuallyeither com.liferay.portlet.documentlibrary.store.FileSystemStore 

or 
com.liferay.portlet.documentlibrary.store.AdvancedFileSystemStore (internally distributes lots of files over more directories to work around limitations of number of files per directory). To use a file system store correctly, you'll also need to configure the property dl.store.file.system.root.dir in your portal-ext.properties to point to a directory on the local filesystem of each node that points to a common file store, SAN, NAS, etc... . The problem with this is that the Liferay documentation doesn't exactly define what kind of file systems are supported or what kind of functionalities (locking, etc... ) they need to support. So it can be a bit of hit and miss finding one that works correctly.

Another option is to use the com.liferay.portlet.documentlibrary.store.CMISStore if you have an Alfresco instance to spare or the com.liferay.portlet.documentlibrary.store.S3Store if you have Amazon S3 buckets available. The only problem with these can be speed as they're usually slower than using a file system.

 

Quartz

The Quartz job scheduler that's available in Liferay also needs to be clustered to prevent problems. This can be done by adding the following line to your portal-ext.properties file:

org.quartz.jobStore.isClustered=true

Cluster Link

In a Liferay cluster all nodes need to be able to talk to each other to keep each other up to date. To enable this, you just need to activate the JGroups based Cluster Link system that's available in Liferay by adding the following two properties to your portal-ext.properties file:

cluster.link.enabled=true cluster.link.autodetect.address=dbserver:dbport

The second property is needed because otherwise the Cluster Link initialization during startup will possibly fail because by default it will try to contact google.com:80 and access to the internet isn't always possible in some environments. For that reason you'll need to have a host/port combination that is reachable by the cluster link and that it can be used to set up itself. The easiest option for this is to use thedatabase server and port that we already know (and can access) and use that to replace dbserver and dbport in the example above.

In order to make the JGroups Cluster Link you'll also need to set the following system properties for your JVM (for example via the JAVA_OPTS of Tomcat):

• -Djava.net.preferIPv4Stack=true
• -Djgroups.bind_addr=<local IP> (replace <local IP> on each node with the actual IP address of the node) 
• -Djgroups.tcpping.initial_hosts=<node 1>[7800],<node 2>[7800] (replace <node 1>, <node 2>, etc... on each node with the actual IP addresses of the corresponding nodes and add more values, separated with a comma if your cluster has more than 2 nodes)

Ehcache: multicast

In a Liferay cluster the different Ehcache based caches on a node also need to be aware of other nodes so that correct and up to date information is shown on nodes after something is changed on one node. When your server environment supports multicast (some virtualization software has issues with this) and your system administrators allow you to use it, it is pretty easy to configure Ehcache to work in a cluster. Just add the following lines to your portal-ext.properties on each node:

net.sf.ehcache.configurationResourceName=/ehcache/hibernate-clustered.xml ehcache.multi.vm.config.location=/ehcache/liferay-multi-vm-clustered.xml

When using multicast isn't possible you'll need to use the information in the next section of this blog.

Ehcache: unicast

In some server environments it might not be possible or allowed to use multicast. Unfortunately this is the default way of communication in a Liferay cluster and the only thoroughly documented way. So when we were faced with the task of setting up a cluster, while only using unicast, we had to do some Sherlock Holmes level investigations. After many hours of Googling, reading forums, trial and error, ... we were able to get it to work.

First off you need to create an JGroups configuration XML that will be the basis of the unicast setup. This XML is what will actually set up JGroups to use TCP instead of UDP. Once you have this file, you just need to provide it as configuration for a couple of properties and things will magically start working. Just create an XML file with the content below and name it unicast.xml (the name is not important as long as you remember to use the same value in the portal-ext.properties) and place it in the WEB-INF/classes directory of Liferay:

<!--     TCP based stack, with flow control and message bundling. This is usually used when IP     multicasting cannot be used in a network, e.g. because it is disabled (routers discard multicast).     Note that TCP.bind_addr and TCPPING.initial_hosts should be set, possibly via system properties, e.g.     -Djgroups.bind_addr=192.168.5.2 and -Djgroups.tcpping.initial_hosts=192.168.5.2[7800]     author: Bela Ban     version: $Id: tcp.xml,v 1.40 2009/12/18 09:28:30 belaban Exp $ --> <config xmlns="urn:org:jgroups"         xmlns:xsi="         xsi:schemaLocation="urn:org:jgroups >     <TCP singleton_name="liferay"          bind_port="7800"          loopback="true"          recv_buf_size="${tcp.recv_buf_size:20M}"          send_buf_size="${tcp.send_buf_size:640K}"          discard_incompatible_packets="true"          max_bundle_size="64K"          max_bundle_timeout="30"          enable_bundling="true"          use_send_queues="true"          sock_conn_timeout="300"          timer.num_threads="4"                     thread_pool.enabled="true"          thread_pool.min_threads="1"          thread_pool.max_threads="10"          thread_pool.keep_alive_time="5000"          thread_pool.queue_enabled="false"          thread_pool.queue_max_size="100"          thread_pool.rejection_policy="discard"          oob_thread_pool.enabled="true"          oob_thread_pool.min_threads="1"          oob_thread_pool.max_threads="8"          oob_thread_pool.keep_alive_time="5000"          oob_thread_pool.queue_enabled="false"          oob_thread_pool.queue_max_size="100"          oob_thread_pool.rejection_policy="discard"/>                                <TCPPING timeout="3000"              initial_hosts="${jgroups.tcpping.initial_hosts:localhost[7800],localhost[7801]}"              port_range="1"              num_initial_members="3"/>     <MERGE2  min_interval="10000"              max_interval="30000"/>     <FD_SOCK/>     <FD timeout="3000" max_tries="3" />     <VERIFY_SUSPECT timeout="1500"  />     <BARRIER />     <pbcast.NAKACK                    use_mcast_xmit="false" gc_lag="0"                    retransmit_timeout="300,600,1200,2400,4800"                    discard_delivered_msgs="true"/>     <UNICAST timeout="300,600,1200" />     <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"                    max_bytes="400K"/>     <pbcast.GMS print_local_addr="true" join_timeout="3000"                 view_bundling="true"/>     <FC max_credits="2M"         min_threshold="0.10"/>     <FRAG2 frag_size="60K"  />     <pbcast.STREAMING_STATE_TRANSFER/>     <!-- <pbcast.STATE_TRANSFER/> -->  </config>

Once you have this file in place you just need to add some additional configuration to your portal-ext.properties to configure the Liferay cluster link and Ehcache to use it:

cluster.link.channel.properties.control=unicast.xml cluster.link.channel.properties.transport.0=unicast.xml ehcache.bootstrap.cache.loader.factory=com.liferay.portal.cache.ehcache.JGroupsBootstrapCacheLoaderFactory ehcache.cache.event.listener.factory=net.sf.ehcache.distribution.jgroups.JGroupsCacheReplicatorFactory ehcache.cache.manager.peer.provider.factory=net.sf.ehcache.distribution.jgroups.JGroupsCacheManagerPeerProviderFactory net.sf.ehcache.configurationResourceName.peerProviderProperties=file=/unicast.xml ehcache.multi.vm.config.location.peerProviderProperties=file=/unicast.xml