Active MQ KahaDB Invalid location

0. Environment
OS: CentOS 5.4
Active MQ: 5.5.1
Active MQ configuration: 

</pre><p><pre name="code" class="html"><persistenceAdapter>
    <kahaDB directory="${activemq.base}/data/kahadb"/>
</persistenceAdapter>

 <transportConnectors>
            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
        </transportConnectors>

1. Problem Description

Pending messages in queue cannot be consumed and we can't delete data in Active MQ administration page.

In our production back end, we encountered this exception:

2015-01-19 12:39:43,813 | ERROR | QueueStorePrefetch383867497 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ Broker[localhost] Scheduler
java.lang.RuntimeException: java.io.IOException: Invalid location: 542164:30466873, : java.lang.NegativeArraySizeException
	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
	at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
	at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1028)
	at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:785)
	at org.apache.activemq.broker.region.Queue.access$100(Queue.java:84)
	at org.apache.activemq.broker.region.Queue$2.run(Queue.java:124)
	at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)
Caused by: java.io.IOException: Invalid location: 542164:30466873, : java.lang.NegativeArraySizeException
	at org.apache.kahadb.journal.DataFileAccessor.readRecord(DataFileAccessor.java:94)
	at org.apache.kahadb.journal.Journal.read(Journal.java:601)
	at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
	at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
	at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
	at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
	at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
	... 9 more
2015-01-19 12:39:43,813 | ERROR | Problem retrieving message for browse | org.apache.activemq.broker.region.Queue | ActiveMQ Broker[localhost] Scheduler
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Invalid location: 542164:30466873, : java.lang.NegativeArraySizeException
	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
	at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
	at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1028)
	at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:785)
	at org.apache.activemq.broker.region.Queue.access$100(Queue.java:84)
	at org.apache.activemq.broker.region.Queue$2.run(Queue.java:124)
	at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)
Caused by: java.lang.RuntimeException: java.io.IOException: Invalid location: 542164:30466873, : java.lang.NegativeArraySizeException
	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
	... 8 more
Caused by: java.io.IOException: Invalid location: 542164:30466873, : java.lang.NegativeArraySizeException
	at org.apache.kahadb.journal.DataFileAccessor.readRecord(DataFileAccessor.java:94)
	at org.apache.kahadb.journal.Journal.read(Journal.java:601)
	at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
	at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
	at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
	at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
	at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
	... 9 more

2. Analysis

3. Workaround

Good suggestion from Reference 3:

ActiveMQ v5.5.1 Corrupt data log found recovery: 

Symptom: 
The ActiveMQ slave process died and will not restarted. 

Root Cause: 
Corrupt data log found 

Root Cause verification: 
Search the affected ActiveMQ log file for the following entries in sequence: 
Corrupt journal records found 
Failed to discard data file 
Failed to start ActiveMQ JMS Message Broker 
shutting down 

Recovery: 
Shutdown the ActiveMQ master instance. 
Rename the Kaha db storage file 
Restart the ActiveMQ Master and Slave instances 

Note: the journal data affected by the corruption will be lost. The affected journal data will need to be identify and resent to the ActiveMQ appropriate queue.

4. Solution

Reference:

[1] http://activemq.apache.org/message-cursors.html

[2] https://issues.apache.org/jira/browse/AMQ-4943

[3] http://activemq.2283324.n4.nabble.com/KahaDB-corruption-td3321382.html