解决 JanusGraph - but mismatched timestamps; no lock column contained our timestamp (2020-10-12T02:09:

1. 问题描述

启动Apache Atlas时,报错信息如下

Caused by: org.janusgraph.diskstorage.PermanentBackendException: Read 1 locks with our rid  97- 99- 49- 50- 49- 48- 50- 54- 49- 51- 56- 49- 54- 45-106-102-112- 49 but mismatched timestamps; no lock column contained our timestamp (2020-10-12T02:09:41.393999800Z)
	at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSeniority(ConsistentKeyLocker.java:528)
	at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:454)
	at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:118)
	at org.janusgraph.diskstorage.locking.AbstractLocker.checkLocks(AbstractLocker.java:351)
	... 107 more

2. 解决办法

JDK版本问题,JDK11的时间精度不兼容此版本JanusGraph默认的时间精度,JAVA_HOME替换成JDK8启动正常(其他JDK版本也可,JDK8亲测可以)

3. 详细原因

从Java 11开始Instant.now()的默认时间精度从milis(毫秒)提高到nano(纳秒),这使得JanusGraphTimestampProviders无论实际配置了什么枚举值,始终使用NANO精度。而cassandra仍然使用MICRO精度,使得JanusGraph时间戳不兼容。

  • 不同JDK版本下运行测试运行
public static void main(String[] args) {
	Instant now = Instant.now();
	System.out.println("default: " + now);

    // JDK8 => out 默认为MILLTS精度
    //default: 2020-10-12T08:09:45.609Z

    // JDK11 => out 默认为NANOS精度
    // default: 2020-10-12T08:05:21.304999800Z
}
  • 以下参考 JanusGraph-issues https://github.com/JanusGraph/janusgraph/issues/1979

Starting from Java 11 the default precision of Instant.now() has been raised for some OSes from milis to nano. See similar issue: h2database/h2database#1178
This makes JanusGraph TimestampProviders always uses NANO precision no matter what enum value is actually configured. Still cassandra uses MICRO precision which makes Janus timestamps incompatible - see stacktrace below.
Fixed version of TimestampProviders:

MICRO {
        @Override
        public Instant getTime() {
            // original => return Instant.now();
            return Instant.now().truncatedTo(ChronoUnit.MICROS);
        }

        @Override
        public Instant getTime(long sinceEpoch) {
            return Instant.ofEpochSecond(0, (sinceEpoch * 1000L));
        }

        @Override
        public ChronoUnit getUnit() {
            return ChronoUnit.MICROS;
        }

        @Override
        public long getTime(Instant timestamp) {
            return timestamp.getEpochSecond() * 1000000L + timestamp.getNano()/1000;

        }
    },

    MILLI {
        @Override
        public Instant getTime() {
            // original => return Instant.now();
            return Instant.now().truncatedTo(ChronoUnit.MILLIS);
        }

        @Override
        public Instant getTime(long sinceEpoch) {
            return Instant.ofEpochMilli(sinceEpoch);
        }

        @Override
        public ChronoUnit getUnit() {
            return ChronoUnit.MILLIS;
        }

        @Override
        public long getTime(Instant timestamp) {
            return timestamp.getEpochSecond() * 1000 + timestamp.getNano() / 1000000;
        }
    };

Note lines:
return Instant.now().truncatedTo(ChronoUnit.MICROS);
return Instant.now().truncatedTo(ChronoUnit.MILLIS);
The above two lines fix the issue.

4. 详细错误堆栈信息如下

2020-10-12 10:09:51,410 ERROR - [main:] ~ Could not commit transaction [1] due to storage exception in system-commit (StandardJanusGraph:724)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
	at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56)
	at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91)
	at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:139)
	at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196)
	at org.janusgraph.diskstorage.BackendTransaction.commit(BackendTransaction.java:150)
	at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:721)
	at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1425)
	at org.janusgraph.graphdb.database.management.ManagementSystem.commit(ManagementSystem.java:195)
	at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphManagement.commit(AtlasJanusGraphManagement.java:129)
	at org.apache.atlas.repository.graph.GraphBackedSearchIndexer.commit(GraphBackedSearchIndexer.java:943)
	at org.apache.atlas.repository.graph.GraphBackedSearchIndexer.initialize(GraphBackedSearchIndexer.java:363)
	at org.apache.atlas.repository.graph.GraphBackedSearchIndexer.<init>(GraphBackedSearchIndexer.java:129)
	at org.apache.atlas.repository.graph.GraphBackedSearchIndexer.<init>(GraphBackedSearchIndexer.java:116)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122)
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:271)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1198)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1100)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1314)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1280)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveMultipleBeans(DefaultListableBeanFactory.java:1178)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1094)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1064)
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1198)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1100)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1136)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1064)
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1198)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1100)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1314)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1280)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveMultipleBeans(DefaultListableBeanFactory.java:1178)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1094)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1064)
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1198)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1100)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
	at org.apache.atlas.web.setup.KerberosAwareListener.contextInitialized(KerberosAwareListener.java:31)
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:843)
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533)
	at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:816)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:778)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
	at org.eclipse.jetty.server.Server.start(Server.java:422)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.Server.doStart(Server.java:389)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.apache.atlas.web.service.EmbeddedServer.start(EmbeddedServer.java:98)
	at org.apache.atlas.Atlas.main(Atlas.java:133)
Caused by: org.janusgraph.diskstorage.locking.PermanentLockingException: Permanent locking failure
	at org.janusgraph.diskstorage.locking.AbstractLocker.checkLocks(AbstractLocker.java:359)
	at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.checkAllLocks(ExpectedValueCheckingTransaction.java:175)
	at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.prepareForMutations(ExpectedValueCheckingTransaction.java:154)
	at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:72)
	at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94)
	at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91)
	at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68)
	at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54)
	... 100 more
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Read 1 locks with our rid  97- 99- 49- 50- 49- 48- 50- 54- 49- 51- 56- 49- 54- 45-106-102-112- 49 but mismatched timestamps; no lock column contained our timestamp (2020-10-12T02:09:41.393999800Z)
	at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSeniority(ConsistentKeyLocker.java:528)
	at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:454)
	at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:118)
	at org.janusgraph.diskstorage.locking.AbstractLocker.checkLocks(AbstractLocker.java:351)
	... 107 more
备注:

刚开始接触数据中台,需要搭建相关开源项目,现在在排坑阶段,搭建过程中如遇问题可以私信讨论~

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值