JanusGraph java 数据导入_JanusGraph批量导入数据代码总结

最新推荐文章于 2022-04-30 18:04:12 发布
最新推荐文章于 2022-04-30 18:04:12 发布
阅读量559 收藏
点赞数
文章标签： JanusGraph java 数据导入
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_30097621/article/details/114919506
版权

这里写自定义目录标题

1. Json导入到本地TinkerGraph

2. CSV导入到本地TinkerGraph

3. Json导入到分布式存储(berkeleyje-es)

本文中的代码基于janusgraph 0.3.1进行演示。数据文件都为janusgraph包中自带的数据文件。

1. Json导入到本地TinkerGraph

1.1 配置

conf/hadoop-graph/hadoop-load-json.properties 配置如下:

#

# Hadoop Graph Configuration

#

gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph

gremlin.hadoop.graphReader=org.apache.tinkerpop.gremlin.hadoop.structure.io.graphson.GraphSONInputFormat

gremlin.hadoop.graphWriter=org.apache.hadoop.mapreduce.lib.output.NullOutputFormat

gremlin.hadoop.inputLocation=./data/grateful-dead.json

gremlin.hadoop.outputLocation=output

gremlin.hadoop.jarsInDistributedCache=true

#

# SparkGraphComputer Configuration

#

spark.master=local[*]

spark.executor.memory=1g

spark.serializer=org.apache.spark.serializer.KryoSerializer

spark.kryo.registrator=org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoRegistrator

1.2 样例Json

{"id":1,"label":"song","inE":{"followedBy":[{"id":3059,"outV":153,"properties":{"weight":1}},{"id":276,"outV":5,"properties":{"weight":2}},{"id":3704,"outV":3,"properties":{"weight":2}},{"id":4383,"outV":62,"pr

operties":{"weight":1}}]},"outE":{"followedBy":[{"id":0,"inV":2,"properties":{"weight":1}},{"id":1,"inV":3,"properties":{"weight":2}},{"id":2,"inV":4,"properties":{"weight":1}},{"id":3,"inV":5,"properties":{"we

ight":1}},{"id":4,"inV":6,"properties":{"weight":1}}],"sungBy":[{"id":7612,"inV":340}],"writtenBy":[{"id":7611,"inV":527}]},"properties":{"name":[{"id":0,"value":"HEY BO DIDDLEY"}],"songType":[{"id":2,"value":"

cover"}],"performances":[{"id":1,"value":5}]}}

{"id":2,"label":"song","inE":{"followedBy":[{"id":0,"outV":1,"properties":{"weight":1}},{"id":323,"outV":34,"properties":{"weight":1}}]},"outE":{"followedBy":[{"id":6190,"inV":123,"properties":{"weight":1}},{"i

d":6191,"inV":50,"properties":{"weight":1}}],"sungBy":[{"id":7666,"inV":525}],"writtenBy":[{"id":7665,"inV":525}]},"properties":{"name":[{"id":3,"value":"IM A MAN"}],"songType":[{"id":5,"value":"cover"}],"perfo

rmances":[{"id":4,"value":1}]}}

s

1.3 代码

readGraph = GraphFactory.open('conf/hadoop-graph/hadoop-load-json.properties')

writeGraphConf = new BaseConfiguration()

writeGraphConf.setProperty("gremlin.graph", "org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph")

writeGraphConf.setProperty("gremlin.tinkergraph.graphFormat", "gryo")

writeGraphConf.setProperty("gremlin.tinkergraph.graphLocation", "/tmp/csv-graph.kryo")

blvp = BulkLoaderVertexProgram.build().bulkLoader(OneTimeBulkLoader).writeGraph(writeGraphConf).create(readGraph)

readGraph.compute(SparkGraphComputer).workers(1).program(blvp).submit().get()

1.4 文件校验

新生成的文件如下

[root@vm03 data]# ls -l /tmp/csv-graph.kryo

-rw-r--r--. 1 root root 726353 May 29 04:09 /tmp/csv-graph.kryo

2. CSV导入到本地TinkerGraph

2.1 配置

conf/hadoop-graph/hadoop-load-csv.properties 配置如下:

#

# Hadoop Graph Configuration

#

gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph

gremlin.hadoop.graphReader=org.apache.tinkerpop.gremlin.hadoop.structure.io.script.ScriptInputFormat

gremlin.hadoop.graphWriter=org.apache.tinkerpop.gremlin.hadoop.structure.io.graphson.GraphSONOutputFormat

gremlin.hadoop.inputLocation=./data/grateful-dead.txt

gremlin.hadoop.outputLocation=output

gremlin.hadoop.jarsInDistributedCache=true

gremlin.hadoop.scriptInputFormat.script=./data/script-input-grateful-dead.groovy

#

# SparkGraphComputer Configuration

#

spark.master=local[*]

spark.executor.memory=1g

spark.serializer=org.apache.spark.serializer.KryoSerializer

spark.kryo.registrator=org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoRegistrator

2.2 样例CSV

1,song,HEY BO DIDDLEY,cover,5 followedBy,2,1|followedBy,3,2|followedBy,4,1|followedBy,5,1|followedBy,6,1|sungBy,340|writtenBy,527 followedBy,3,2|followedBy,5,2|followedBy,62,1|followedBy,153,1

2,song,IM A MAN,cover,1 followedBy,50,1|followedBy,123,1|sungBy,525|writtenBy,525 followedBy,1,1|followedBy,34,1

3,song,NOT FADE AWAY,cover,531 followedBy,81,1|followedBy,86,5|followedBy,127,10|followedBy,59,1|followedBy,83,3|followedBy,103,2|followedBy,68,1|followedBy,134,2|followedBy,131,1|followedBy,151,1|followedBy,3

2.3 代码

script-input-grateful-dead.groovy 代码如下：

def parse(line) {

def (vertex, outEdges, inEdges) = line.split(/\t/, 3)

def (v1id, v1label, v1props) = vertex.split(/,/, 3)

def v1 = graph.addVertex(T.id, v1id.toInteger(), T.label, v1label)

switch (v1label) {

case "song":

def (name, songType, performances) = v1props.split(/,/)

v1.property("name", name)

v1.property("songType", songType)

v1.property("performances", performances.toInteger())

break

case "artist":

v1.property("name", v1props)

break

default:

throw new Exception("Unexpected vertex label: ${v1label}")

}

[[outEdges, true], [inEdges, false]].each { def edges, def out ->

edges.split(/\|/).grep().each { def edge ->

def parts = edge.split(/,/)

def otherV, eLabel, weight = null

if (parts.size() == 2) {

(eLabel, otherV) = parts

} else {

(eLabel, otherV, weight) = parts

}

def v2 = graph.addVertex(T.id, otherV.toInteger())

def e = out ? v1.addOutEdge(eLabel, v2) : v1.addInEdge(eLabel, v2)

if (weight != null) e.property("weight", weight.toInteger())

}

}

return v1

}

janusgraph代码：

readGraph = GraphFactory.open('conf/hadoop-graph/hadoop-load-csv.properties')

writeGraphConf = new BaseConfiguration()

writeGraphConf.setProperty("gremlin.graph", "org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph")

writeGraphConf.setProperty("gremlin.tinkergraph.graphFormat", "gryo")

writeGraphConf.setProperty("gremlin.tinkergraph.graphLocation", "/tmp/csv-graph2.kryo")

blvp = BulkLoaderVertexProgram.build().bulkLoader(OneTimeBulkLoader).writeGraph(writeGraphConf).create(readGraph)

readGraph.compute(SparkGraphComputer).workers(1).program(blvp).submit().get()

g = GraphFactory.open(writeGraphConf).traversal()

g.V().valueMap(true)

2.4 文件校验

新生成的文件如下

[root@vm03 data]# ls -l /tmp/csv-graph2.kryo

-rw-r--r--. 1 root root 339939 May 29 04:56 /tmp/csv-graph2.kryo

3. Json导入到分布式存储(berkeleyje-es)

3.1 配置

conf/hadoop-graph/hadoop-load-json-ber-es.properties 配置如下:

#

# Hadoop Graph Configuration

#

gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph

gremlin.hadoop.graphReader=org.apache.tinkerpop.gremlin.hadoop.structure.io.graphson.GraphSONInputFormat

gremlin.hadoop.graphWriter=org.apache.hadoop.mapreduce.lib.output.NullOutputFormat

gremlin.hadoop.inputLocation=./data/grateful-dead.json

gremlin.hadoop.outputLocation=output

gremlin.hadoop.jarsInDistributedCache=true

#

# SparkGraphComputer Configuration

#

spark.master=local[*]

spark.executor.memory=1g

spark.serializer=org.apache.spark.serializer.KryoSerializer

spark.kryo.registrator=org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoRegistrator

./conf/janusgraph-berkeleyje-es-bulkload.properties 配置如下：

gremlin.graph=org.janusgraph.core.JanusGraphFactory

storage.backend=berkeleyje

storage.directory=../db/berkeley

index.search.backend=elasticsearch

3.2 样例Json

{"id":1,"label":"song","inE":{"followedBy":[{"id":3059,"outV":153,"properties":{"weight":1}},{"id":276,"outV":5,"properties":{"weight":2}},{"id":3704,"outV":3,"properties":{"weight":2}},{"id":4383,"outV":62,"pr

operties":{"weight":1}}]},"outE":{"followedBy":[{"id":0,"inV":2,"properties":{"weight":1}},{"id":1,"inV":3,"properties":{"weight":2}},{"id":2,"inV":4,"properties":{"weight":1}},{"id":3,"inV":5,"properties":{"we

ight":1}},{"id":4,"inV":6,"properties":{"weight":1}}],"sungBy":[{"id":7612,"inV":340}],"writtenBy":[{"id":7611,"inV":527}]},"properties":{"name":[{"id":0,"value":"HEY BO DIDDLEY"}],"songType":[{"id":2,"value":"

cover"}],"performances":[{"id":1,"value":5}]}}

{"id":2,"label":"song","inE":{"followedBy":[{"id":0,"outV":1,"properties":{"weight":1}},{"id":323,"outV":34,"properties":{"weight":1}}]},"outE":{"followedBy":[{"id":6190,"inV":123,"properties":{"weight":1}},{"i

d":6191,"inV":50,"properties":{"weight":1}}],"sungBy":[{"id":7666,"inV":525}],"writtenBy":[{"id":7665,"inV":525}]},"properties":{"name":[{"id":3,"value":"IM A MAN"}],"songType":[{"id":5,"value":"cover"}],"perfo

rmances":[{"id":4,"value":1}]}}

s

3.3 代码

outputGraphConfig = './conf/janusgraph-berkeleyje-es-bulkload.properties'

readGraph = GraphFactory.open('conf/hadoop-graph/hadoop-load-json-ber-es.properties')

blvp = BulkLoaderVertexProgram.build().writeGraph(outputGraphConfig).create(readGraph)

readGraph.compute(SparkGraphComputer).workers(1).program(blvp).submit().get()

g = GraphFactory.open(outputGraphConfig).traversal()

g.V().valueMap(true)

3.4 验证

通过gremlin-server搭建服务进行验证

gremline-server配置文件如下(gremlin-server-berkeleyje-bulkload.yaml)，与gremlin-server-berkeleyje.yaml类似，下面的位置进行调整：

graph: conf/janusgraph-berkeleyje-es-bulkload.properties

./gremlin-server.sh conf/gremlin-server/gremlin-server-berkeleyje-bulkload.yaml 启动服务

通过graphexp进行查询

weixin_30097621
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
JanusGraph java 数据导入_JanusGraph实战笔记·数据写入·查询
weixin_32865039的博客
03-01 274
JanusGraph实战笔记·数据写入·查询发布时间：2018-08-28 11:24,浏览次数：1907, 标签：JanusGraphData Types·JanusGraph如何表示、写入和查询数组类型？JanusGraph·How to represent, write or query an array in JanusGraph?JanusGraph如何表示一个定点具有多个同名属性Jan...
janusgraph 引入 java_JanusGraph入门第一课和官方文档踩坑
weixin_42361764的博客
03-06 894
入门第一课是在IDEA里创建一个项目，有些小曲折。这里运行的Demo是读取janusgraph示例的“神之图”(Graph of the Gods)数据并打印，采用Hbase+ES组合，JanusGraph版本是0.4.0，诸位可参考以下开始janusgraph的调用之旅~官网Java依赖org.janusgraphjanusgraph-core0.4.0org.apache.tinkerpopg...
Janusgraph数据批量导入
weixin_46805835的博客
12-18 1241
Janusgraph数据批量导入其实很简单,Janusgraph自带了一个批量导入的教程就在下面这个文件里 janusgraph-0.3.3-hadoop2/data/grateful-dead-janusgraph-schema.groovy 说一下导入的大致流程 一：创建图架构 二:创建Json格式数据文件 三：配置存储Janusgraph的后端数据库Hbase 四:指定Json数据的文件路径 下面以Janusgraph自带的数据讲解具体讲解批量导入流程 一：创建图架构 Janusgraph自带的图架构
图解JanusGraph系列 - JanusGraph指标监控报警（Monitoring JanusGraph）
Tech Station
12-14 747
讲述如何对JanusGraph的指标收集和监控+报警配置
图解Janusgraph系列-查询图数据过程源码分析
Tech Station
12-14 2082
如何基于图数据库查询图数据过程？源码分析
基于Java的JanusGraph分布式图数据库设计源码
最新发布
05-25
本源码为基于Java的JanusGraph分布式图数据库设计，包含1292个Java文件、85个Markdown文件等，共1734个文件。该项目旨在为用户提供一个全面、便捷的分布式图数据库解决方案，通过Java、Shell、Python、CSS技术的结合...
janusgraph图数据库代码
10-29
使用java创建图数据库以及实现图数据库的增删改查，并存入hbase中
janusgraph.org:JanusGraph 网站
07-23
janusgraph.org 此 repo 生成在上提供的内容要进行更改，您应该安装本地设置，一旦您准备好提交更改，请通过您在 GitHub 上发布的分支提供指向可预览版本的指针。 请参阅以下说明以了​​解以下两者。在本地预览更改...
ldbc_snb_janusgraph:LDBC SNB在Janusgraph上| 非常感谢我的队友！
05-11
LDBC SNB在Janusgraph上Janusgraph实施UCAS大数据项目在JanusGraph上运行基准测试运行Janusgraph 使用此链接下载Janusgraph。 解压缩 在配置了后端存储的情况下运行Janusgraph，请参阅bin/janusgraph start|status|...
janusgraph-demo:janusgraph的api测试
03-10
janusgraph-demo:janusgraph的api测试
janusgraph 引入 java_java – 将数据添加到JanusGraph数据库
weixin_36455339的博客
02-26 607
我不得不使用JanusGraph从java程序中检索并添加数据.但我不知道如何连接到db.Below的特定图形是我用于在数据库中插入数据的代码但是当我尝试从gremlin控制台看到数据时,我找不到相同的数据.JanusGraph graph = JanusGraphFactory.open("conf/janusgraph-cassandra-es.properties");GraphTraver...
JanusGraph批量导入数据代码总结
不积跬步，无以至千里；不积小流，无以成江海！
06-03 9052
这里写自定义目录标题说明1. Json导入到本地TinkerGraph1.1 配置1.2 样例Json1.3 代码1.4 文件校验2. CSV导入到本地TinkerGraph2.1 配置2.2 样例CSV2.3 代码2.4 文件校验3. Json导入到分布式存储(berkeleyje-es)3.1 配置3.2 样例Json3.3 代码3.4 验证 说明 本文中的代码基于janusgraph 0.3...
龙族幻想导入数据id_关于JanusGraph图数据批量快速导入的方案和想法（bulk load data）...
weixin_39941732的博客
12-23 428
大家好，我是洋仔，JanusGraph图解系列文章，实时更新~ 图数据库文章总目录：整理所有图相关文章，请移步(超链)：【图数据库系列】-文章总目录_洋仔聊编程-CSDN博客​liyangyang.blog.csdn.net 源码分析相关可查看github（码文不易，求个star~）： https://github.com/YYDreamer/janusgraph​github.com版本：Ja...
java 连接janusgraph_JanusGraph技术总结
weixin_32827773的博客
12-26 1774
1 图数据库现状图数据库产品可以分为原生和多模两类。原生指专门针对图数据存储和计算研发的产品，包括neo4jjanusGraph等，多模指多个大厂推出的数据库产品中能够兼容处理图数据。多模形态产品的主要问题是针对图谱数据做兼容处理，用于适配其底层的存储机制。原生图数据库对在技术的专注度上会更加集中，包括图计算引擎实现和图存储算法。分析DBEngine的图数据库排名。得分最高的是neo4j，长期保持...
图解JanusGraph系列 - 关于JanusGraph图数据批量快速导入的方案和想法（bulk load data）
Tech Station
04-30 2791
依托官网聊一下关于图数据批量快速导入的一些方案、方案使用场景和一些想法
java 连接janusgraph_基于JanusGraph的大数据图数据库
weixin_34254464的博客
12-01 1372
导读 知识图谱是近来很火的概念，很多领域都希望能用知识图谱解决一些问题。在零售领域其实也有使用知识图谱的场景，比如阿里使用知识图谱进行商品智能导购以及商品“巡检”，而我想试验着把知识图谱应用到精准营销的场景。先抛开知识图谱复杂的概念不谈，先来了解下图数据库——janusGraph。 目前比较火的图数据库一个是Neo4j，它是原生的图数据库，即查询存储都自己完成，但是社区版只支持单机；而...
Titan/JanusGraph的Hadoop-Gremlin批量数据导入注意事项
liliang199的专栏
09-27 5041
Titan/JanusGraph的Hadoop-Gremlin海量数据批量导入需要注意的问题
janusgraph 引入 java,janusgraph 默认server使用手记
weixin_33140809的博客
03-19 207
启动server(with Cassandra/ES)janusgraph-0.3.0-hadoop2 bin/janusgraph.sh startForking Cassandra...Running `nodetool statusthrift`. OK (returned exit status 0 and printed string "running").Forking Elastic...
janusgraph 引入 java,从头开始学JanusGraph 0.4
weixin_42666807的博客
03-19 443
介绍JanusGraph支持三种数据库后端: Apache Cassandra,Apache HBase, andOracle Berkeley DB Java EditionBerkeley DB是非分布式的数据库, 仅在学习和测试中使用HBase保证一致性Cassandra保证可用性JanusGraph的优势支持数据量巨大的图形支持巨大的并发事务和操作图形处理。JanusGraph事务容量...
janusgraph如何导入数据
04-24
JanusGraph可以通过使用Gremlin或TinkerPop接口从各种数据源导入数据。通常，您将数据存储在一个运行Apache Cassandra或Apache HBase的分布式存储系统中，并使用JanusGraph的数据导入工具将数据导入JanusGraph。具体...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值