Titan 是一个在服务器集群搭建的分布式的图形数据库,特别为存储和处理大规模图形而优化。集群很容易扩展以支持更大的数据集,Titan有一个很好的插件式性能,这个性能让它搭建在一些成熟的数据库技术上像 Apache Cassandra、Apache HBase、 Oracle BerkeleyDB。插件式索引架构可以整合 ElasticSearch 和Lucene技术。内置实现 Blueprints graph API,支持 TinkerPop所有的技术。
特性
1、支持不同的分布式存储层
- Apache Cassandra (distributed)
- Apache HBase (distributed)
- Oracle BerkeleyDB (local)
- Persistit (local)
2、可以增加数据集的大小和用户基数弹性扩展
3、分布式存储的复制,高容错性
4、支持很多字符集和热备份
5、支持 ACID 和 eventual consistency(最终一致性)
6、支持的索引
- ElasticSearch
- Apache Lucene
7、内置实现 TinkerPop graph API
- Gremlin graph query language
- Frames object-to-graph mapper
- Rexster graph server
- Blueprints standard graph API
安装部署
下载解压titan-1.0.0-hadoop2.zip,测试环境版本:hadoop-2.7.2 + hbase-1.2.3 + elasticsearch-1.5.2
删除titan-1.0.0-hadoop2/lib目录下的hadoop-core-1.2.1.jar,拷贝titan-hadoop-1.0.0.jar,titan-hadoop-core-1.0.0.jar到titan-1.0.0-hadoop2/lib目录
可以使用命令$bin/titan.sh start测试是否安装成功
编辑conf/titan-hbase-es.properties文件
storage.backend=hbase
storage.hostname=host-10,host-11,host-12
index.search.backend=elasticsearch
index.search.hostname=host-10
index.search.elasticsearch.interface=TRANSPORT_CLIENT
index.search.elasticsearch.cluster-name=data
index.search.elasticsearch.client-only=true
Gremlin Shell测试,示例来自官方,在hbase,elasticsearch上都会生成一个titan库
$bin/gremlin.sh
gremlin>graph=TitanFactory.open('conf/titan-hbase-es.properties')
==>standardtitangraph[hbase:[host-115]]
gremlin>GraphOfTheGodsFactory.load(graph)
==>null
gremlin>g=graph.traversal()
graphtraversalsource[standardtitangraph[hbase:[host-115]], standard]
gremlin>saturn=g.V().has('name','saturn').next()
==>v[256]gremlin>g.V(saturn).valueMap()
==>[name:[saturn],age:[10000]]
gremlin>g.V(saturn).in('father').in('father').values('name')
==>hercules
gremlin>g.E().has('place',geoWithin(Geoshape.circle(37.97,23.72,50)))
==>e[a9x-co8-9hx-39s][16424-battled->4240]
==>e[9vp-co8-9hx-9ns][16424-battled->12520]
gremlin>g.E().has('place',geoWithin(Geoshape.circle(37.97,23.72,50))).as('source').inV().as('god2').select('source').outV().as('god1').select('god1','god2').by('name')
==>[god1:hercules,god2:hydra]
==>[god1:hercules,god2:nemean]
gremlin>hercules=g.V(saturn).repeat(__.in('father')).times(2).next()
==>v[1536]
gremlin>g.V(hercules).out('father','mother')
==>v[1024]
==>v[1792]
gremlin>g.V(hercules).out('father','mother').values('name')
==>jupiter
==>alcmene
gremlin>g.V(hercules).out('father','mother').label()
==>god
==>human
gremlin>hercules.label()
==>demigod
gremlin>g.V(hercules).out('father','mother')
==>v[1024]
==>v[1792]
gremlin>g.V(hercules).out('father','mother').values('name')
==>jupiter
==>alcmenegremlin>g.V(hercules).out('father','mother').label()
==>god
==>human
gremlin>hercules.label()
==>demigod
gremlin>g.V(hercules).outE('battled').has('time',gt(1)).inV().values('name').toString()==>[GraphStep([v[24744]],vertex),VertexStep(OUT,[battled],edge),HasStep([time.gt(1)]),EdgeVertexStep(IN),PropertiesStep
gremlin>pluto=g.V().has('name','pluto').next()
==>v[2048]
gremlin>g.V(pluto).out('lives').in('lives').values('name')
==>pluto
==>cerberus
gremlin>// pluto can't be his own cohabitant
gremlin>g.V(pluto).out('lives').in('lives').where(is(neq(pluto))).values('name')
==>cerberusgremlin>g.V(pluto).as('x').out('lives').in('lives').where(neq('x')).values('name')
==>cerberus
gremlin>// where do pluto's brothers live?
gremlin>g.V(pluto).out('brother').out('lives').values('name')
==>sky
==>sea
gremlin>// which brother lives in which place?
gremlin>g.V(pluto).out('brother').as('god').out('lives').as('place').select()
==>[god:v[1024],place:v[512]]
==>[god:v[1280],place:v[768]]
gremlin>// what is the name of the brother and the name of the place?
gremlin>g.V(pluto).out('brother').as('god').out('lives').as('place').select().by('name')
==>[god:jupiter,place:sky]
==>[god:neptune,place:sea]
gremlin>g.V(pluto).outE('lives').values('reason')
==>nofearofdeathgremlin>g.E().has('reason',textContains('loves'))
==>e[6xs-sg-m51-e8][1024-lives->512]
==>e[70g-zk-m51-lc][1280-lives->768]
gremlin>g.E().has('reason',textContains('loves')).as('source').values('reason').as('reason').select('source').outV().values('name').as('god').select('source').inV().values('name').as('thing').select('god','reason','thing')
==>[god:neptune,reason:loveswaves,thing:sea]
==>[god:jupiter,reason:lovesfreshbreezes,thing:sky]
Java API调用操作Titan
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<hadoop.version>2.7.2</hadoop.version>
<hbase.version>1.2.3</hbase.version>
<titan.version>1.0.0</titan.version>
</properties>
<!-- Guava依赖包 -->
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>16.0.1</version>
</dependency>
<!-- Elastic Search依赖包 -->
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>1.5.2</version>
<exclusions>
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- Hadoop依赖报 -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-annotations</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-auth</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-server-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
<!-- HBase依赖报 -->
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-common</artifactId>
<version>${hbase.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>${hbase.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>${hbase.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-protocol</artifactId>
<version>${hbase.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-annotations</artifactId>
<version>${hbase.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-hadoop2-compat</artifactId>
<version>${hbase.version}</version>
</dependency>
<!-- Titan依赖包 -->
<dependency>
<groupId>com.thinkaurelius.titan</groupId>
<artifactId>titan-core</artifactId>
<version>${titan.version}</version>
</dependency>
<dependency>
<groupId>com.thinkaurelius.titan</groupId>
<artifactId>titan-cassandra</artifactId>
<version>${titan.version}</version>
</dependency>
<dependency>
<groupId>com.thinkaurelius.titan</groupId>
<artifactId>titan-hbase</artifactId>
<version>${titan.version}</version>
</dependency>
<dependency>
<groupId>com.thinkaurelius.titan</groupId>
<artifactId>titan-es</artifactId>
<version>${titan.version}</version>
</dependency>
<dependency>
<groupId>com.thinkaurelius.titan</groupId>
<artifactId>titan-hadoop</artifactId>
<version>${titan.version}</version>
</dependency>
<dependency>
<groupId>com.thinkaurelius.titan</groupId>
<artifactId>titan-hadoop-core</artifactId>
<version>${titan.version}</version>
</dependency>
import java.io.File;
import java.util.Iterator;
import java.util.Map;
import org.apache.tinkerpop.gremlin.process.traversal.Order;
import org.apache.tinkerpop.gremlin.process.traversal.P;
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversal;
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__;
import org.apache.tinkerpop.gremlin.structure.Direction;
import org.apache.tinkerpop.gremlin.structure.Edge;
import org.apache.tinkerpop.gremlin.structure.T;
import org.apache.tinkerpop.gremlin.structure.Vertex;
import com.thinkaurelius.titan.core.EdgeLabel;
import com.thinkaurelius.titan.core.Multiplicity;
import com.thinkaurelius.titan.core.PropertyKey;
import com.thinkaurelius.titan.core.TitanFactory;
import com.thinkaurelius.titan.core.TitanGraph;
import com.thinkaurelius.titan.core.TitanTransaction;
import com.thinkaurelius.titan.core.attribute.Geoshape;
import com.thinkaurelius.titan.core.attribute.Text;
import com.thinkaurelius.titan.core.schema.ConsistencyModifier;
import com.thinkaurelius.titan.core.schema.TitanGraphIndex;
import com.thinkaurelius.titan.core.schema.TitanManagement;
public class GraphOfTheGodsFactory {
public static final String INDEX_NAME = "search";
public static TitanGraph create(final String directory) {
TitanFactory.Builder config = TitanFactory.build();
config.set("storage.backend", "berkeleyje");
config.set("storage.directory", directory);
config.set("index." + INDEX_NAME + ".backend", "elasticsearch");
config.set("index." + INDEX_NAME + ".directory", directory + File.separator + "es");
config.set("index." + INDEX_NAME + ".elasticsearch.local-mode", true);
config.set("index." + INDEX_NAME + ".elasticsearch.client-only", false);
TitanGraph graph = config.open();
GraphOfTheGodsFactory.load(graph);
return graph;
}
public static TitanGraph create() {
TitanFactory.Builder builder = TitanFactory.build();
builder.set("storage.backend", "hbase");
builder.set("storage.hostname", "host-10");
builder.set("storage.tablename", "titan");
builder.set("index.search.backend", "elasticsearch");
builder.set("index.search.hostname", "host-10");
builder.set("index.search.elasticsearch.interface", "TRANSPORT_CLIENT");
builder.set("index.search.elasticsearch.cluster-name", "data");
builder.set("index.search.elasticsearch.client-only", "true");
TitanGraph graph = builder.open();
return graph;
}
public static void loadWithoutMixedIndex(final TitanGraph graph,
boolean uniqueNameCompositeIndex) {
load(graph, null, uniqueNameCompositeIndex);
}
public static void load(final TitanGraph graph) {
load(graph, INDEX_NAME, true);
}
public static void load(final TitanGraph graph, String mixedIndexName,
boolean uniqueNameCompositeIndex) {
// Create Schema
TitanManagement mgmt = graph.openManagement();
final PropertyKey name = mgmt.makePropertyKey("name").dataType(String.class).make();
TitanManagement.IndexBuilder nameIndexBuilder = mgmt.buildIndex("name", Vertex.class).addKey(name);
if (uniqueNameCompositeIndex) nameIndexBuilder.unique();
TitanGraphIndex namei = nameIndexBuilder.buildCompositeIndex();
mgmt.setConsistency(namei, ConsistencyModifier.LOCK);
final PropertyKey age = mgmt.makePropertyKey("age").dataType(Integer.class).make();
if (null != mixedIndexName)
mgmt.buildIndex("vertices", Vertex.class).addKey(age).buildMixedIndex(mixedIndexName);
final PropertyKey time = mgmt.makePropertyKey("time").dataType(Integer.class).make();
final PropertyKey reason = mgmt.makePropertyKey("reason").dataType(String.class).make();
final PropertyKey place = mgmt.makePropertyKey("place").dataType(Geoshape.class).make();
if (null != mixedIndexName)
mgmt.buildIndex("edges", Edge.class).addKey(reason).addKey(place)
.buildMixedIndex(mixedIndexName);
mgmt.makeEdgeLabel("father").multiplicity(Multiplicity.MANY2ONE).make();
mgmt.makeEdgeLabel("mother").multiplicity(Multiplicity.MANY2ONE).make();
EdgeLabel battled = mgmt.makeEdgeLabel("battled").signature(time).make();
mgmt.buildEdgeIndex(battled, "battlesByTime", Direction.BOTH,
Order.decr, time);
mgmt.makeEdgeLabel("lives").signature(reason).make();
mgmt.makeEdgeLabel("pet").make();
mgmt.makeEdgeLabel("brother").make();
mgmt.makeVertexLabel("titan").make();
mgmt.makeVertexLabel("location").make();
mgmt.makeVertexLabel("god").make();
mgmt.makeVertexLabel("demigod").make();
mgmt.makeVertexLabel("human").make();
mgmt.makeVertexLabel("monster").make();
mgmt.commit();
TitanTransaction tx = graph.newTransaction();
// vertices
Vertex saturn = tx.addVertex(T.label, "titan", "name", "saturn", "age", 10000);
Vertex sky = tx.addVertex(T.label, "location", "name", "sky");
Vertex sea = tx.addVertex(T.label, "location", "name", "sea");
Vertex jupiter = tx.addVertex(T.label, "god", "name", "jupiter", "age", 5000);
Vertex neptune = tx.addVertex(T.label, "god", "name", "neptune", "age", 4500);
Vertex hercules = tx.addVertex(T.label, "demigod", "name", "hercules", "age", 30);
Vertex alcmene = tx.addVertex(T.label, "human", "name", "alcmene", "age", 45);
Vertex pluto = tx.addVertex(T.label, "god", "name", "pluto", "age", 4000);
Vertex nemean = tx.addVertex(T.label, "monster", "name", "nemean");
Vertex hydra = tx.addVertex(T.label, "monster", "name", "hydra");
Vertex cerberus = tx.addVertex(T.label, "monster", "name", "cerberus");
Vertex tartarus = tx.addVertex(T.label, "location", "name", "tartarus");
// edges
jupiter.addEdge("father", saturn);
jupiter.addEdge("lives", sky, "reason", "loves fresh breezes");
jupiter.addEdge("brother", neptune);
jupiter.addEdge("brother", pluto);
neptune.addEdge("lives", sea).property("reason", "loves waves");
neptune.addEdge("brother", jupiter);
neptune.addEdge("brother", pluto);
hercules.addEdge("father", jupiter);
hercules.addEdge("mother", alcmene);
hercules.addEdge("battled", nemean, "time", 1, "place", Geoshape.point(38.1f, 23.7f));
hercules.addEdge("battled", hydra, "time", 2, "place", Geoshape.point(37.7f, 23.9f));
hercules.addEdge("battled", cerberus, "time", 12, "place", Geoshape.point(39f, 22f));
pluto.addEdge("brother", jupiter);
pluto.addEdge("brother", neptune);
pluto.addEdge("lives", tartarus, "reason", "no fear of death");
pluto.addEdge("pet", cerberus);
cerberus.addEdge("lives", tartarus);
// commit the transaction to disk
tx.commit();
}
public static void query(TitanGraph graph) {
GraphTraversalSource g = graph.traversal();
System.out.println(g.V().has("name", "hercules").next().value("name"));
System.out.println(g.V().has("name", "hercules").next().values("name", "age"));
Iterator<Object> iterator = g.V().has("name","hercules").next().values("name", "age");
while(iterator.hasNext()){
Object object = iterator.next();
System.out.println(object);
}
Vertex saturn = g.V().has("name","saturn").next();
System.out.println(saturn);
//得到 saturn的孙子节点
System.out.println(g.V(saturn).in("father").in("father").next().value("age"));
GraphTraversal<Edge, Edge> a = g.E().has("place", P.eq(Geoshape.point(38.1f, 23.7f)));
System.out.println(a);
while(a.hasNext()){
Edge e = a.next();
System.out.println(e.keys());
System.out.println(e.label());
System.out.println(e.outVertex().value("name"));
System.out.println(e.inVertex().value("name"));
System.out.println(e.value("time") + " : " + e.value("place"));
}
Vertex hercules = g.V().has("name","hercules").next();
System.out.println(g.V(hercules).out("mother","father").values("name"));
GraphTraversal<Vertex,Vertex> mF = g.V(hercules).out("mother", "father");
while(mF.hasNext()){
Vertex v = mF.next();
System.out.println(v.label() + " : " + v.value("name"));
}
System.out.println(g.V(saturn).repeat(__.in("father")).times(2).next().value("name"));
GraphTraversal<Vertex,Vertex> monsters = g.V(hercules).out("battled");
while(monsters.hasNext()){
Vertex monster = monsters.next();
System.out.println(monster.label() + " : " + monster.value("name"));
}
monsters = g.V(hercules).outE("battled").has("time",P.eq(1)).inV();
while(monsters.hasNext()){
Vertex monster = monsters.next();
System.out.println(monster.label() + " : " + monster.value("name"));
}
Vertex pluto = g.V().has("name","pluto").next();
GraphTraversal<Vertex,Vertex> liveInTartarusVertex = g.V(pluto).out("lives").in("lives");
while(liveInTartarusVertex.hasNext()){
Vertex vertex = liveInTartarusVertex.next();
System.out.println(vertex.value("name"));
}
GraphTraversal<Vertex,Vertex> liveInTartarusVertexNo = g.V(pluto).out("lives").in("lives").where(
__.is(P.neq(pluto)));
while(liveInTartarusVertexNo.hasNext()){
Vertex vertex = liveInTartarusVertexNo.next();
System.out.println(vertex.value("name"));
}
GraphTraversal<Vertex,Vertex> liveInTartarusVertexNot = g.V(pluto).as("x").out("lives").in("lives").where(P.neq("x"));
while(liveInTartarusVertexNot.hasNext()){
Vertex vertex = liveInTartarusVertexNot.next();
System.out.println("======"+vertex.value("name"));
}
GraphTraversal<Vertex, Map<String, Vertex>> brothers =
g.V(pluto).out("brother").as("god").out("lives").as("place").select("god","place");
while(brothers.hasNext()){
Map<String,Vertex> map = brothers.next();
System.out.println(map);
for(Map.Entry<String,Vertex> entry:map.entrySet()){
System.out.println(entry.getKey()+" : "+entry.getValue().value("name"));
}
}
System.out.println(g.V(pluto).outE("lives").next().value("reason"));
GraphTraversal<Edge,Edge> reasons = g.E().has("reason").as("r").values("reason").is(Text.textContains("loves")).select("r");
System.out.println(reasons);
while(reasons.hasNext()){
Edge e = reasons.next();
System.out.println(e.keys());
System.out.println(e.label());
System.out.println(e.value("reason"));
}
GraphTraversal<Edge,Map<String,Object>> reasons2 =
g.E().has("reason").as("source").values("reason").is(Text.textContains("loves")).as("reason").select("source")
.outV().values("name").as("god").select("source").inV().values("name").as("thing").select("god","reason","thing");
while(reasons2.hasNext()){
Map<String,Object> map = reasons2.next();
System.out.println(map);
for(Map.Entry<String,Object> entry:map.entrySet()){
System.out.println(entry.getKey() + " : " + entry.getValue());
}
}
}
/**
*
* Calls {@link TitanFactory#open(String)}, passing the Titan configuration
* file path
* which must be the sole element in the {@code args} array, then calls
* {@link #load(com.thinkaurelius.titan.core.TitanGraph)} on the opened
* graph,
* then calls {@link com.thinkaurelius.titan.core.TitanGraph#close()}
* and returns.
* <p/>
* This method may call {@link System#exit(int)} if it encounters an error,
* such as
* failure to parse its arguments. Only use this method when executing main
* from
* a command line. Use one of the other methods on this class (
* {@link #create(String)}
* or {@link #load(com.thinkaurelius.titan.core.TitanGraph)}) when calling
* from
* an enclosing application.
* @param args
* a singleton array containing a path to a Titan config
* properties file
*/
public static void main(String args[]) {
// if (null == args || 1 != args.length) {
// System.err.println("Usage: GraphOfTheGodsFactory <titan-config-file>");
// System.exit(1);
// }
// TitanGraph g = TitanFactory.open(args[0]);
// load(g);
// g.close();
TitanGraph graph = GraphOfTheGodsFactory.create();
load(graph);
query(graph);
graph.close();
}
}
注意:
Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
HBase与Guava版本不匹配,这里Guava版本需要在17以下
java.lang.ClassNotFoundException: com.thinkaurelius.titan.diskstorage.hbase.HBaseCompat1_00
需要在项目中新建一个com.thinkaurelius.titan.diskstorage.hbase.HBaseCompat1_00,内容与
com.thinkaurelius.titan.diskstorage.hbase.HBaseCompat1_0一致即可。