一、Java流
1.流之collect
collect是将流中的元素收集到一个集合或其他数据结构中。collect
操作接受一个Collector
参数,该参数定义了如何收集元素和生成最终的结果。
例子:
List<String> strings = Arrays.asList("Hello", "world", "!");
String result = strings.stream()
.collect(Collectors.joining(" "));
System.out.println(result); // 输出:Hello world !
2.流之foreach
forEach(Consumer<? super T> action)
:forEach
方法用于遍历流中的每个元素,并对每个元素执行给定的操作。
例子:
List<String> strings = Arrays.asList("apple", "banana", "orange"); strings.stream() .forEach(s -> System.out.println(s.length()));
3.流之Filter
用于筛选流中满足特定条件的元素
例子:
List<Integer> numbers = Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
List<Integer> evenNumbers = numbers.stream()
.filter(n -> n % 2 == 0)
.collect(Collectors.toList());
System.out.println(evenNumbers); // 输出:[2, 4, 6, 8, 10]
4.流之reduce
通过对流中的元素进行逐个操作,并将操作的结果累积到一个最终的值上。它接受一个二元操作符作为参数,该操作符定义了如何将两个元素进行操作并生成一个新的值。
例子:
List<Integer> numbers = Arrays.asList(1, 2, 3, 4, 5);
int sum = numbers.stream()
.reduce(0, (a, b) -> a + b);
System.out.println(sum); // 输出:15
5.流之count
count
和forEach
是常用的最终操作,用于对流中的元素进行计数和遍历操作。
count()
:count
方法用于统计流中的元素数量,并返回统计结果。
下面是一个使用count
操作的案例,统计列表中大于等于 5 的元素个数:
例子:
List<Integer> numbers = Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
long count = numbers.stream()
.filter(n -> n >= 5)
.count();
System.out.println(count);
5.流之find
find
是一类常用的最终操作,用于从流中查找满足特定条件的元素,并返回一个 Optional
对象。
Java 提供了两种 find
操作:findFirst
和 findAny
。
例子1:
List<Integer> numbers = Arrays.asList(1, 3, 4, 2, 5);
Optional<Integer> firstEven = numbers.stream()
.filter(n -> n % 2 == 0)
.findFirst();
if (firstEven.isPresent()) {
System.out.println("First even number: " + firstEven.get()); // 输出:First even number: 4
} else {
System.out.println("No even number found.");
}
例子2:
List<String> strings = Arrays.asList("apple", "banana", "orange");
Optional<String> anyStartsWithA = strings.stream()
.filter(s -> s.startsWith("a"))
.findAny();
if (anyStartsWithA.isPresent()) {
System.out.println("Element starts with 'a': " + anyStartsWithA.get()); // 输出:Element starts with 'a': apple
} else {
System.out.println("No element starts with 'a' found.");
}
6.流之matching
用于检查流中的元素是否满足特定的条件。Java 提供了三个 Matching 操作:allMatch
、anyMatch
和 noneMatch
。
例子1:
List<Integer> numbers = Arrays.asList(2, 4, 6, 8, 10);
boolean allEven = numbers.stream()
.allMatch(n -> n % 2 == 0);
System.out.println(allEven); // 输出:true
例子2:
List<String> strings = Arrays.asList("apple", "banana", "orange");
boolean anyStartsWithA = strings.stream()
.anyMatch(s -> s.startsWith("a"));
System.out.println(anyStartsWithA); // 输出:true
例子3:
List<Integer> numbers = Arrays.asList(1, 2, 3, 4, 5);
boolean noneNegative = numbers.stream()
.noneMatch(n -> n < 0);
System.out.println(noneNegative); // 输出:true
7.流之distinct
它用于去除流中的重复元素,并生成一个新的流。
distinct
操作会根据元素的 hashCode
和 equals
方法来判断元素是否重复。只有在流中第一次出现的元素会被保留,后续出现的重复元素会被过滤掉。
例子:
List<Integer> numbers = Arrays.asList(1, 2, 2, 3, 3, 4, 5, 5);
List<Integer> distinctNumbers = numbers.stream()
.distinct()
.collect(Collectors.toList());
System.out.println(distinctNumbers); // 输出:[1, 2, 3, 4, 5]
8.流之sorted
它用于对流中的元素进行排序,并生成一个新的流。
sorted
操作可以用于对元素进行升序或降序排序。默认情况下,它会按照元素的自然顺序进行排序,或者根据指定的比较器进行排序。
例子:
List<Integer> numbers = Arrays.asList(5, 3, 8, 2, 1, 4);
List<Integer> sortedNumbers = numbers.stream()
.sorted()
.collect(Collectors.toList());
System.out.println(sortedNumbers); // 输出:[1, 2, 3, 4, 5, 8]
9.流之limit
用于对流进行截取或跳过操作,生成一个新的流。
例子:
List<Integer> numbers = Arrays.asList(1, 2, 3, 4, 5, 6);
List<Integer> limitedNumbers = numbers.stream()
.limit(3)
.collect(Collectors.toList());
System.out.println(limitedNumbers); // 输出:[1, 2, 3]
10.流之map
用于将流中的每个元素映射为另一个元素,并生成一个新的流。
map
接受一个函数作为参数,该函数定义了元素的映射规则。流中的每个元素都会被传递给该函数,并根据函数的映射规则生成一个新的元素。
例子:
List<String> names = Arrays.asList("John", "Sarah", "Tom", "Alice");
List<Integer> nameLengths = names.stream()
.map(String::length)
.collect(Collectors.toList());
System.out.println(nameLengths); // 输出:[4, 5, 3, 5]
11.流之flatmap
用于将流中的每个元素映射为多个元素,并将这些元素扁平化为一个新的流。
flatMap
接受一个函数作为参数,该函数定义了元素的映射规则,并返回一个流。流中的每个元素都会被传递给该函数,并将其映射为一个流。最后,所有映射后的流会被扁平化为一个新的流
例子:
List<List<Integer>> numbers = Arrays.asList(
Arrays.asList(1, 2),
Arrays.asList(3, 4),
Arrays.asList(5, 6)
);
List<Integer> flattenedNumbers = numbers.stream()
.flatMap(List::stream)
.collect(Collectors.toList());
System.out.println(flattenedNumbers); // 输出:[1, 2, 3, 4, 5, 6]
二、大数据流
1.apach spark
Apache Spark是一个开源的分布式计算系统,用于处理大型数据集。它被设计为比Hadoop的MapReduce更快,更灵活,使其非常适合大数据处理和分析。Spark 提供了一个统一的 API 来处理不同的数据源,并支持多种编程语言,包括 Java。
Spark 是围绕集群计算模型构建的,该模型允许它跨多台计算机横向扩展计算。它包括几个核心组件,包括为分布式计算提供基本功能的Spark Core;Spark SQL,它提供了一个用于处理结构化数据的SQL接口 和 Spark Streaming支持数据流的实时处理。
Spark的一个关键特性是它能够在内存中缓存数据,从而可以更快地处理经常访问的数据集。Spark 还包括机器学习库,可用于使用大型数据集构建和训练预测模型。
Spark 最常见的用例之一是数据处理和分析。例如,公司可以使用 Spark 实时处理大量客户数据,以深入了解客户行为和偏好。Spark的另一个用例是机器学习,它可用于训练和测试大型数据集以构建预测模型。
例子:计算文本文件中每个单词的出现次数
// create a SparkConf object
SparkConf conf = new SparkConf().setAppName("Word Count");
// create a JavaSparkContext object
JavaSparkContext sc = new JavaSparkContext(conf);
// read in a text file as an RDD of strings
JavaRDD<String> lines = sc.textFile("path/to/text/file");
// split each line into words and flatten the results
JavaRDD<String> words = lines.flatMap(line -> Arrays.asList(line.split(" ")).iterator());
// map each word to a tuple containing the word and a count of 1
JavaPairRDD<String, Integer> wordCounts = words.mapToPair(word -> new Tuple2<>(word, 1));
// reduce by key to sum the counts for each word
JavaPairRDD<String, Integer> counts = wordCounts.reduceByKey((a, b) -> a + b);
// output the results
counts.foreach(pair -> System.out.println(pair._1() + ": " + pair._2()));
例子2:处理大型推文数据集
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
public class TweetProcessor {
public static void main(String[] args) {
// Create Spark configuration
SparkConf conf = new SparkConf()
.setAppName("TweetProcessor")
.setMaster("local[*]");
// Create Spark context
JavaSparkContext sc = new JavaSparkContext(conf);
// Create Spark session
SparkSession spark = SparkSession
.builder()
.appName("TweetProcessor")
.getOrCreate();
// Load tweets dataset from HDFS
String tweetsPath = "hdfs://localhost:9000/tweets/*";
JavaRDD<String> tweets = sc.textFile(tweetsPath);
// Convert tweets to Spark SQL dataset
Dataset<Row> tweetsDS = spark.read().json(tweets);
// Register dataset as a temporary view
tweetsDS.createOrReplaceTempView("tweets");
// Query the dataset to find the most popular hashtags
Dataset<Row> hashtags = spark.sql("SELECT entities.hashtags.text AS hashtag, COUNT(*) AS count " +
"FROM tweets " +
"WHERE entities.hashtags IS NOT NULL " +
"GROUP BY entities.hashtags.text " +
"ORDER BY count DESC " +
"LIMIT 10");
// Show the results
hashtags.show();
// Stop Spark context
sc.stop();
}
}
此示例从 HDFS 加载推文数据集,将其转换为 Spark SQL 数据集,然后查询数据集以查找最流行的主题标签。它使用 Spark 的 SQL 功能对数据集运行 SQL 查询,并返回前 10 个主题标签及其计数。该示例使用 SparkConf 和 JavaSparkContext 类创建 Spark 上下文,并使用SparkSession类创建 Spark 会话。最后,它会在处理完成时停止 Spark 上下文。
例子3:Spark 处理客户事务的大型数据集
// Load data from HDFS
JavaRDD<String> input = sc.textFile("hdfs://path/to/data");
// Parse each line of input into a Transaction object
JavaRDD<Transaction> transactions = input.map(Transaction::parse);
// Group transactions by customer ID
JavaPairRDD<Integer, Iterable<Transaction>> customerTransactions = transactions.groupBy(Transaction::getCustomerId);
// Compute total spent by each customer
JavaPairRDD<Integer, Double> customerSpend = customerTransactions.mapValues(transactions -> {
double totalSpend = 0.0;
for (Transaction t : transactions) {
totalSpend += t.getAmount();
}
return totalSpend;
});
// Save results to HDFS
customerSpend.saveAsTextFile("hdfs://path/to/results");
此代码使用 Spark 从 HDFS 读取客户交易数据集,按客户 ID 对交易进行分组,计算每个客户花费的总金额,并将结果保存到 HDFS。
2、spark 详细使用
2.1、添加依赖包
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.4.8</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.4.8</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.4.8</version>
</dependency>
</dependencies>
2.2、编码
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
public class TopCustomers {
public static void main(String[] args) {
// Set up Spark configuration
SparkConf conf = new SparkConf()
.setAppName("TopCustomers")
.setMaster("local[*]");
JavaSparkContext sc = new JavaSparkContext(conf);
// Load transaction data from file
JavaRDD<String> transactions = sc.textFile("transactions.csv");
// Parse transaction data and convert to Spark SQL Dataset
SparkSession spark = SparkSession.builder().appName("TopCustomers").getOrCreate();
Dataset<Row> data = spark.read().option("header", true).csv(transactions);
// Group transactions by customer ID and sum the total spending
Dataset<Row> spendingByCustomer = data.groupBy("customer_id")
.sum("transaction_amount")
.orderBy(org.apache.spark.sql.functions.desc("sum(transaction_amount)"));
// Show the top 10 spenders
spendingByCustomer.show(10);
}
}
2.3、使用 Spark 从 HDFS 读取过滤的销售交易,并计算每个产品和区域的总销售额。
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.functions;
public class SalesSummaryJob {
public static void main(String[] args) {
SparkConf conf = new SparkConf().setAppName("Sales Summary Job");
JavaSparkContext sc = new JavaSparkContext(conf);
SparkSession spark = SparkSession.builder().appName("Sales Summary Job").getOrCreate();
Dataset<Row> sales = spark.read().csv("hdfs:///output/part-*");
sales = sales.selectExpr("_c1 as product", "_c2 as region", "_c3 as amount");
sales = sales.groupBy("product", "region").agg(functions.sum("amount").as("total_sales"));
sales.show();
spark.stop();
}
}
2.4、可以使用 spark-submit 命令运行此 Spark 作业:
spark-submit --class SalesSummaryJob --master yarn --deploy-mode client sales.jar
3、hadoop
用于使用HDFS和MapReduce进行分布式文件存储和处理。
3.1、map 映射
public class FilterMapper extends Mapper<LongWritable, Text, Text, NullWritable> {
private Text outputKey = new Text();
private NullWritable outputValue = NullWritable.get();
private SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd");
private Date filterDate;
public void setup(Context context) throws IOException, InterruptedException {
Configuration conf = context.getConfiguration();
String dateString = conf.get("filter.date");
try {
filterDate = dateFormat.parse(dateString);
} catch (ParseException e) {
throw new InterruptedException("Invalid date format: " + dateString);
}
}
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String[] fields = value.toString().split(",");
String dateString = fields[0];
try {
Date date = dateFormat.parse(dateString);
if (date.after(filterDate)) {
outputKey.set(value);
context.write(outputKey, outputValue);
}
} catch (ParseException e) {
// Ignore invalid date format
}
}
}
3.2、reduce规约
public class FilterReducer extends Reducer<Text, NullWritable, Text, NullWritable> {
private NullWritable outputValue = NullWritable.get();
public void reduce(Text key, Iterable<NullWritable> values, Context context) throws IOException, InterruptedException {
context.write(key, outputValue);
}
}
3.3、hadoop命令运行
我们可以使用 Hadoop 命令行工具运行此 MapReduce 作业,将过滤日期作为参数传递:
hadoop jar filter.jar FilterJob -D filter.date=2022-01-01 /input /output
4、stom 用于实时数据流。
4.1、场景
Storm 以流的形式读取销售摘要数据,并查找每个区域的最畅销产品。
4.2、实现
public class TopSellingProductsTopology {
public static void main(String[] args) throws Exception {
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("sales-spout", new SalesSummarySpout());
builder.setBolt("region-bolt", new RegionBolt())
.shuffleGrouping("sales-spout");
builder.setBolt("product-bolt", new ProductBolt())
.fieldsGrouping("region-bolt", new Fields("region"));
builder.setBolt("top-bolt", new TopProductsBolt())
.fieldsGrouping("product-bolt", new Fields("region", "product"));
Config conf = new Config();
conf.setDebug(true);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology("top-selling-products", conf, builder.createTopology());
Thread.sleep(10000);
cluster.killTopology("top-selling-products");
cluster.shutdown();
}
}
4.3、执行
storm jar top.jar TopSellingProductsTopology
5、flink用于流处理
5.1、场景
使用 Flink 从 HDFS 读取销售摘要数据,并将结果写入 Cassandra 和 HBase。
5.2、实现
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.DataSet;
import org.apache.flink.api.java.ExecutionEnvironment;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.api.java.tuple.Tuple4;
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.cassandra.CassandraSink;
import org.apache.flink.streaming.connectors.cassandra.ClusterBuilder;
import org.apache.flink.streaming.connectors.hbase.HBaseSink;
import org.apache.flink.streaming.connectors.hbase.HBaseTableDescriptor;
import org.apache.flink.streaming.connectors.hbase.HBaseTestingCluster;
import org.apache.flink.streaming.connectors.hbase.HBaseTestingClusterSchema;
import org.apache.flink.streaming.connectors.hbase.HBaseTestingSinkFunction;
import com.datastax.driver.core.Cluster;
import com.datastax.driver.core.Cluster.Builder;
import com.datastax.driver.core.Session;
import com.datastax.driver.mapping.Mapper;
import com.datastax.driver.mapping.MappingManager;
import java.util.ArrayList;
import java.util.List;
public class SalesProcessingJob {
public static void main(String[] args) throws Exception {
// set up the execution environment
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
// define the input data source
DataStream<String> sales = env.readTextFile("hdfs://localhost:9000/sales.csv");
// parse the input data and filter out transactions before a certain date
DataStream<Sale> filteredSales = sales
.map(new SaleParser())
.filter(new DateFilter());
// summarize sales by product and region
DataStream<Tuple3<String, String, Double>> productSales = filteredSales
.keyBy(sale -> Tuple2.of(sale.getProduct(), sale.getRegion()))
.window(TumblingEventTimeWindows.of(Time.days(1)))
.reduce(new SaleAggregator());
// find the best-selling products in each region
DataStream<Tuple4<String, String, Double, Double>> bestSellingProducts = productSales
.keyBy(1)
.window(TumblingEventTimeWindows.of(Time.days(7)))
.apply(new BestSellingProducts());
// write the results to Cassandra and HBase
CassandraSink.addSink(bestSellingProducts)
.setClusterBuilder(new ClusterBuilder() {
@Override
protected Cluster buildCluster(Cluster.Builder builder) {
return builder.addContactPoint("127.0.0.1").build();
}
})
.build();
HBaseSink<Tuple4<String, String, Double, Double>> hbaseSink = new HBaseSink<>(new HBaseTestingCluster(), new HBaseTestingSinkFunction());
HBaseTableDescriptor hbaseTableDescriptor = new HBaseTableDescriptor();
hbaseTableDescriptor.addColumn("product", "region");
hbaseTableDescriptor.addColumn("product", "sales");
hbaseSink.configure(hbaseTableDescriptor);
env.addSink(hbaseSink);
env.execute();
}
private static class SaleParser implements MapFunction<String, Sale> {
@Override
public Sale map(String value) throws Exception {
String[] fields = value.split(",");
return new Sale(
fields[0], // timestamp
fields[1], // region
fields[2], // product
Double.parseDouble(fields[3]), // price
Integer.parseInt(fields[4]) // quantity
);
}
}
private static class DateFilter implements FilterFunction<Sale> {
@Override
public boolean filter(Sale sale) throws Exception {
return sale.getTimestamp().compareTo("2022-01-01") >= 0;
}
}
private static class SaleAggregator implements ReduceFunction<Sale> {
@Override
public Sale reduce(Sale sale1, Sale sale2) throws Exception {
return new Sale(
sale1.getTimestamp(),
sale1.getRegion(),
sale1.getProduct(),
sale1.getPrice() + sale2.getPrice(),
sale1.getQuantity() + sale2.getQuantity()
);
}
}
private static class BestSellingProducts extends RichWindowFunction<Tuple3<String, String, Double>, Tuple4<String, String, Double, Double>, Tuple, TimeWindow> {
private transient MapState<String, Double> salesByProduct;
@Override
public void open(Configuration parameters) throws Exception {
super.open(parameters);
MapStateDescriptor<String, Double> descriptor = new MapStateDescriptor<>("salesByProduct", String.class, Double.class);
salesByProduct = getRuntimeContext(). getMapState(descriptor);
}
@Override
public void apply(Tuple key, TimeWindow window, Iterable<Tuple3<String, String, Double>> sales, Collector<Tuple4<String, String, Double, Double>> out) throws Exception {
salesByProduct.clear();
for (Tuple3<String, String, Double> sale : sales) {
String product = sale.f0;
Double salesAmount = sale.f2;
if (!salesByProduct.contains(product)) {
salesByProduct.put(product, salesAmount);
} else {
salesByProduct.put(product, salesByProduct.get(product) + salesAmount);
}
}
for (String product : salesByProduct.keys()) {
Double salesAmount = salesByProduct.get(product);
out.collect(Tuple4.of(product, key.getField(1), salesAmount, (salesAmount / window.getEnd())));
}
}
}
getMapState(descriptor);
}
@Override
public void apply(Tuple key, TimeWindow window, Iterable<Tuple3<String, String, Double>> sales, Collector<Tuple4<String, String, Double, Double>> out) throws Exception {
salesByProduct.clear();
for (Tuple3<String, String, Double> sale : sales) {
String product = sale.f0;
Double salesAmount = sale.f2;
if (!salesByProduct.contains(product)) {
salesByProduct.put(product, salesAmount);
} else {
salesByProduct.put(product, salesByProduct.get(product) + salesAmount);
}
}
for (String product : salesByProduct.keys()) {
Double salesAmount = salesByProduct.get(product);
out.collect(Tuple4.of(product, key.getField(1), salesAmount, (salesAmount / window.getEnd())));
}
}
}
5.3、解释
在这段代码中,我们首先搭建了Flink的执行环境,并指定了我们要使用事件时间作为时间特征。 然后我们将输入数据源定义为包含销售交易的文本文件。
6.storm 结合 flink 使用
6.1、代码
// Storm topology
TopologyBuilder builder = new TopologyBuilder();
// Kafka Spout
KafkaSpout kafkaSpout = new KafkaSpout(kafkaSpoutConfig);
builder.setSpout("kafkaSpout", kafkaSpout);
// Bolt for real-time analysis
RealtimeAnalysisBolt realtimeAnalysisBolt = new RealtimeAnalysisBolt();
builder.setBolt("realtimeAnalysisBolt", realtimeAnalysisBolt).shuffleGrouping("kafkaSpout");
// Cassandra Bolt for writing analyzed data
CassandraBolt cassandraBolt = new CassandraBolt(cassandraClusterConfig, cassandraWriteConfig);
builder.setBolt("cassandraBolt", cassandraBolt).shuffleGrouping("realtimeAnalysisBolt");
// Submit topology to Storm cluster
Config stormConfig = new Config();
StormSubmitter.submitTopology("my-topology", stormConfig, builder.createTopology());
// Flink job
final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
// Hadoop File Input Format
TextInputFormat inputFormat = new TextInputFormat(new Path("hdfs://namenode/user/data"));
// Read data from HDFS
DataSet<String> data = env.createInput(inputFormat).map(new MapFunction<String, String>() {
@Override
public String map(String value) {
return value;
}
});
// Flink analysis
FlinkAnalysis flinkAnalysis = new FlinkAnalysis();
DataSet<String> analyzedData = data.map(flinkAnalysis);
// Write analyzed data to HBase
HBaseOutputFormat hBaseOutputFormat = new HBaseOutputFormat(hBaseConfiguration);
hBaseOutputFormat.configure(hBaseConfiguration);
analyzedData.output(hBaseOutputFormat);
env.execute("my-job");
6.2、代码场景
这个例子演示了如何将 Storm 和 Flink 一起使用,以实时和批量的方式分析数据。数据由Kafka摄取,存储在HDFS中,由Storm分析,并写入Cassandra。然后,Flink 从 HDFS 读取相同的数据,进行分析并写入 HBase。
7、hbase 用于查询和存储数据
7.1、hbase概念
HBASE是一个高可靠性、高性能、面向列、可伸缩的分布式存储系统,利用HBASE技术可在廉价PC Server上搭建起大规模结构化存储集群。
HBASE的目标是存储并处理大型的数据,更具体来说是仅需使用普通的硬件配置,就能够处理由成千上万的行和列所组成的大型数据。
HBASE是Google Bigtable的开源实现,但是也有很多不同之处。比如:Google Bigtable使用GFS作为其文件存储系统,HBASE利用Hadoop HDFS作为其文件存储系统;Google运行MAPREDUCE来处理Bigtable中的海量数据,HBASE同样利用Hadoop MapReduce来处理HBASE中的海量数据;Google Bigtable利用Chubby作为协同服务,HBASE利用Zookeeper作为协同服务。
7.2、与传统数据库相比
(1)传统数据库遇到的问题:
1)数据量很大的时候无法存储;
2)没有很好的备份机制;
3)数据达到一定数量开始缓慢,很大的话基本无法支撑;
(2)HBASE优势:
1)线性扩展,随着数据量增多可以通过节点扩展进行支撑;
2)数据存储在hdfs上,备份机制健全;
3)通过zookeeper协调查找数据,访问速度快。
7.3、hbase集群角色
(1)一个或者多个主节点,Hmaster;
(2)多个从节点,HregionServer;
(3)HBase依赖项,zookeeper;
7.4、hbase详细了解
https://www.cnblogs.com/cfas/p/15788481.html
7.5、安装教程:
https://note.youdao.com/s/9NkiHOOy
路虽远,行则将至。事虽难,做则必成