Pubsub简介和Pub使用详解
一.简介
google cloud pub/sub是一种具有传递和接受消息的事件驱动以及流分析系统,跟kafka比较相似。它具有以下特点:
-
集成支持C++,C#,go,java,node.js,python,ruby
-
消息传递有序且发送效率支持从零到数百GB/秒
-
无需分区的自动扩展和自动配置
-
可跨区域系统(pubsub服务器在全球所有的Google Cloud区域中运行)
-
易上手且成本低
二.Pubsub配置
Google cloud上配置pubsub是非常容易上手的,直接访问Google Cloud Pubsub,在上面创建project(项目),Topic(主题),Subscription(订阅)。
Topic:发布者向其发送消息的命名资源(数据往这发送)
Subscription:一种命名资源,表示来自单个特定主题的消息流,该消息流将传递给订阅应用程序(从Topic订阅发送的信息)
message:发布者发送到主题并最终传递给订阅者的数据和(可选)属性的组合。
发布者应用程序创建并向主题发送消息。订阅者应用程序创建对主题的订阅,以从中接收消息。通讯可以是一对多(扇出),多对一(扇入)和多对多。
三.消息流
三.账户密钥
pubsub配置完成之后为了使第三方可以使用,我们需要为它生成一个账户密钥。Google cloud Pubsub客户端库会在使用的时候自动的读取GOOGLE_APPLICATION_CREDENTIALS
这个环境变量,这个环境变量指向的便是你生成的账户密钥。Idea或者eclipse在使用的时候需要将次环境变量配置到项目的Environment variables中。下面是一个账户密钥的事例:
{
"type": "service_account",
"project_id": "concise-option-296909",
"private_key_id": "9555ce9d54842de70af939f24893e1b0ec9567b9",
"private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQDNBE8fesU33n9g\ntFiWTjtlaIEpAsAzdlF6+Okglv0wLZi6q++cyQ5dqIwzMfob9T+06gK9OpCuPdKf\nhbpujp2Lq8a5m1TrDXMtgdfrnsZs+lGJmwrtG5OcXWxzG/czFfHf8xCYDQVpWkZF\nEK3El4kM7g9Lm2cgQFpylb1ZEduBSW0S1RELuU9fnbEIMAnM9Yrv6vadB9O9rBG0\nvtiFzmqyEE1mNCL+WjeQwbi4G907iagi/cXznXJN/PevPKsWhJ0rIhNY2lQbeuxy\ndYwmqCNuGleToxCD7XQotxI1D0ota6tyXAbc9RSmZDx5Ok1FVhhFrzK/z6xqiUnC\nJqKLOkBZAgMBAAECggEADJQoGqFYul3PaOqHyOkR4d3wlsiRbFblFutwukwpSqxx\ng3qLIA6B8CMoPvfhBZgdiYcWFfAi8l3er2TwFFpxtiEa90ZdhnJzz6X4zn8WY5dU\nK2I04GlhvCVLQyDDIQJGrdvm971MyLL+PryQPNCjWwS8MYJeaG8qXX9sd6uN0TfI\n+JaMERsUeiKOLjSrSdmjpmMguydPB6iClMxBEzcD9rA3jdTqZ+iJ2RJ8iqKjnEdI\nsg2ryHuXgensnQCTu92U+BdZLr12H2x69RgcSNbqXx3ECVqy5xjvsbW8rzpuMOPw\ngONS14UbTlWTeg4Txb4UrpYPUI8sa/mAEMNSVxNMEQKBgQDpDr60PhMCfxjm3555\nsniKH8Im139/hjE9auppXs2NfjX8oBdtoWQuiYZZenngA14Ed3KMKvCHJe7GfnHb\n+ClcCgTPF/KPm0GoEMGQ87C1twHiTzoUA3M5f2WqhXTqXFSdtcAjAmYb/BAnGp1D\n8U0jBbVAjMJuX12oZOJHGlqjjQKBgQDhMumQVL0u0CeQHEpqsthU6fVMaUdAOk91\nJvOTKEVbaCyu/v/aTOu8p3UT6pnjUA1FvdszR7p64a1i+11SHVFVRL83AWO7yTW9\nwXO36FZniaEjg0PaU4ooqqCkzqoi/tuASE14lj+XWyLBKcybEhYsZ9j5OhZpU7DR\nCIDVRZqW/QKBgGyX40JG/kKWiDrjjYs1Hrd1btj8Zrs8AuuUSt+IPg0k9vDcg64g\nmnaYvBIrYGiCCSfZpHUteN/wS98Pgew2KMdrH6yDgxFpDWHLVPY+pmM1f9o8glpa\n9TCgbPXUs5KULRVDX+CwOHXnsHsYCpli8UdVXIxUSjB1nUVepJTu8bbtAoGAIJpO\nVibufKiGEccGvwvFZgW5DabwOcw3VK6e9N8Pq3wZe/xO7jvUpStZN9jvDom4YjBq\nW7I2vTOO9pWNnSXBcKbTSBNbbQRkU2R+bFGmsLasLYgDesg9aEXLnBCPcSGUha3y\nCVM3QaJgFbcHeCvWKfee4fYW6IjFQH4KqNW628ECgYA/itvSAuWYY2/8le+J1DJz\nEQf0Ri3gHZ2HWLGo771avxLfaV2WRZRf34PyHbzr6qiLDKDIn959nL8qdQAaHtI3\nwC2ys+TuNlQW/3QHUwizdHzWiZm2CGvGhVKy3mYZ1hw9ipz9MOU+ZIxRnegJn2Ua\nvf0A9D9eMvMklMk0CxNgIg==\n-----END PRIVATE KEY-----\n",
"client_email": "test-487@concise-option-296909.iam.gserviceaccount.com",
"client_id": "117770248371861669412",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/test-487%40concise-option-296909.iam.gserviceaccount.com"
}
四.Pub实例
Gradle:
// google cloud pubsub
implementation 'com.google.cloud:google-cloud-pubsub:1.108.7'
java版:
package com.common;
import com.google.api.core.ApiFuture;
import com.google.api.core.ApiFutureCallback;
import com.google.api.core.ApiFutures;
import com.google.cloud.pubsub.v1.Publisher;
import com.google.common.util.concurrent.MoreExecutors;
import com.google.protobuf.ByteString;
import com.google.pubsub.v1.PubsubMessage;
import com.google.pubsub.v1.TopicName;
import java.io.IOException;
public class PubDemo {
// 初始化TopicName
private final TopicName topicName = TopicName.of("projectId", "topic");
private Publisher publisher;
public PubDemo() {
initPublisher();
}
// 初始化Publisher
private void initPublisher() {
try{
publisher = Publisher.newBuilder(topicName).build();
} catch (IOException e) {
e.printStackTrace();
}
}
public void sendMessage(String message) {
// 创建Pubsub发送数据的message类型
PubsubMessage pubsubMessage = PubsubMessage.newBuilder().setData(ByteString.copyFromUtf8(message)).build();
// 发送message
ApiFuture<String> apiFuture = publisher.publish(pubsubMessage);
//异步处理发送的结果
ApiFutures.addCallback(
apiFuture,
callback(), // 回调处理
MoreExecutors.directExecutor() // 处理的线程
);
}
private ApiFutureCallback<String> callback() {
return new ApiFutureCallback<>() {
@Override
public void onFailure(Throwable t) {
System.out.println("Send message to google pubsub failed");
}
@Override
public void onSuccess(String result) {
System.out.println("Send message to google pubsub success.");
}
};
}
}
Koltin版:
import com.google.api.core.ApiFutures
import com.google.cloud.pubsub.v1.Publisher
import com.google.common.util.concurrent.MoreExecutors
import com.google.protobuf.ByteString
import com.google.pubsub.v1.PubsubMessage
import com.google.pubsub.v1.TopicName
import com.google.api.core.ApiFutureCallback
class PubsubProducer() {
private val topicName: TopicName = TopicName.of("projectId", "topic")
private val publisher: Publisher = Publisher.newBuilder(topicName).build()
fun sendMessage(message: String) {
val result = publisher.publish(message.pubsubMessage())
ApiFutures.addCallback(
result,
callback(message),
MoreExecutors.directExecutor()
)
}
private fun String.pubsubMessage() = PubsubMessage.newBuilder().setData(ByteString.copyFromUtf8(this)).build()
// 异步回调
private fun callback(message: String) = object : ApiFutureCallback<String> {
override fun onFailure(t: Throwable) {
logger.error("Send message to google pubsub failed, message: $message")
}
override fun onSuccess(result: String?) {
logger.info("Send message to google pubsub succeeded")
}
}
}