简介
-
Apache Pulsar 介绍
Apache Pulsar 作为 Apache 软件基金会顶级项目，是下一代云原生分布式消息流平台，集消息、存储、轻量化函数式计算为一体，采用计算与存储分离架构设计，支持多租户、持久化存储、跨地域复制、分层存储，具有强一致性、高吞吐、低延时及高可扩展性等流数据存储特性，是云原生时代解决实时消息流数据传输、存储和计算的最佳解决方案。
Pulsar 的 Broker 没有状态，不存储数据。BookKeeper 负责存储数据，其中的 Bookie 支持水平扩容，元数据的信息存储在 ZooKeeper 上。这种架构也方便容器化的环境进行扩缩容，支持高可用等场景。 -
KubeSphere 介绍
KubeSphere 是建立在 Kubernetes 之上的面向云原生应用的分布式操作系统，完全开源，支持多云与多集群管理、应用商店、可观测性（监控、告警及审计等）和多租户等功能，提供全栈的 IT 自动化运维能力，简化企业的 DevOps 工作流。
在 KubeSphere 上安装 Apache Pulsar 作为一个以应用为中心的 K8s 容器平台，KubeSphere 为用户提供多种安装应用的方式。为快速部署应用，这里给大家推荐以下两种方法：- 直接从 KubeSphere 的应用商店中安装 Pulsar。KubeSphere 从 3.2.0 版本开始便新增了“动态加载应用商店”的功能。想要贡献应用的小伙伴可以直接向 KubeSphere 的 Helm 仓库贡献应用的 Helm Chart，待 PR 审核通过后应用商店上会加载最新的应用列表。
- 添加 Apache Pulsar 的 Helm 仓库至 KubeSphere 的应用仓库，然后从应用仓库中安装 Pulsar 应用模板。此安装方式类似在命令行使用 helm 相关命令添加仓库并安装。
注：KubeSphere 的应用商店与应用的全生命周期管理功能基于开源项目 OpenPitrix。在安装 Pulsar 前，你需要先在
KubeSphere 中启用 OpenPitrix。我们通过第二种方式在 KubeSphere Console 中添加 Pulsar 的 Helm 仓库并安装。
安装准备
-
准备一个 KubeSphere 集群
你可以直接使用 KubeKey 安装或通过 ks-installer 在已有 K8s 集群上搭建 KubeSphere 集群。同时，请注意 Pulsar 文档中对 K8s 集群和 Helm 的版本要求。 -
创建好企业空间和项目
企业空间是 KubeSphere 中的一种逻辑单元，用于管理项目、DevOps 项目和应用模板等资源，管理员可以管理其中的资源访问权限；项目指 K8s 中的 Namespace。为方便演示，本文统一采用 admin 用户操作。基于 KubeSphere 的多租户体系，在实际环境中你可以根据组织中每个租户的职责，创建不同的帐户并分配相应角色。在你的企业空间空间下，点击应用管理 > 应用仓库，然后点击添加。定义仓库名称后，添加 https://pulsar.apache.org/charts 仓库地址。仓库地址验证成功后，点击确定。
仓库添加成功后，我们还需要运行一个脚本，创建一些和 Pulsar 相关的 Secret 等资源，方便我们后续访问 Grafana 和 Pulsar Manager 等工具。先前往你的 K8s 节点，执行以下命令将 pulsar-helm-chart 的 GitHub 仓库克隆到本地。需要注意的是，在该节点上必须要可以执行 kubectl 和 helm 命令。
git clone https://github.com/apache/pulsar-helm-chart cd pulsar-helm-chart
执行以下命令运行 prepare_helm_release.sh 脚本。其中 -n 指相关的 Secret 生成的 Namespace，-k 指使用 helm 安装应用的 release 名称，这两个地方需要和我们在 KubeSphere 上指定的项目和应用名称相对应。有关该脚本可用参数的更多信息，可运行 ./scripts/pulsar/prepare_helm_release.sh --help 查看详情。
./scripts/pulsar/prepare_helm_release.sh -n pulsar -k ks-pulsar
前往 KubeSphere 中所创建的项目，本示例中的项目名为 pulsar。
点击应用负载 > 应用，在基于模板的应用标签页下，点击创建。
选择从应用模板，在应用模板的下拉列表中选中 Pulsar 的仓库。点击 Pulsar 后，我们可以在这里查看 Chart 文件的一些信息。
点击安装。定义应用名称（即刚才执行脚本时的 release 名称），选择版本以及安装位置（如果启用多集群功能并添加了多个集群，这里还可以选择 Pulsar 安装的所在集群），点击下一步。
转到应用设置页面，这里类似于使用 helm 命令时自定义 values.yaml 文件的各项参数。我们设置 namespace 为项目名称，initialize 为 true（仅首次安装 Pulsar 时需要设置），并启用 Pulsar Manager（设置 components.pulsar_manager 为 true），然后点击安装。有关这些参数更具体的信息，可参考 Pulsar 文档。
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
###
### K8S Settings
###
### Namespace to deploy pulsar
# The namespace to use to deploy the pulsar components, if left empty
# will default to .Release.Namespace (aka helm --namespace).
namespace: "pulsar"
namespaceCreate: false
## clusterDomain as defined for your k8s cluster
clusterDomain: cluster.local
###
### Global Settings
###
## Set to true on install
initialize: true
## Set cluster name
# clusterName:
## Pulsar Metadata Prefix
##
## By default, pulsar stores all the metadata at root path.
## You can configure to have a prefix (e.g. "/my-pulsar-cluster").
## If you do so, all the pulsar and bookkeeper metadata will
## be stored under the provided path
metadataPrefix: ""
## Port name prefix
##
## Used for Istio support which depends on a standard naming of ports
## See https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/#explicit-protocol-selection
## Prefixes are disabled by default
tcpPrefix: "" # For Istio this will be "tcp-"
tlsPrefix: "" # For Istio this will be "tls-"
## Persistence
##
## If persistence is enabled, components that have state will
## be deployed with PersistentVolumeClaims, otherwise, for test
## purposes, they will be deployed with emptyDir
##
## This is a global setting that is applied to all components.
## If you need to disable persistence for a component,
## you can set the `volume.persistence` setting to `false` for
## that component.
##
## Deprecated in favor of using `volumes.persistence`
persistence: true
## Volume settings
volumes:
persistence: true
# configure the components to use local persistent volume
# the local provisioner should be installed prior to enable local persistent volume
local_storage: false
## RBAC
##
## Configure settings related to RBAC such as limiting broker access to single
## namespece or enabling PSP
rbac:
enabled: false
psp: false
limit_to_namespace: false
## AntiAffinity
##
## Flag to enable and disable `AntiAffinity` for all components.
## This is a global setting that is applied to all components.
## If you need to disable AntiAffinity for a component, you can set
## the `affinity.anti_affinity` settings to `false` for that component.
affinity:
anti_affinity: true
# Set the anti affinity type. Valid values:
# requiredDuringSchedulingIgnoredDuringExecution - rules must be met for pod to be scheduled (hard) requires at least one node per replica
# preferredDuringSchedulingIgnoredDuringExecution - scheduler will try to enforce but not guranentee
type: requiredDuringSchedulingIgnoredDuringExecution
## Components
##
## Control what components of Apache Pulsar to deploy for the cluster
components:
# zookeeper
zookeeper: true
# bookkeeper
bookkeeper: true
# bookkeeper - autorecovery
autorecovery: true
# broker
broker: true
# functions
functions: true
# proxy
proxy: true
# toolset
toolset: true
# pulsar manager
pulsar_manager: true
## Monitoring Components
##
## Control what components of the monitoring stack to deploy for the cluster
monitoring:
# monitoring - prometheus
prometheus: true
# monitoring - grafana
grafana: true
# monitoring - node_exporter
node_exporter: true
# alerting - alert-manager
alert_manager: true
## which extra components to deploy (Deprecated)
extra:
# Pulsar proxy
proxy: false
# Bookkeeper auto-recovery
autoRecovery: false
# Pulsar dashboard
# Deprecated
# Replace pulsar-dashboard with pulsar-manager
dashboard: false
# pulsar manager
pulsar_manager: false
# Monitoring stack (prometheus and grafana)
monitoring: false
# Configure Kubernetes runtime for Functions
functionsAsPods: false
## Images
##
## Control what images to use for each component
images:
zookeeper:
repository: registry.cn-zhangjiakou.aliyuncs.com/qy566/pulsar-all
tag: 2.7.4
pullPolicy: IfNotPresent
bookie:
repository: registry.cn-zhangjiakou.aliyuncs.com/qy566/pulsar-all
tag: 2.7.4
pullPolicy: IfNotPresent
autorecovery:
repository: registry.cn-zhangjiakou.aliyuncs.com/qy566/pulsar-all
tag: 2.7.4
pullPolicy: IfNotPresent
broker:
repository: registry.cn-zhangjiakou.aliyuncs.com/qy566/pulsar-all
tag: 2.7.4
pullPolicy: IfNotPresent
proxy:
repository: registry.cn-zhangjiakou.aliyuncs.com/qy566/pulsar-all
tag: 2.7.4
pullPolicy: IfNotPresent
functions:
repository: registry.cn-zhangjiakou.aliyuncs.com/qy566/pulsar-all
tag: 2.7.4
prometheus:
repository: prom/prometheus
tag: v2.17.2
pullPolicy: IfNotPresent
grafana:
repository: streamnative/apache-pulsar-grafana-dashboard-k8s
tag: 0.0.10
pullPolicy: IfNotPresent
pulsar_manager:
repository: apachepulsar/pulsar-manager
tag: v0.1.0
pullPolicy: IfNotPresent
hasCommand: false
## TLS
## templates/tls-certs.yaml
##
## The chart is using cert-manager for provisioning TLS certs for
## brokers and proxies.
tls:
enabled: false
ca_suffix: ca-tls
# common settings for generating certs
common:
# 90d
duration: 2160h
# 15d
renewBefore: 360h
organization:
- pulsar
keySize: 4096
keyAlgorithm: rsa
keyEncoding: pkcs8
# settings for generating certs for proxy
proxy:
enabled: false
cert_name: tls-proxy
# settings for generating certs for broker
broker:
enabled: false
cert_name: tls-broker
# settings for generating certs for bookies
bookie:
enabled: false
cert_name: tls-bookie
# settings for generating certs for zookeeper
zookeeper:
enabled: false
cert_name: tls-zookeeper
# settings for generating certs for recovery
autorecovery:
cert_name: tls-recovery
# settings for generating certs for toolset
toolset:
cert_name: tls-toolset
# Enable or disable broker authentication and authorization.
auth:
authentication:
enabled: false
provider: "jwt"
jwt:
# Enable JWT authentication
# If the token is generated by a secret key, set the usingSecretKey as true.
# If the token is generated by a private key, set the usingSecretKey as false.
usingSecretKey: false
authorization:
enabled: false
superUsers:
# broker to broker communication
broker: "broker-admin"
# proxy to broker communication
proxy: "proxy-admin"
# pulsar-admin client to broker/proxy communication
client: "admin"
######################################################################
# External dependencies
######################################################################
## cert-manager
## templates/tls-cert-issuer.yaml
##
## Cert manager is used for automatically provisioning TLS certificates
## for components within a Pulsar cluster
certs:
internal_issuer:
apiVersion: cert-manager.io/v1alpha2
enabled: false
component: internal-cert-issuer
type: selfsigning
# 90d
duration: 2160h
# 15d
renewBefore: 360h
issuers:
selfsigning:
######################################################################
# Below are settings for each component
######################################################################
## Pulsar: Zookeeper cluster
## templates/zookeeper-statefulset.yaml
##
zookeeper:
# use a component name that matches your grafana configuration
# so the metrics are correctly rendered in grafana dashboard
component: zookeeper
# the number of zookeeper servers to run. it should be an odd number larger than or equal to 3.
replicaCount: 3
updateStrategy:
type: RollingUpdate
podManagementPolicy: OrderedReady
# If using Prometheus-Operator enable this PodMonitor to discover zookeeper scrape targets
# Prometheus-Operator does not add scrape targets based on k8s annotations
podMonitor:
enabled: false
interval: 10s
scrapeTimeout: 10s
# True includes annotation for statefulset that contains hash of corresponding configmap, which will cause pods to restart on configmap change
restartPodsOnConfigMapChange: false
ports:
http: 8000
client: 2181
clientTls: 2281
follower: 2888
leaderElection: 3888
# nodeSelector:
# cloud.google.com/gke-nodepool: default-pool
probe:
liveness:
enabled: true
failureThreshold: 10
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
readiness:
enabled: true
failureThreshold: 10
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
startup:
enabled: false
failureThreshold: 30
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
affinity:
anti_affinity: true
# Set the anti affinity type. Valid values:
# requiredDuringSchedulingIgnoredDuringExecution - rules must be met for pod to be scheduled (hard) requires at least one node per replica
# preferredDuringSchedulingIgnoredDuringExecution - scheduler will try to enforce but not guranentee
type: requiredDuringSchedulingIgnoredDuringExecution
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8000"
tolerations: []
gracePeriod: 30
resources:
requests:
memory: 256Mi
cpu: 0.1
# extraVolumes and extraVolumeMounts allows you to mount other volumes
# Example Use Case: mount ssl certificates
# extraVolumes:
# - name: ca-certs
# secret:
# defaultMode: 420
# secretName: ca-certs
# extraVolumeMounts:
# - name: ca-certs
# mountPath: /certs
# readOnly: true
extraVolumes: []
extraVolumeMounts: []
volumes:
# use a persistent volume or emptyDir
persistence: true
data:
name: data
size: 5Gi
local_storage: true
## If you already have an existent storage class and want to reuse it, you can specify its name with the option below
##
storageClassName: nfs-storage
#
## Instead if you want to create a new storage class define it below
## If left undefined no storage class will be defined along with PVC
##
# storageClass:
# type: pd-ssd
# fsType: xfs
# provisioner: kubernetes.io/gce-pd
## Zookeeper configmap
## templates/zookeeper-configmap.yaml
##
configData:
PULSAR_MEM: >
-Xms64m -Xmx128m
PULSAR_GC: >
-XX:+UseG1GC
-XX:MaxGCPauseMillis=10
-Dcom.sun.management.jmxremote
-Djute.maxbuffer=10485760
-XX:+ParallelRefProcEnabled
-XX:+UnlockExperimentalVMOptions
-XX:+DoEscapeAnalysis
-XX:+DisableExplicitGC
-XX:+PerfDisableSharedMem
## Zookeeper service
## templates/zookeeper-service.yaml
##
service:
annotations:
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
## Zookeeper PodDisruptionBudget
## templates/zookeeper-pdb.yaml
##
pdb:
usePolicy: true
maxUnavailable: 1
## Pulsar: Bookkeeper cluster
## templates/bookkeeper-statefulset.yaml
##
bookkeeper:
# use a component name that matches your grafana configuration
# so the metrics are correctly rendered in grafana dashboard
component: bookie
## BookKeeper Cluster Initialize
## templates/bookkeeper-cluster-initialize.yaml
metadata:
## Set the resources used for running `bin/bookkeeper shell initnewcluster`
##
resources:
# requests:
# memory: 4Gi
# cpu: 2
replicaCount: 4
updateStrategy:
type: RollingUpdate
podManagementPolicy: Parallel
# If using Prometheus-Operator enable this PodMonitor to discover bookie scrape targets
# Prometheus-Operator does not add scrape targets based on k8s annotations
podMonitor:
enabled: false
interval: 10s
scrapeTimeout: 10s
# True includes annotation for statefulset that contains hash of corresponding configmap, which will cause pods to restart on configmap change
restartPodsOnConfigMapChange: false
ports:
http: 8000
bookie: 3181
# nodeSelector:
# cloud.google.com/gke-nodepool: default-pool
probe:
liveness:
enabled: true
failureThreshold: 60
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
readiness:
enabled: true
failureThreshold: 60
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
startup:
enabled: false
failureThreshold: 30
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 5
affinity:
anti_affinity: true
# Set the anti affinity type. Valid values:
# requiredDuringSchedulingIgnoredDuringExecution - rules must be met for pod to be scheduled (hard) requires at least one node per replica
# preferredDuringSchedulingIgnoredDuringExecution - scheduler will try to enforce but not guranentee
type: requiredDuringSchedulingIgnoredDuringExecution
annotations: {}
tolerations: []
gracePeriod: 30
resources:
requests:
memory: 512Mi
cpu: 0.2
# extraVolumes and extraVolumeMounts allows you to mount other volumes
# Example Use Case: mount ssl certificates
# extraVolumes:
# - name: ca-certs
# secret:
# defaultMode: 420
# secretName: ca-certs
# extraVolumeMounts:
# - name: ca-certs
# mountPath: /certs
# readOnly: true
extraVolumes: []
extraVolumeMounts: []
volumes:
# use a persistent volume or emptyDir
persistence: true
journal:
name: journal
size: 5Gi
local_storage: true
## If you already have an existent storage class and want to reuse it, you can specify its name with the option below
##
storageClassName: nfs-storage
#
## Instead if you want to create a new storage class define it below
## If left undefined no storage class will be defined along with PVC
##
# storageClass:
# type: pd-ssd
# fsType: xfs
# provisioner: kubernetes.io/gce-pd
ledgers:
name: ledgers
size: 5Gi
local_storage: true
storageClassName: nfs-storage
# storageClass:
# ...
## use a single common volume for both journal and ledgers
useSingleCommonVolume: false
common:
name: common
size: 10Gi
local_storage: true
storageClassName: nfs-storage
# storageClass: ## this is common too
# ...
## Bookkeeper configmap
## templates/bookkeeper-configmap.yaml
##
configData:
# we use `bin/pulsar` for starting bookie daemons
PULSAR_MEM: >
-Xms128m
-Xmx256m
-XX:MaxDirectMemorySize=256m
PULSAR_GC: >
-XX:+UseG1GC
-XX:MaxGCPauseMillis=10
-XX:+ParallelRefProcEnabled
-XX:+UnlockExperimentalVMOptions
-XX:+DoEscapeAnalysis
-XX:ParallelGCThreads=4
-XX:ConcGCThreads=4
-XX:G1NewSizePercent=50
-XX:+DisableExplicitGC
-XX:-ResizePLAB
-XX:+ExitOnOutOfMemoryError
-XX:+PerfDisableSharedMem
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintHeapAtGC
-verbosegc
-Xloggc:/var/log/bookie-gc.log
-XX:G1LogLevel=finest
# configure the memory settings based on jvm memory settings
dbStorage_writeCacheMaxSizeMb: "32"
dbStorage_readAheadCacheMaxSizeMb: "32"
dbStorage_rocksDB_writeBufferSizeMB: "8"
dbStorage_rocksDB_blockCacheSize: "8388608"
## Bookkeeper Service
## templates/bookkeeper-service.yaml
##
service:
spec:
publishNotReadyAddresses: true
## Bookkeeper PodDisruptionBudget
## templates/bookkeeper-pdb.yaml
##
pdb:
usePolicy: true
maxUnavailable: 1
## Pulsar: Bookkeeper AutoRecovery
## templates/autorecovery-statefulset.yaml
##
autorecovery:
# use a component name that matches your grafana configuration
# so the metrics are correctly rendered in grafana dashboard
component: recovery
replicaCount: 1
# If using Prometheus-Operator enable this PodMonitor to discover autorecovery scrape targets
# # Prometheus-Operator does not add scrape targets based on k8s annotations
podMonitor:
enabled: false
interval: 10s
scrapeTimeout: 10s
# True includes annotation for statefulset that contains hash of corresponding configmap, which will cause pods to restart on configmap change
restartPodsOnConfigMapChange: false
ports:
http: 8000
# nodeSelector:
# cloud.google.com/gke-nodepool: default-pool
affinity:
anti_affinity: true
# Set the anti affinity type. Valid values:
# requiredDuringSchedulingIgnoredDuringExecution - rules must be met for pod to be scheduled (hard) requires at least one node per replica
# preferredDuringSchedulingIgnoredDuringExecution - scheduler will try to enforce but not guranentee
type: requiredDuringSchedulingIgnoredDuringExecution
annotations: {}
# tolerations: []
gracePeriod: 30
resources:
requests:
memory: 64Mi
cpu: 0.05
## Bookkeeper auto-recovery configmap
## templates/autorecovery-configmap.yaml
##
configData:
BOOKIE_MEM: >
-Xms64m -Xmx64m
## Pulsar Zookeeper metadata. The metadata will be deployed as
## soon as the last zookeeper node is reachable. The deployment
## of other components that depends on zookeeper, such as the
## bookkeeper nodes, broker nodes, etc will only start to be
## deployed when the zookeeper cluster is ready and with the
## metadata deployed
pulsar_metadata:
component: pulsar-init
image:
# the image used for running `pulsar-cluster-initialize` job
repository: registry.cn-zhangjiakou.aliyuncs.com/qy566/pulsar-all
tag: 2.7.4
pullPolicy: IfNotPresent
## set an existing configuration store
# configurationStore:
configurationStoreMetadataPrefix: ""
configurationStorePort: 2181
## optional, you can provide your own zookeeper metadata store for other components
# to use this, you should explicit set components.zookeeper to false
#
# userProvidedZookeepers: "zk01.example.com:2181,zk02.example.com:2181"
## Pulsar: Broker cluster
## templates/broker-statefulset.yaml
##
broker:
# use a component name that matches your grafana configuration
# so the metrics are correctly rendered in grafana dashboard
component: broker
replicaCount: 3
# If using Prometheus-Operator enable this PodMonitor to discover broker scrape targets
# Prometheus-Operator does not add scrape targets based on k8s annotations
podMonitor:
enabled: false
interval: 10s
scrapeTimeout: 10s
# True includes annotation for statefulset that contains hash of corresponding configmap, which will cause pods to restart on configmap change
restartPodsOnConfigMapChange: false
ports:
http: 8080
https: 8443
pulsar: 6650
pulsarssl: 6651
# nodeSelector:
# cloud.google.com/gke-nodepool: default-pool
probe:
liveness:
enabled: true
failureThreshold: 10
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
readiness:
enabled: true
failureThreshold: 10
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
startup:
enabled: false
failureThreshold: 30
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
affinity:
anti_affinity: true
# Set the anti affinity type. Valid values:
# requiredDuringSchedulingIgnoredDuringExecution - rules must be met for pod to be scheduled (hard) requires at least one node per replica
# preferredDuringSchedulingIgnoredDuringExecution - scheduler will try to enforce but not guranentee
type: preferredDuringSchedulingIgnoredDuringExecution
annotations: {}
tolerations: []
gracePeriod: 30
resources:
requests:
memory: 512Mi
cpu: 0.2
# extraVolumes and extraVolumeMounts allows you to mount other volumes
# Example Use Case: mount ssl certificates
# extraVolumes:
# - name: ca-certs
# secret:
# defaultMode: 420
# secretName: ca-certs
# extraVolumeMounts:
# - name: ca-certs
# mountPath: /certs
# readOnly: true
extraVolumes: []
extraVolumeMounts: []
## Broker configmap
## templates/broker-configmap.yaml
##
configData:
PULSAR_MEM: >
-Xms128m -Xmx256m -XX:MaxDirectMemorySize=256m
PULSAR_GC: >
-XX:+UseG1GC
-XX:MaxGCPauseMillis=10
-Dio.netty.leakDetectionLevel=disabled
-Dio.netty.recycler.linkCapacity=1024
-XX:+ParallelRefProcEnabled
-XX:+UnlockExperimentalVMOptions
-XX:+DoEscapeAnalysis
-XX:ParallelGCThreads=4
-XX:ConcGCThreads=4
-XX:G1NewSizePercent=50
-XX:+DisableExplicitGC
-XX:-ResizePLAB
-XX:+ExitOnOutOfMemoryError
-XX:+PerfDisableSharedMem
managedLedgerDefaultEnsembleSize: "2"
managedLedgerDefaultWriteQuorum: "2"
managedLedgerDefaultAckQuorum: "2"
## Broker service
## templates/broker-service.yaml
##
service:
annotations: {}
## Broker PodDisruptionBudget
## templates/broker-pdb.yaml
##
pdb:
usePolicy: true
maxUnavailable: 1
### Broker service account
## templates/broker-service-account.yaml
service_account:
annotations: {}
## Pulsar: Functions Worker
## templates/function-worker-configmap.yaml
##
functions:
component: functions-worker
## Pulsar: Proxy Cluster
## templates/proxy-statefulset.yaml
##
proxy:
# use a component name that matches your grafana configuration
# so the metrics are correctly rendered in grafana dashboard
component: proxy
replicaCount: 3
# If using Prometheus-Operator enable this PodMonitor to discover proxy scrape targets
# Prometheus-Operator does not add scrape targets based on k8s annotations
podMonitor:
enabled: false
interval: 10s
scrapeTimeout: 10s
# True includes annotation for statefulset that contains hash of corresponding configmap, which will cause pods to restart on configmap change
restartPodsOnConfigMapChange: false
# nodeSelector:
# cloud.google.com/gke-nodepool: default-pool
probe:
liveness:
enabled: true
failureThreshold: 10
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
readiness:
enabled: true
failureThreshold: 10
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
startup:
enabled: false
failureThreshold: 30
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
affinity:
anti_affinity: true
# Set the anti affinity type. Valid values:
# requiredDuringSchedulingIgnoredDuringExecution - rules must be met for pod to be scheduled (hard) requires at least one node per replica
# preferredDuringSchedulingIgnoredDuringExecution - scheduler will try to enforce but not guranentee
type: requiredDuringSchedulingIgnoredDuringExecution
annotations: {}
tolerations: []
gracePeriod: 30
resources:
requests:
memory: 128Mi
cpu: 0.2
# extraVolumes and extraVolumeMounts allows you to mount other volumes
# Example Use Case: mount ssl certificates
# extraVolumes:
# - name: ca-certs
# secret:
# defaultMode: 420
# secretName: ca-certs
# extraVolumeMounts:
# - name: ca-certs
# mountPath: /certs
# readOnly: true
extraVolumes: []
extraVolumeMounts: []
## Proxy configmap
## templates/proxy-configmap.yaml
##
configData:
PULSAR_MEM: >
-Xms64m -Xmx64m -XX:MaxDirectMemorySize=64m
PULSAR_GC: >
-XX:+UseG1GC
-XX:MaxGCPauseMillis=10
-Dio.netty.leakDetectionLevel=disabled
-Dio.netty.recycler.linkCapacity=1024
-XX:+ParallelRefProcEnabled
-XX:+UnlockExperimentalVMOptions
-XX:+DoEscapeAnalysis
-XX:ParallelGCThreads=4
-XX:ConcGCThreads=4
-XX:G1NewSizePercent=50
-XX:+DisableExplicitGC
-XX:-ResizePLAB
-XX:+ExitOnOutOfMemoryError
-XX:+PerfDisableSharedMem
## Proxy service
## templates/proxy-service.yaml
##
ports:
http: 80
https: 443
pulsar: 6650
pulsarssl: 6651
service:
annotations: {}
type: LoadBalancer
## Proxy ingress
## templates/proxy-ingress.yaml
##
ingress:
enabled: false
annotations: {}
tls:
enabled: false
## Optional. Leave it blank if your Ingress Controller can provide a default certificate.
secretName: ""
hostname: ""
path: "/"
## Proxy PodDisruptionBudget
## templates/proxy-pdb.yaml
##
pdb:
usePolicy: true
maxUnavailable: 1
## Pulsar Extra: Dashboard
## templates/dashboard-deployment.yaml
## Deprecated
##
dashboard:
component: dashboard
replicaCount: 1
# nodeSelector:
# cloud.google.com/gke-nodepool: default-pool
annotations: {}
tolerations: []
gracePeriod: 0
image:
repository: apachepulsar/pulsar-dashboard
tag: latest
pullPolicy: IfNotPresent
resources:
requests:
memory: 1Gi
cpu: 250m
## Dashboard service
## templates/dashboard-service.yaml
##
service:
annotations: {}
ports:
- name: server
port: 80
ingress:
enabled: false
annotations: {}
tls:
enabled: false
## Optional. Leave it blank if your Ingress Controller can provide a default certificate.
secretName: ""
## Required if ingress is enabled
hostname: ""
path: "/"
port: 80
## Pulsar ToolSet
## templates/toolset-deployment.yaml
##
toolset:
component: toolset
useProxy: true
replicaCount: 1
# True includes annotation for statefulset that contains hash of corresponding configmap, which will cause pods to restart on configmap change
restartPodsOnConfigMapChange: false
# nodeSelector:
# cloud.google.com/gke-nodepool: default-pool
annotations: {}
tolerations: []
gracePeriod: 30
resources:
requests:
memory: 256Mi
cpu: 0.1
# extraVolumes and extraVolumeMounts allows you to mount other volumes
# Example Use Case: mount ssl certificates
# extraVolumes:
# - name: ca-certs
# secret:
# defaultMode: 420
# secretName: ca-certs
# extraVolumeMounts:
# - name: ca-certs
# mountPath: /certs
# readOnly: true
extraVolumes: []
extraVolumeMounts: []
## Bastion configmap
## templates/bastion-configmap.yaml
##
configData:
PULSAR_MEM: >
-Xms64M
-Xmx128M
-XX:MaxDirectMemorySize=128M
#############################################################
### Monitoring Stack : Prometheus / Grafana
#############################################################
## Monitoring Stack: Prometheus
## templates/prometheus-deployment.yaml
##
## Deprecated in favor of using `prometheus.rbac.enabled`
prometheus_rbac: false
prometheus:
component: prometheus
rbac:
enabled: true
replicaCount: 1
# True includes annotation for statefulset that contains hash of corresponding configmap, which will cause pods to restart on configmap change
restartPodsOnConfigMapChange: false
# nodeSelector:
# cloud.google.com/gke-nodepool: default-pool
annotations: {}
tolerations: []
gracePeriod: 5
port: 9090
enableAdminApi: false
resources:
requests:
memory: 256Mi
cpu: 0.1
volumes:
# use a persistent volume or emptyDir
persistence: true
data:
name: data
size: 5Gi
local_storage: true
## If you already have an existent storage class and want to reuse it, you can specify its name with the option below
##
storageClassName: nfs-storage
#
## Instead if you want to create a new storage class define it below
## If left undefined no storage class will be defined along with PVC
##
# storageClass:
# type: pd-standard
# fsType: xfs
# provisioner: kubernetes.io/gce-pd
## Prometheus service
## templates/prometheus-service.yaml
##
service:
annotations: {}
## Monitoring Stack: Grafana
## templates/grafana-deployment.yaml
##
grafana:
component: grafana
replicaCount: 1
# True includes annotation for statefulset that contains hash of corresponding configmap, which will cause pods to restart on configmap change
restartPodsOnConfigMapChange: false
# nodeSelector:
# cloud.google.com/gke-nodepool: default-pool
annotations: {}
tolerations: []
gracePeriod: 30
resources:
requests:
memory: 250Mi
cpu: 0.1
## Grafana service
## templates/grafana-service.yaml
##
service:
type: LoadBalancer
port: 3000
targetPort: 3000
annotations: {}
plugins: []
## Grafana configMap
## templates/grafana-configmap.yaml
##
configData: {}
## Grafana ingress
## templates/grafana-ingress.yaml
##
ingress:
enabled: false
annotations: {}
labels: {}
tls: []
## Optional. Leave it blank if your Ingress Controller can provide a default certificate.
## - secretName: ""
## Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
extraPaths: []
hostname: ""
protocol: http
path: /grafana
port: 80
admin:
user: pulsar
password: pulsar
## Components Stack: pulsar_manager
## templates/pulsar-manager.yaml
##
pulsar_manager:
component: pulsar-manager
replicaCount: 1
# True includes annotation for statefulset that contains hash of corresponding configmap, which will cause pods to restart on configmap change
restartPodsOnConfigMapChange: false
# nodeSelector:
# cloud.google.com/gke-nodepool: default-pool
annotations: {}
tolerations: []
gracePeriod: 30
resources:
requests:
memory: 250Mi
cpu: 0.1
configData:
REDIRECT_HOST: "http://127.0.0.1"
REDIRECT_PORT: "9527"
DRIVER_CLASS_NAME: org.postgresql.Driver
URL: jdbc:postgresql://127.0.0.1:5432/pulsar_manager
LOG_LEVEL: DEBUG
## If you enabled authentication support
## JWT_TOKEN: <token>
## SECRET_KEY: data:base64,<secret key>
## Pulsar manager service
## templates/pulsar-manager-service.yaml
##
service:
type: LoadBalancer
port: 9527
targetPort: 9527
annotations: {}
## Pulsar manager ingress
## templates/pulsar-manager-ingress.yaml
##
ingress:
enabled: false
annotations: {}
tls:
enabled: false
## Optional. Leave it blank if your Ingress Controller can provide a default certificate.
secretName: ""
hostname: ""
path: "/"
## If set use existing secret with specified name to set pulsar admin credentials.
existingSecretName:
admin:
user: pulsar
password: pulsar
等待一段时间后，查看 Pulsar 的各项组件是否已正常运行。
使用 Pulsar Manager 创建 Tenant、Namespace 以及 Topic
Pulsar 集群搭建好后，我们可以使用 Web 工具 Pulsar Manager 来管理 Pulsar 中的租户（Tenant）、命名空间（Namespace）以及主题（Topic）。这些资源可用来确保同一个组织中不同部门间数据的相互隔离，如下图所示。
【在 KubeSphere 上部署 Apache Pulsar】
前往应用负载 > 服务，进入暴露 Pulsar Manager 的 Service 的详情页面，并根据其中 LoadBalancer 类型的 IP 地址加端口号访问 Pulsar Manager。这里注意，根据部署环境的不同你可能还需要配置相应的防火墙或端口转发规则。
通过默认帐号密码 pulsar/pulsar 登录，然后添加环境。Service URL 这里填 Broker 的 Service 地址与端口号，本示例中为 http://ks-pulsar-broker:8080。
进入 Pulsar Manager 的 Dashboard 后，在不同菜单下分别添加 Tenant、Namespace 以及 Topic。定义 Topic 时我们需要选择该 Topic 的类型（持久化或非持久化），并指定 Topic 下的分区（Partition）数量。在 Pulsar 中，Topic 的格式为：{persistent | non-persistent}😕/tenant/namespace/topic。我们需要先记录下这些信息，后面向该 Topic 发送信息时会用到。
向 Pulsar 集群发送信息并消费 现在我们创建好了相关资源，可以尝试向 Pulsar 集群发送信息并消费。
在本地先下载好 Pulsar 的二进制文件，然后解压。
进入 Pulsar 目录后修改 conf/client.conf 文件中的 webServiceUrl 和 brokerServiceUrl 地址，用于将 Pulsar 客户端和 Pulsar 集群绑定。本示例中启用了代理访问集群，这里的地址需要填代理所暴露的 IP 地址。我们可以在 KubeSphere 的服务页面找到代理的 Service，并查看其详情页面中的 IP 地址。
webServiceUrl=http://34.71.201.104:8080
brokerServiceUrl=pulsar://34.71.201.104:6650
修改完后，在 Pulsar 目录下执行以下命令创建订阅并消费信息，命令中的 kubesphere/kubesphere/kubesphere 对应刚才在 Pulsar Manager 中所创建的 Tenant、Namespace 以及 Topic。
./bin/pulsar-client consume -s sub kubesphere/kubesphere/kubesphere -n 0
注意：运行前请务必确保已安装适用的 Java JRE/JDK 版本。
再打开一个命令行窗口，执行以下命令生产信息。
./bin/pulsar-client produce kubesphere/kubesphere/kubesphere -m “Hello KubeSphere and Pulsar” -n 10
可以看到消息已经成功消费。
通过 Grafana 监控 Pulsar 指标 Pulsar 的 Helm Chart 在部署时会安装 Grafana 提供与 Pulsar 相关的可观测性指标，便于我们进行后续的运维分析。
与访问 Pulsar Manager 类似，前往应用负载 > 服务，进入暴露 Grafana 的 Service 的详情页面，并根据其中 LoadBalancer 类型的 IP 地址加端口号访问 Grafana。
通过默认帐号密码 pulsar/pulsar 登录。在左上角选择一个 Dashboard 以查看 Pulsar 相关指标。
可以看到相关数据成功显示在 Grafana 上。
总结 本文演示了在 KubeSphere 上部署 Apache Pulsar 的操作步骤。借助 Pulsar 天然适配云原生环境的特性，我们可以借助 KubeSphere 运维友好的操作界面轻松部署并管理 Pulsar。感兴趣的朋友可以阅读 KubeSphere 和 Pulsar 的官方文档了解更多信息。
扫一扫
1565
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
