参考文档:
Deploy Portainer on Charmed Kubernetes
Using Portainer to manage Canonical Charmed Kubernetes
原文:
1 概述
如果您对 Kubernetes 的复杂性感到害怕,但仍需要使用容器对您的业务应用程序进行现代化改造,请放心,您不是唯一一个。容器编排平台解决了许多问题,但也带来了新问题,因此请继续阅读以了解一种可以帮助您获得收益的新方法。
什么是Charmed Kubernetes?
Charmed Kubernetes 是 Canonical 用于数据中心的纯上游 Kubernetes。它允许定制 Kubernetes 部署,因为它支持多个云基础设施和大量 CNCF landscape工具集。 Charmed Kubernetes 基于 Juju 生命周期自动化框架,消除了运行 Kubernetes 的复杂性。
什么是Portainer?
Portainer 是一个开源容器管理工具,适用于 Kubernetes、Docker、Docker Swarm 和 Azure ACI。它允许通过直观的用户界面部署和管理容器,而无需编写代码。
本教程的一些先决条件:
- 通过 ssh 终端访问 Linux 机器,最好是最新版本的 Ubuntu; 您需要在将用于部署的机器上安装 snap 包管理应用程序;
- 访问 AWS、GCP 或 Azure 等云服务提供商。我们将在本教程中使用 Azure 作为CSP,因此请确保您知道如果您使用不同的提供程序,什么可以是等效的接口/命令。
2 初始配置
在初始设置期间,您将学习如何安装设置 Charmed Kubernetes 和部署集群所需的工具。
安装需要的工具:
1 在您的终端上运行以下命令
- snap install juju --classic
- snap install kubectl --classic
- apt -y install nmap -> 这是可选的,可用于检查你的节点的公共 IP 地址上打开的端口
3. 与您的云提供商一起设置 Juju
1 使用 juju 添加云提供商凭据:juju add-credential azure
2 系统将提示您为此凭证命名。例如,您可以使用 azure
3 下一个提示是选择要用于部署的区域。我选择了eastus:
4 下一步是选择身份验证类型。您只需按 Enter 即可继续使用 [interactive]:
5 下一步是键入要用于部署 Charmed Kubernetes 群集的 Azure 订阅 ID:
6 将出现一个完成凭据设置的链接,您需要在浏览器中打开该链接并粘贴代码以进行身份验证。每次通过 juju 添加凭据时都会生成一个唯一的代码:
7 如果一切顺利,以下消息将出现在您的浏览器窗口中:
4 部署 Charmed Kubernetes 集群
1 启动您将在云提供商上部署的 juju 控制器的引导程序: juju bootstrap azure 确保将 name 替换为您的偏好之一。在这个例子中,我将使用charmed01。此过程最多可能需要 15 分钟
2 部署您的 Charmed Kubernetes 集群:
- 对于完整的集群,您可以使用
juju deploy Charmed-kubernetes
。这将部署至少 8 台具有2个冗余主节点的机器。 - 另一种方法是使用一个主节点 + 一个工作节点部署一个较小的集群,然后添加更多节点:
juju deploy cs:bundle/kubernetes-core-1200
- 使用
juju add-unit kubernetes-worker -n 2
添加至少 2 个节点。您可以在上面的第 2 步之后立即运行它。
3 您可以使用 watch -c juju status --color
监控集群的部署进度:
4 这至少需要 20 分钟以上。如果一切顺利,您应该会在屏幕上看到以下状态:
您可以按 CTRL+C 退出 juju 状态屏幕并返回到终端窗口上的提示。
5 确保您的主文件夹中有一个名为 .kube 的文件夹。如果不创建,则使用 mkdir -p $HOME/.kube
6 从主节点复制配置文件,以便使用 kubectl 管理集群: juju scp kubernetes-master/0:config ~/.kube/config
7 运行快速测试以确保您可以使用 kubectl 和 kubectl get node
访问您的集群。您应该会在集群上看到您的工作节点:
5 部署 OpenEBS
OpenEBS 是 Kubernetes 的绝佳存储选项。请访问他们的网站以了解有关此项目的更多信息。
1 首先在所有节点上启用 iscsid 服务。
juju ssh kubernetes-master/0 'sudo systemctl enable --now iscsid'
2 通过将 kubernetes-master/0 替换为 kubernetes-worker/0 对每个工作节点重复此命令,例如:
juju ssh kubernetes-worker/0 'sudo systemctl enable --now iscsid'
3 在您的主节点上启用运行特权服务:
juju config kubernetes-master allow-privileged=true
4 下载在集群上部署 OpenEBS 所需的 yaml 文件:
wget https://openebs.github.io/charts/openebs-operator.yaml
5 即使我们在主节点上启用了运行特权服务,我们仍然需要对文件中的特权字符串进行一些小的更改:
sed -i 's/privileged: true/privileged: false/g' openebs-operator.yaml && cat openebs-operator.yaml
6 使用 kubectl 部署 OpenEBS:
kubectl apply -f openebs-operator.yaml
7 检查 OpenEBS 部署的状态:
kubectl get pod -n openebs
8 一旦所有容器在您的集群上成功运行,部署将完成:
kubectl get pod -n openebs
NAME READY STATUS RESTARTS AGE
maya-apiserver-8679c68656-mwljv 1/1 Running 0 6m36s
openebs-admission-server-75758ffd7c-h9lb8 1/1 Running 0 6m31s
openebs-localpv-provisioner-7d77896b9f-hl5ch 1/1 Running 0 6m30s
openebs-ndm-jkhb5 1/1 Running 0 6m32s
openebs-ndm-lc5r8 1/1 Running 0 6m32s
openebs-ndm-operator-6658f6756d-hsgvz 1/1 Running 0 6m31s
openebs-ndm-sw8mt 1/1 Running 0 6m32s
openebs-provisioner-58c95774d9-dqzxp 1/1 Running 0 6m36s
openebs-snapshot-operator-5d748758df-wx47z 2/2 Running 0 6m34s
9 最后,确保 openebs-jiva-default storageclass 设置为默认值:
kubectl patch storageclass openebs-jiva-default -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
输出类似:
storageclass.storage.k8s.io/openebs-jiva-default patched
10 运行 以确保补丁有效:
kubectl get sc
输出类似:
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
openebs-device openebs.io/local Delete WaitForFirstConsumer false 7m8s
openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 7m9s
openebs-jiva-default (default) openebs.io/provisioner-iscsi Delete Immediate false 7m15s
openebs-snapshot-promoter volumesnapshot.external-storage.k8s.io/snapshot-promoter Delete Immediate false 7m10s
这是必需的,否则无法安装 Portainer。
6 部署Portainer
1 安装 Portainer 非常简单。您只需要运行 以下命令即可使用 NodePort 安装 Portainer:
kubectl apply -n portainer -f https://raw.githubusercontent.com/portainer/k8s/master/deploy/manifests/portainer/portainer.yaml
2 您可以使用 kubectl get pod -n portainer
检查安装状态。一旦您看到 STATUS 正在运行,那么 Portainer 就可以使用了:
3 Portainer,当部署在 NodePort 上时将在端口 30777 上运行。我们需要在 Azure(或您部署集群的 CSP)上打开这个端口,以确保它可以在您节点的公共 IP 上访问。您可以通过转到 Azure 门户上的“所有资源”页面并在 juju-default-xxxxxxxx 资源组中选择 juju-internal-nsg 来执行此操作:
4 单击入站安全规则:
5 创建一个规则,将打开端口 30777,如下所示:
6 您应该可以通过使用任何节点的公共 IP 地址 + 端口 30777 打开浏览器来访问 Portainer。在我的情况下,我的主节点的公共 IP 是 20.83.8.178,因此打开 Portainer 的链接是
http:// 20.83.8.178:30777:
在此页面上,您需要为 Portainer 设置管理员用户。
7 下一步是将您的 Portainer 实例连接到您的 Kubernetes 集群:
8 最后一步是通过启用指标服务器功能在 Portainer 上配置集群的 Kubernetes 功能:
9 并通过 RWO、RWX 和卷扩展启用 openebs-jiva-default 存储类:
10 单击“保存配置”,您的 Charmed Kubernetes 集群就可以使用 Portainer 进行管理了!!
7 就这样,伙计们
以下是实际操作:
1 由于已经按照前面文章ubuntu20.04下使用juju+maas环境部署k8s-2-部署charmed kubernetes #679部署charmed k8s,所以本文的章节3-设置juju和章节4-部署charmed kubernetes部分可以省略。
2 根据章节5部署openebs。
3 部署和配置portainer。
3.1 部署portainer
kubectl apply -n portainer -f https://raw.githubusercontent.com/portainer/k8s/master/deploy/manifests/portainer/portainer.yaml
3.2 检查portainer安装状态,显示status为running.
kubectl get pod -n portainer
3.3 显示node-ip。
实际就是kubernetes-worker节点ip。
两种办法:
juju status kubernetes-worker
或
kubectl get node -o wide
在本文中,node ip为:
10.0.4.102
10.0.4.106
10.0.4.108
node-port为:30777
3.4 在maas 服务器上,验证登陆。
curl 10.0.4.102:30777
输出为:
<!DOCTYPE html
><html lang="en" ng-app="portainer" ng-strict-di>
<head>
<meta charset="utf-8" />
<title>Portainer</title>
<meta name="description" content="" />
<meta name="author" content="Portainer.io" />
<!-- HTML5 shim, for IE6-8 support of HTML5 elements -->
<!--[if lt IE 9]>
<script src="//html5shim.googlecode.com/svn/trunk/html5.js"></script>
<![endif]-->
<!-- Fav and touch icons -->
<link rel="apple-touch-icon" sizes="180x180" href="dc4d092847be46242d8c013d1bc7c494.png" />
<link rel="icon" type="image/png" sizes="32x32" href="5ba13dcb526292ae707310a54e103cd1.png" />
<link rel="icon" type="image/png" sizes="16x16" href="f9508a64a1beb81be174e194573f7450.png" />
<link rel="mask-icon" href="07745d55b001c85826eedd479285cdbb.svg" color="#5bbad5" />
<link rel="shortcut icon" href="data:image/vnd.microsoft.icon;base64,AAABAAMAMDAAAAEAIACoJQAANgAAACAgAAABACAAqBAAAN4lAAAQEAAAAQAgAGgEAACGNgAAKAAAADAAAABgAAAAAQAgAAAAAAAAJAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAAAAAAAAAAAAAAAAAA/9o4BAAAAAAAAAAAAAAAAP/aOAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAT/2jgEAAAAAP/aOAT/2jh8/9o4x//aONj/2jjH/9o4fP/aOBoAAAAAAAAAAAAAAAD/2jgE/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAAAAAAAAAAAA/9o4BP/aOMf/2jj//9o4///aOP//2jj//9o4///aONj/2jgy/9o4fP/aONj/2jjw/9o42P/aOK3/2jgaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4Gv/aOFr/2jhG/9o4rf/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jjw/9o4///aOP//2jj//9o4///aOP//2jjw/9o4MgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jh8/9o48P/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o42P/aOAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOJL/2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOFoAAAAA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4Mv/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOJIAAAAA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4fP/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOHwAAAAA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4kv/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOEYAAAAA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4fP/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4xwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4Mv/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4xwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOJL/2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o42AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jiS/9o48P/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4rQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4Mv/aOFr/2jh8/9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4WgAAAAD/2jgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BAAAAAD/2jha/9o48P/aONj/2jhG/9o4rf/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jitAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aOEb/2jj//9o4///aOP//2jjH/9o4Gv/aOPD/2jj//9o4///aOP//2jj//9o4///aONj/2jiS/9o48P/aOP//2jj//9o48P/aOHz/2jgaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BP/aONj/2jj//9o4///aOP//2jj//9o4kv/aOBr/2jiS/9o48P/aOPD/2jjw/9o4kv/aOEb/2jgE/9o4Gv/aOEb/2jha/9o4Gv/aOHz/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4Rv/aOP//2jj//9o4///aOP//2jj//9o4///aOMf/2jha/9o4Rv/aOEb/2jhG/9o4fP/aOPD/2jhGAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4fP/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOPD/2jj//9o4///aOP//2jh8AAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4kv/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o48P/aOP//2jh8AAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4kv/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jh8AAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4Wv/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jhGAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4Gv/aONj/2jj//9o48P/aOPD/2jj//9o48P/aOPD/2jjw/9o4///aOPD/2jjw/9o4///aONj/2jgEAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOFr/2jiS/9o4fP/aOHz/2jha/9o4fP/aOHz/2jh8/9o4Wv/aOHz/2jh8/9o4kv/aOEYAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aOHz/2jit/9o4rf/aOJL/2jiS/9o4rf/aOK3/2jit/9o4kv/aOK3/2jit/9o4rf/aOFoAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aOHz/2jh8/9o4fP/aOHz/2jiS/9o4fP/aOHz/2jh8/9o4kv/aOHz/2jh8/9o4fP/aOFoAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aOHz/2jit/9o4kv/aOJL/2jiS/9o4kv/aOJL/2jiS/9o4kv/aOK3/2jiS/9o4rf/aOFoAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOFr/2jit/9o4rf/aOJL/2jha/9o4rf/aOK3/2jit/9o4Wv/aOHz/2jh8/9o4fP/aODIAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aOHz/2jit/9o4rf/aOJL/2jh8/9o4rf/aOK3/2jit/9o4RgAAAAD/2jgEAAAAAAAAAAAAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aOHz/2jiS/9o4fP/aOHz/2jiS/9o4fP/aOHz/2jiS/9o4Rv/aODL/2jgy/9o4Mv/aOBoAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aOHz/2jiS/9o4kv/aOJL/2jiS/9o4kv/aOJL/2jiS/9o4kv/aOMf/2jit/9o4x//aOFoAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOFr/2jiS/9o4kv/aOHz/2jha/9o4kv/aOJL/2jiS/9o4fP/aOJL/2jh8/9o4fP/aOFoAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4Rv/aOK3/2jiS/9o4kv/aOFoAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgE/9o4BP/aOAQAAAAA/9o4BP/aOAQAAAAA/9o4Wv/aOMf/2jh8/9o4x//aOHwAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4Wv/aOHwAAAAA/9o4Wv/aOHwAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAP/aOAT/2jgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4Wv/aOHwAAAAA/9o4Wv/aOHwAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jga/9o4Mv/aODL/2jgy/9o4Mv/aODL/2jgy/9o4Mv/aODL/2jga/9o4fP/aOJL/2jga/9o4fP/aOJL/2jga/9o4Mv/aOFr/2jj//9o4Wv/aOMf/2jjY/9o4Gv/aODL/2jgy/9o4Mv/aODL/2jgy/9o4Mv/aODL/2jgy/9o4Mv/aODL/2jgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BAAAAAD/2jh8/9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jhGAAAAAP/aOAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jga/9o4Rv/aOEb/2jhG/9o4Rv/aOEb/2jh8/9o48P/aOP//2jjH/9o4Rv/aODL/2jhG/9o4Rv/aOEb/2jhG/9o4Rv/aOFr/2jj//9o4fP/aODL/2jhG/9o4Rv/aOEb/2jhG/9o4Mv/aOEb/2jjH/9o4///aOPD/2jh8/9o4Rv/aOEb/2jgaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4Gv/aOJL/2jj//9o42P/aOEYAAAAAAAAAAAAAAAAAAAAAAAAAAP/aODL/2jj//9o4RgAAAAD/2jgEAAAAAAAAAAAAAAAA/9o4Rv/aONj/2jj//9o4kv/aOBoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BP/aOAT/2jgE/9o4BP/aOAT/2jgEAAAAAAAAAAD/2jgy/9o4x//aOP//2jjH/9o4MgAAAAD/2jgEAAAAAP/aODL/2jj//9o4RgAAAAD/2jgEAAAAAP/aODL/2jjH/9o4///aOMf/2jgyAAAAAAAAAAD/2jgE/9o4BP/aOAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAAAAAAAP/aOFr/2jjw/9o4///aOJL/2jgaAAAAAP/aODL/2jj//9o4RgAAAAD/2jga/9o4kv/aOP//2jjw/9o4WgAAAAAAAAAA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgE/9o4BAAAAAD/2jgE/9o4fP/aOP//2jjw/9o4Wv/aODL/2jj//9o4Rv/aOFr/2jjw/9o4///aOHz/2jgEAAAAAP/aOAT/2jgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAAAAAAAP/aOBr/2jit/9o4///aOPD/2jj//9o48P/aOP//2jit/9o4GgAAAAAAAAAA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BAAAAAAAAAAA/9o4Mv/aONj/2jj//9o42P/aOEYAAAAAAAAAAP/aOAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aODL/2jj//9o4RgAAAAD/2jgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aODL/2jjw/9o4RgAAAAD/2jgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP///wAA//+PAAD//gMP//8AAP/8AAf//wAA/+AAA///AAD/gAAD//8AAP+AAAH//wAA/4AAA///AAD/AAAD//8AAP+AAAP//wAA/4AAA///AAD/gAAD//8AAP/AAAP//wAA//wAB///AAD/5AAH//8AAP/CAB///wAA/4EH9///AAD/gPun//8AAP+AA6f//wAA/wADp///AAD/AAOn//8AAP+AA6f//wAA/4ADp///AAD/3/en//8AAP/AB6f//wAA//u/p///AAD/wAen//8AAP/Ef6f//wAA/8R/p///AAD/23+n//8AAP/AB6f//wAA/8xfp///AAD//8en//8AAP//16f//wAAp///AAD///+n//8AAP//26f//wAA/wAAAAD/AAD//H+/x/8AAP/+P7+P/wAA//+Pvj//AAD//+O4//8AAP//+bP//wAA///8B///AAD///8f//8AAP///7///wAAv///AAD///8AACgAAAAgAAAAQAAAAAEAIAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BP/aOAT/2jgEAAAAAP/aOCP/2jhA/9o4IwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAT/2jic/9o4/v/aOP7/2jj+/9o4Y//aOCP/2jic/9o4nP/aOGMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOED/2jiO/9o4tP/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOI4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jiO/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOCMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4QP/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4YwAAAAD/2jgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BAAAAAD/2jhj/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jhAAAAAAP/aOAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aOED/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o40gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BP/aONL/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jjS/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4I//aOI7/2jjS/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOLQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgE/9o4jv/aOID/2jiO/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4QAAAAAD/2jgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOJz/2jj+/9o4/v/aOGP/2ji0/9o4/v/aOP7/2jj+/9o4jv/aOI7/2jjS/9o4tP/aOID/2jgjAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgj/9o4/v/aOP7/2jj+/9o46//aOID/2jiA/9o4jv/aOI7/2jiAAAAAAP/aOGP/2jhj/9o4tP/aOEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aOGP/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOOv/2jjr/9o4/v/aOOsAAAAA/9o4gP/aOI7/2ji0/9o4QAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4Y//aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o46wAAAAD/2jiA/9o4jv/aOLT/2jhAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jhA/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jjSAAAAAP/aOID/2jiO/9o4tP/aOEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jiO/9o4tP/aOJz/2jic/9o4tP/aOJz/2jic/9o4tP/aOEAAAAAA/9o4gP/aOI7/2ji0/9o4QAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOID/2jiO/9o4gP/aOI7/2jiO/9o4gP/aOI7/2jiO/9o4QAAAAAD/2jiA/9o4jv/aOLT/2jhAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4gP/aOI7/2jiO/9o4jv/aOI7/2jiO/9o4nP/aOJz/2jhAAAAAAP/aOID/2jiO/9o4tP/aOEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jiA/9o4tP/aOID/2jiO/9o4tP/aOGP/2jhA/9o4QP/aOCMAAAAA/9o4gP/aOI7/2ji0/9o4QAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOID/2jiO/9o4gP/aOI7/2jiO/9o4Y//aOAT/2jgj/9o4BAAAAAD/2jiA/9o4jv/aOLT/2jhAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4gP/aOLT/2jiO/9o4nP/aOJz/2jiO/9o4nP/aOLT/2jhAAAAAAP/aOID/2jiO/9o4tP/aOEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgj/9o4I//aOCP/2jgj/9o4I//aOED/2jiO/9o4jv/aOEAAAAAA/9o4gP/aOI7/2ji0/9o4QAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4QP/aOI7/2jiA/9o4YwAAAAD/2jiO/9o4jv/aOLT/2jhAAAAAAP/aOAT/2jgE/9o4BP/aOAT/2jgE/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jhA/9o4QP/aOCP/2jhjAAAAAP/aOID/2jiO/9o4tP/aOEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BP/aOID/2jiO/9o4gP/aOID/2jiA/9o4Y//aOJz/2ji0/9o4nP/aOLT/2jiA/9o40v/aONL/2jjS/9o4nP/aOID/2jiO/9o4gP/aOGP/2jiA/9o4jv/aOGMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgE/9o4jv/aOJz/2jiO/9o4nP/aOOv/2jj+/9o4tP/aOID/2jiO/9o4jv/aOI7/2jjS/9o40v/aOID/2jic/9o4jv/aOID/2ji0/9o4/v/aOOv/2jic/9o4gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BP/aOID/2jjS/9o4YwAAAAD/2jgEAAAAAP/aOID/2jicAAAAAP/aOAQAAAAA/9o4Y//aONL/2jiA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAT/2jgE/9o4BP/aOAQAAAAAAAAAAP/aOCP/2ji0/9o46//aOGMAAAAA/9o4gP/aOJwAAAAA/9o4Y//aOOv/2ji0/9o4IwAAAAAAAAAA/9o4BP/aOAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAAAAAAD/2jhA/9o40v/aOLT/2jic/9o4tP/aOLT/2jjr/9o4QAAAAAAAAAAA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BAAAAAAAAAAA/9o4Y//aOOv/2jjr/9o4gP/aOAQAAAAA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BP/aOAQAAAAA/9o4gP/aOJwAAAAA/9o4BP/aOAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgj/9o4IwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA///DP///AA///AAP//wAD//8AA///AAP//wAD//+AA///gAf//xAH//8Ad///AEf//wBH//8AR///AMf//wDH//8Ax///B8f//wfH//8AxMf///zHx//+CABP/gAAB//zzz///Mz///8Dx8///8oAAAAEAAAACAAAAABACAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jh4/9o4rv/aODX/2jhW/9o4HwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOGj/2jjL/9o4/f/aOP3/2jj9/9o4/f/aOOL/2jgMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aODX/2jj9/9o4/f/aOP3/2jj9/9o4/f/aOP3/2jj9/9o4NQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgf/9o44v/aOP3/2jj9/9o4/f/aOP3/2jj9/9o44v/aOAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOFb/2jjL/9o4/f/aOP3/2jj9/9o4/f/aOK4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAz/2jji/9o4y//aOK7/2jjL/9o4eP/aOJL/2jhoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jg1/9o4/f/aOP3/2jj9/9o4/f/aOHj/2jh4/9o4eAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4DP/aOMv/2jjL/9o4y//aOOL/2jg1/9o4hv/aOHgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jh4/9o4hv/aOIb/2jiS/9o4H//aOIb/2jh4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4kv/aOK7/2jiG/9o4Nf/aOAz/2jiG/9o4eAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOFb/2jhW/9o4aP/aOJL/2jgf/9o4hv/aOHgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOB//2jho/9o4H//aOIb/2jh4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jg1/9o4eP/aOJL/2jiu/9o4eP/aOHj/2jjL/9o4kv/aOGj/2jiu/9o4kv/aODUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgM/9o4aP/aOIb/2jgf/9o4aP/aOB//2jiG/9o4aP/aOAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgf/9o4eP/aOMv/2jh4/9o4HwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jhWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP3/AADwPwAA4D8AAOA/AADwPwAA4X8AAOH/AADhfwAA8X8AAON/AAD9fwAA/38AAPMnAAD93wAA/38AAP//AAA=" />
<meta name="msapplication-config" content="4806ce9049e1e082dd3da4063ceb0eea.xml" />
<meta name="theme-color" content="#ffffff" />
<link href="vendor.1.css" rel="stylesheet"><link href="main.a22eefad1e7c2ab4e884.css" rel="stylesheet"></head>
<body ng-controller="MainController">
<div
id="page-wrapper"
ng-class="{
open: toggle && ['portainer.auth', 'portainer.init.admin', 'portainer.init.endpoint'].indexOf($state.current.name) === -1,
nopadding: ['portainer.auth', 'portainer.init.admin', 'portainer.init.endpoint', 'portainer.logout'].indexOf($state.current.name) > -1 || applicationState.loading
}"
ng-cloak
>
<div id="sideview" ui-view="sidebar" ng-if="!applicationState.loading"></div>
<div id="content-wrapper">
<div class="page-content">
<div class="page-wrapper" ng-if="applicationState.loading">
<!-- loading box -->
<div class="container simple-box">
<div class="col-md-6 col-md-offset-3 col-sm-6 col-sm-offset-3">
<!-- loading box logo -->
<div class="row">
<img ng-if="logo" ng-src="{{ logo }}" class="simple-box-logo" />
<img ng-if="!logo" src="5da83cfb4883a59354abeff852cb7394.png" class="simple-box-logo" alt="Portainer" />
</div>
<!-- !loading box logo -->
<!-- panel -->
<div class="row" style="text-align: center;">
Loading Portainer...
<i class="fa fa-cog fa-spin" style="margin-left: 5px;"></i>
</div>
<!-- !panel -->
</div>
</div>
<!-- !loading box -->
</div>
<!-- Main Content -->
<div id="view" ui-view="content" ng-if="!applicationState.loading"></div> </div
><!-- End Page Content --> </div
><!-- End Content Wrapper --> </div
><!-- End Page Wrapper -->
<script type="text/javascript" src="vendor.a22eefad1e7c2ab4e884.js"></script><script type="text/javascript" src="main.a22eefad1e7c2ab4e884.js"></script></body></html
>
3.5 通过互联网访问
由于本文网络拓扑是nat+maas+k8s,故需要在nat上添加ip重定向放可访问内网node。
可以参考Ubuntu 20.04防火墙 UFW做NAT转换,IP伪装,端口重定向,端口映射配置,将公网地址的某端口,重定向到10.0.4.102:30777端口。类似的,也可重定向到10.0.4.106:30777或10.0.4.108:30777
3.6首先配置用户名密码。
3.7 下一步是将您的 Portainer 实例连接到您的 Kubernetes 集群:
3.8 最后一步是通过启用指标服务器功能在 Portainer 上配置集群的 Kubernetes 功能:
3.9 并通过 RWO、RWX 和卷扩展启用 openebs-jiva-default 存储类:
3.10 单击“保存配置”,您的 Charmed Kubernetes 集群就可以使用 Portainer 进行管理了!!
卸载openebs
卸载openebs,当进行到kubectl delete namespace openebs
时,
输出如下:
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Error from server (InternalError): Internal error occurred: failed calling webhook "admission-webhook.openebs.io": Post "https://admission-server-svc.openebs.svc:443/validate?timeout=5s": service "admission-server-svc" not found
使用kubectl get namespace
验证时,
kubectl get namespace
NAME STATUS AGE
admin Active 25h
controller-my-controller Active 6h46m
default Active 4d4h
ingress-nginx-kubernetes-worker Active 4d4h
kube-node-lease Active 4d4h
kube-public Active 4d4h
kube-system Active 4d4h
kubernetes-dashboard Active 4d4h
openebs Active 3d10h
portainer Active 3d10h
经过查找文档,发现需要编辑配置文件如下:
kubectl edit validatingwebhookconfigurations openebs-validation-webhook-cfg
将此项变更为:failurePolicy: Ignore
编辑后,再执行
kubectl delete namespace openebs --force --grace-period=0
输出为:
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
namespace "openebs" force deleted
验证:
kubectl get namespace
NAME STATUS AGE
admin Active 25h
controller-my-controller Active 6h50m
default Active 4d4h
ingress-nginx-kubernetes-worker Active 4d4h
kube-node-lease Active 4d4h
kube-public Active 4d4h
kube-system Active 4d4h
kubernetes-dashboard Active 4d4h
portainer Active 3d10h