ubuntu20.04下使用juju+maas环境部署k8s-11-在charmed k8s上部署portainer

最新推荐文章于 2024-06-19 16:41:36 发布

injexengge

最新推荐文章于 2024-06-19 16:41:36 发布

阅读量2.1k

点赞数

分类专栏： juju k8s portainer 文章标签： docker 运维 ssh

本文链接：https://blog.csdn.net/m0_49212388/article/details/120080791

版权

juju 同时被 3 个专栏收录

60 篇文章 2 订阅

订阅专栏

k8s

16 篇文章 1 订阅

订阅专栏

portainer

1 篇文章 0 订阅

订阅专栏

参考文档：

Deploy Portainer on Charmed Kubernetes

Using Portainer to manage Canonical Charmed Kubernetes
原文：

1 概述

如果您对 Kubernetes 的复杂性感到害怕，但仍需要使用容器对您的业务应用程序进行现代化改造，请放心，您不是唯一一个。容器编排平台解决了许多问题，但也带来了新问题，因此请继续阅读以了解一种可以帮助您获得收益的新方法。

什么是Charmed Kubernetes？

Charmed Kubernetes 是 Canonical 用于数据中心的纯上游 Kubernetes。它允许定制 Kubernetes 部署，因为它支持多个云基础设施和大量 CNCF landscape工具集。 Charmed Kubernetes 基于 Juju 生命周期自动化框架，消除了运行 Kubernetes 的复杂性。

什么是Portainer？

Portainer 是一个开源容器管理工具，适用于 Kubernetes、Docker、Docker Swarm 和 Azure ACI。它允许通过直观的用户界面部署和管理容器，而无需编写代码。

本教程的一些先决条件：

通过 ssh 终端访问 Linux 机器，最好是最新版本的 Ubuntu；您需要在将用于部署的机器上安装 snap 包管理应用程序；
访问 AWS、GCP 或 Azure 等云服务提供商。我们将在本教程中使用 Azure 作为CSP，因此请确保您知道如果您使用不同的提供程序，什么可以是等效的接口/命令。

2 初始配置

在初始设置期间，您将学习如何安装设置 Charmed Kubernetes 和部署集群所需的工具。

安装需要的工具：

1 在您的终端上运行以下命令

snap install juju --classic
snap install kubectl --classic
apt -y install nmap -> 这是可选的，可用于检查你的节点的公共 IP 地址上打开的端口

3. 与您的云提供商一起设置 Juju

1 使用 juju 添加云提供商凭据：juju add-credential azure
2 系统将提示您为此凭证命名。例如，您可以使用 azure
3 下一个提示是选择要用于部署的区域。我选择了eastus：
4 下一步是选择身份验证类型。您只需按 Enter 即可继续使用 [interactive]：
5 下一步是键入要用于部署 Charmed Kubernetes 群集的 Azure 订阅 ID：
6 将出现一个完成凭据设置的链接，您需要在浏览器中打开该链接并粘贴代码以进行身份验证。每次通过 juju 添加凭据时都会生成一个唯一的代码：
7 如果一切顺利，以下消息将出现在您的浏览器窗口中：

4 部署 Charmed Kubernetes 集群

1 启动您将在云提供商上部署的 juju 控制器的引导程序： juju bootstrap azure 确保将 name 替换为您的偏好之一。在这个例子中，我将使用charmed01。此过程最多可能需要 15 分钟

2 部署您的 Charmed Kubernetes 集群：

对于完整的集群，您可以使用 juju deploy Charmed-kubernetes。这将部署至少 8 台具有2个冗余主节点的机器。
另一种方法是使用一个主节点 + 一个工作节点部署一个较小的集群，然后添加更多节点： juju deploy cs:bundle/kubernetes-core-1200
使用 juju add-unit kubernetes-worker -n 2 添加至少 2 个节点。您可以在上面的第 2 步之后立即运行它。

3 您可以使用 watch -c juju status --color 监控集群的部署进度：

4 这至少需要 20 分钟以上。如果一切顺利，您应该会在屏幕上看到以下状态：

您可以按 CTRL+C 退出 juju 状态屏幕并返回到终端窗口上的提示。

5 确保您的主文件夹中有一个名为 .kube 的文件夹。如果不创建，则使用 mkdir -p $HOME/.kube

6 从主节点复制配置文件，以便使用 kubectl 管理集群： juju scp kubernetes-master/0:config ~/.kube/config

7 运行快速测试以确保您可以使用 kubectl 和 kubectl get node访问您的集群。您应该会在集群上看到您的工作节点：

5 部署 OpenEBS

OpenEBS 是 Kubernetes 的绝佳存储选项。请访问他们的网站以了解有关此项目的更多信息。

1 首先在所有节点上启用 iscsid 服务。

juju ssh kubernetes-master/0 'sudo systemctl enable --now iscsid'

2 通过将 kubernetes-master/0 替换为 kubernetes-worker/0 对每个工作节点重复此命令，例如：

 juju ssh kubernetes-worker/0 'sudo systemctl enable --now iscsid'

3 在您的主节点上启用运行特权服务：

 juju config kubernetes-master allow-privileged=true

4 下载在集群上部署 OpenEBS 所需的 yaml 文件：

wget https://openebs.github.io/charts/openebs-operator.yaml

5 即使我们在主节点上启用了运行特权服务，我们仍然需要对文件中的特权字符串进行一些小的更改：

sed -i 's/privileged: true/privileged: false/g' openebs-operator.yaml && cat openebs-operator.yaml

6 使用 kubectl 部署 OpenEBS：

kubectl apply -f openebs-operator.yaml

7 检查 OpenEBS 部署的状态：

 kubectl get pod -n openebs

8 一旦所有容器在您的集群上成功运行，部署将完成：

 kubectl get pod -n openebs
NAME                                           READY   STATUS    RESTARTS   AGE
maya-apiserver-8679c68656-mwljv                1/1     Running   0          6m36s
openebs-admission-server-75758ffd7c-h9lb8      1/1     Running   0          6m31s
openebs-localpv-provisioner-7d77896b9f-hl5ch   1/1     Running   0          6m30s
openebs-ndm-jkhb5                              1/1     Running   0          6m32s
openebs-ndm-lc5r8                              1/1     Running   0          6m32s
openebs-ndm-operator-6658f6756d-hsgvz          1/1     Running   0          6m31s
openebs-ndm-sw8mt                              1/1     Running   0          6m32s
openebs-provisioner-58c95774d9-dqzxp           1/1     Running   0          6m36s
openebs-snapshot-operator-5d748758df-wx47z     2/2     Running   0          6m34s

9 最后，确保 openebs-jiva-default storageclass 设置为默认值：

kubectl patch storageclass openebs-jiva-default -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

输出类似：

storageclass.storage.k8s.io/openebs-jiva-default patched

10 运行以确保补丁有效：

kubectl get sc

输出类似：

NAME                             PROVISIONER                                                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
openebs-device                   openebs.io/local                                           Delete          WaitForFirstConsumer   false                  7m8s
openebs-hostpath                 openebs.io/local                                           Delete          WaitForFirstConsumer   false                  7m9s
openebs-jiva-default (default)   openebs.io/provisioner-iscsi                               Delete          Immediate              false                  7m15s
openebs-snapshot-promoter        volumesnapshot.external-storage.k8s.io/snapshot-promoter   Delete          Immediate              false                  7m10s

这是必需的，否则无法安装 Portainer。

6 部署Portainer

1 安装 Portainer 非常简单。您只需要运行以下命令即可使用 NodePort 安装 Portainer：

kubectl apply -n portainer -f https://raw.githubusercontent.com/portainer/k8s/master/deploy/manifests/portainer/portainer.yaml

2 您可以使用 kubectl get pod -n portainer 检查安装状态。一旦您看到 STATUS 正在运行，那么 Portainer 就可以使用了：
在这里插入图片描述

3 Portainer，当部署在 NodePort 上时将在端口 30777 上运行。我们需要在 Azure（或您部署集群的 CSP）上打开这个端口，以确保它可以在您节点的公共 IP 上访问。您可以通过转到 Azure 门户上的“所有资源”页面并在 juju-default-xxxxxxxx 资源组中选择 juju-internal-nsg 来执行此操作：

4 单击入站安全规则：

5 创建一个规则，将打开端口 30777，如下所示：

6 您应该可以通过使用任何节点的公共 IP 地址 + 端口 30777 打开浏览器来访问 Portainer。在我的情况下，我的主节点的公共 IP 是 20.83.8.178，因此打开 Portainer 的链接是
http:// 20.83.8.178:30777：

在此页面上，您需要为 Portainer 设置管理员用户。

7 下一步是将您的 Portainer 实例连接到您的 Kubernetes 集群：

8 最后一步是通过启用指标服务器功能在 Portainer 上配置集群的 Kubernetes 功能：

9 并通过 RWO、RWX 和卷扩展启用 openebs-jiva-default 存储类：

10 单击“保存配置”，您的 Charmed Kubernetes 集群就可以使用 Portainer 进行管理了！！

7 就这样，伙计们

以下是实际操作：

1 由于已经按照前面文章ubuntu20.04下使用juju+maas环境部署k8s-2-部署charmed kubernetes #679部署charmed k8s，所以本文的章节3-设置juju和章节4-部署charmed kubernetes部分可以省略。

2 根据章节5部署openebs。

3 部署和配置portainer。
3.1 部署portainer

kubectl apply -n portainer -f https://raw.githubusercontent.com/portainer/k8s/master/deploy/manifests/portainer/portainer.yaml

3.2 检查portainer安装状态，显示status为running.

kubectl get pod -n portainer

3.3 显示node-ip。
实际就是kubernetes-worker节点ip。
两种办法：

juju status kubernetes-worker

或

kubectl get node -o wide

在本文中，node ip为：
10.0.4.102
10.0.4.106
10.0.4.108

node-port为：30777

3.4 在maas 服务器上，验证登陆。

curl 10.0.4.102:30777

输出为：

<!DOCTYPE html
><html lang="en" ng-app="portainer" ng-strict-di>
  <head>
    <meta charset="utf-8" />
    <title>Portainer</title>
    <meta name="description" content="" />
    <meta name="author" content="Portainer.io" />

    <!-- HTML5 shim, for IE6-8 support of HTML5 elements -->
    <!--[if lt IE 9]>
      <script src="//html5shim.googlecode.com/svn/trunk/html5.js"></script>
    <![endif]-->

    <!-- Fav and touch icons -->
    <link rel="apple-touch-icon" sizes="180x180" href="dc4d092847be46242d8c013d1bc7c494.png" />
    <link rel="icon" type="image/png" sizes="32x32" href="5ba13dcb526292ae707310a54e103cd1.png" />
    <link rel="icon" type="image/png" sizes="16x16" href="f9508a64a1beb81be174e194573f7450.png" />
    <link rel="mask-icon" href="07745d55b001c85826eedd479285cdbb.svg" color="#5bbad5" />
    <link rel="shortcut icon" href="data:image/vnd.microsoft.icon;base64,AAABAAMAMDAAAAEAIACoJQAANgAAACAgAAABACAAqBAAAN4lAAAQEAAAAQAgAGgEAACGNgAAKAAAADAAAABgAAAAAQAgAAAAAAAAJAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAAAAAAAAAAAAAAAAAA/9o4BAAAAAAAAAAAAAAAAP/aOAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAT/2jgEAAAAAP/aOAT/2jh8/9o4x//aONj/2jjH/9o4fP/aOBoAAAAAAAAAAAAAAAD/2jgE/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAAAAAAAAAAAA/9o4BP/aOMf/2jj//9o4///aOP//2jj//9o4///aONj/2jgy/9o4fP/aONj/2jjw/9o42P/aOK3/2jgaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4Gv/aOFr/2jhG/9o4rf/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jjw/9o4///aOP//2jj//9o4///aOP//2jjw/9o4MgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jh8/9o48P/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o42P/aOAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOJL/2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOFoAAAAA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4Mv/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOJIAAAAA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4fP/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOHwAAAAA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4kv/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOEYAAAAA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4fP/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4xwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4Mv/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4xwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOJL/2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o42AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jiS/9o48P/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4rQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4Mv/aOFr/2jh8/9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4WgAAAAD/2jgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BAAAAAD/2jha/9o48P/aONj/2jhG/9o4rf/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jitAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aOEb/2jj//9o4///aOP//2jjH/9o4Gv/aOPD/2jj//9o4///aOP//2jj//9o4///aONj/2jiS/9o48P/aOP//2jj//9o48P/aOHz/2jgaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BP/aONj/2jj//9o4///aOP//2jj//9o4kv/aOBr/2jiS/9o48P/aOPD/2jjw/9o4kv/aOEb/2jgE/9o4Gv/aOEb/2jha/9o4Gv/aOHz/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4Rv/aOP//2jj//9o4///aOP//2jj//9o4///aOMf/2jha/9o4Rv/aOEb/2jhG/9o4fP/aOPD/2jhGAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4fP/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOPD/2jj//9o4///aOP//2jh8AAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4kv/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o48P/aOP//2jh8AAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4kv/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jh8AAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4Wv/aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jhGAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4Gv/aONj/2jj//9o48P/aOPD/2jj//9o48P/aOPD/2jjw/9o4///aOPD/2jjw/9o4///aONj/2jgEAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOFr/2jiS/9o4fP/aOHz/2jha/9o4fP/aOHz/2jh8/9o4Wv/aOHz/2jh8/9o4kv/aOEYAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aOHz/2jit/9o4rf/aOJL/2jiS/9o4rf/aOK3/2jit/9o4kv/aOK3/2jit/9o4rf/aOFoAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aOHz/2jh8/9o4fP/aOHz/2jiS/9o4fP/aOHz/2jh8/9o4kv/aOHz/2jh8/9o4fP/aOFoAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aOHz/2jit/9o4kv/aOJL/2jiS/9o4kv/aOJL/2jiS/9o4kv/aOK3/2jiS/9o4rf/aOFoAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOFr/2jit/9o4rf/aOJL/2jha/9o4rf/aOK3/2jit/9o4Wv/aOHz/2jh8/9o4fP/aODIAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aOHz/2jit/9o4rf/aOJL/2jh8/9o4rf/aOK3/2jit/9o4RgAAAAD/2jgEAAAAAAAAAAAAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aOHz/2jiS/9o4fP/aOHz/2jiS/9o4fP/aOHz/2jiS/9o4Rv/aODL/2jgy/9o4Mv/aOBoAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aOHz/2jiS/9o4kv/aOJL/2jiS/9o4kv/aOJL/2jiS/9o4kv/aOMf/2jit/9o4x//aOFoAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOFr/2jiS/9o4kv/aOHz/2jha/9o4kv/aOJL/2jiS/9o4fP/aOJL/2jh8/9o4fP/aOFoAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4Rv/aOK3/2jiS/9o4kv/aOFoAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgE/9o4BP/aOAQAAAAA/9o4BP/aOAQAAAAA/9o4Wv/aOMf/2jh8/9o4x//aOHwAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4Wv/aOHwAAAAA/9o4Wv/aOHwAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAP/aOAT/2jgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4Wv/aOHwAAAAA/9o4Wv/aOHwAAAAAAAAAAP/aODL/2jj//9o4Mv/aOK3/2jjHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jga/9o4Mv/aODL/2jgy/9o4Mv/aODL/2jgy/9o4Mv/aODL/2jga/9o4fP/aOJL/2jga/9o4fP/aOJL/2jga/9o4Mv/aOFr/2jj//9o4Wv/aOMf/2jjY/9o4Gv/aODL/2jgy/9o4Mv/aODL/2jgy/9o4Mv/aODL/2jgy/9o4Mv/aODL/2jgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BAAAAAD/2jh8/9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jj//9o4///aOP//2jhGAAAAAP/aOAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jga/9o4Rv/aOEb/2jhG/9o4Rv/aOEb/2jh8/9o48P/aOP//2jjH/9o4Rv/aODL/2jhG/9o4Rv/aOEb/2jhG/9o4Rv/aOFr/2jj//9o4fP/aODL/2jhG/9o4Rv/aOEb/2jhG/9o4Mv/aOEb/2jjH/9o4///aOPD/2jh8/9o4Rv/aOEb/2jgaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4Gv/aOJL/2jj//9o42P/aOEYAAAAAAAAAAAAAAAAAAAAAAAAAAP/aODL/2jj//9o4RgAAAAD/2jgEAAAAAAAAAAAAAAAA/9o4Rv/aONj/2jj//9o4kv/aOBoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BP/aOAT/2jgE/9o4BP/aOAT/2jgEAAAAAAAAAAD/2jgy/9o4x//aOP//2jjH/9o4MgAAAAD/2jgEAAAAAP/aODL/2jj//9o4RgAAAAD/2jgEAAAAAP/aODL/2jjH/9o4///aOMf/2jgyAAAAAAAAAAD/2jgE/9o4BP/aOAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAAAAAAAP/aOFr/2jjw/9o4///aOJL/2jgaAAAAAP/aODL/2jj//9o4RgAAAAD/2jga/9o4kv/aOP//2jjw/9o4WgAAAAAAAAAA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgE/9o4BAAAAAD/2jgE/9o4fP/aOP//2jjw/9o4Wv/aODL/2jj//9o4Rv/aOFr/2jjw/9o4///aOHz/2jgEAAAAAP/aOAT/2jgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAAAAAAAP/aOBr/2jit/9o4///aOPD/2jj//9o48P/aOP//2jit/9o4GgAAAAAAAAAA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BAAAAAAAAAAA/9o4Mv/aONj/2jj//9o42P/aOEYAAAAAAAAAAP/aOAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aODL/2jj//9o4RgAAAAD/2jgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aODL/2jjw/9o4RgAAAAD/2jgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP///wAA//+PAAD//gMP//8AAP/8AAf//wAA/+AAA///AAD/gAAD//8AAP+AAAH//wAA/4AAA///AAD/AAAD//8AAP+AAAP//wAA/4AAA///AAD/gAAD//8AAP/AAAP//wAA//wAB///AAD/5AAH//8AAP/CAB///wAA/4EH9///AAD/gPun//8AAP+AA6f//wAA/wADp///AAD/AAOn//8AAP+AA6f//wAA/4ADp///AAD/3/en//8AAP/AB6f//wAA//u/p///AAD/wAen//8AAP/Ef6f//wAA/8R/p///AAD/23+n//8AAP/AB6f//wAA/8xfp///AAD//8en//8AAP//16f//wAAp///AAD///+n//8AAP//26f//wAA/wAAAAD/AAD//H+/x/8AAP/+P7+P/wAA//+Pvj//AAD//+O4//8AAP//+bP//wAA///8B///AAD///8f//8AAP///7///wAAv///AAD///8AACgAAAAgAAAAQAAAAAEAIAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BP/aOAT/2jgEAAAAAP/aOCP/2jhA/9o4IwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAT/2jic/9o4/v/aOP7/2jj+/9o4Y//aOCP/2jic/9o4nP/aOGMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOED/2jiO/9o4tP/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOI4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jiO/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOCMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4QP/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4YwAAAAD/2jgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BAAAAAD/2jhj/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jhAAAAAAP/aOAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aOED/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o40gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BP/aONL/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jjS/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4I//aOI7/2jjS/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOLQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgE/9o4jv/aOID/2jiO/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4QAAAAAD/2jgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOJz/2jj+/9o4/v/aOGP/2ji0/9o4/v/aOP7/2jj+/9o4jv/aOI7/2jjS/9o4tP/aOID/2jgjAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgj/9o4/v/aOP7/2jj+/9o46//aOID/2jiA/9o4jv/aOI7/2jiAAAAAAP/aOGP/2jhj/9o4tP/aOEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAP/aOGP/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOOv/2jjr/9o4/v/aOOsAAAAA/9o4gP/aOI7/2ji0/9o4QAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4Y//aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o46wAAAAD/2jiA/9o4jv/aOLT/2jhAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jhA/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jj+/9o4/v/aOP7/2jjSAAAAAP/aOID/2jiO/9o4tP/aOEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jiO/9o4tP/aOJz/2jic/9o4tP/aOJz/2jic/9o4tP/aOEAAAAAA/9o4gP/aOI7/2ji0/9o4QAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOID/2jiO/9o4gP/aOI7/2jiO/9o4gP/aOI7/2jiO/9o4QAAAAAD/2jiA/9o4jv/aOLT/2jhAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4gP/aOI7/2jiO/9o4jv/aOI7/2jiO/9o4nP/aOJz/2jhAAAAAAP/aOID/2jiO/9o4tP/aOEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jiA/9o4tP/aOID/2jiO/9o4tP/aOGP/2jhA/9o4QP/aOCMAAAAA/9o4gP/aOI7/2ji0/9o4QAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOID/2jiO/9o4gP/aOI7/2jiO/9o4Y//aOAT/2jgj/9o4BAAAAAD/2jiA/9o4jv/aOLT/2jhAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4gP/aOLT/2jiO/9o4nP/aOJz/2jiO/9o4nP/aOLT/2jhAAAAAAP/aOID/2jiO/9o4tP/aOEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgj/9o4I//aOCP/2jgj/9o4I//aOED/2jiO/9o4jv/aOEAAAAAA/9o4gP/aOI7/2ji0/9o4QAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BAAAAAAAAAAAAAAAAP/aOAQAAAAA/9o4QP/aOI7/2jiA/9o4YwAAAAD/2jiO/9o4jv/aOLT/2jhAAAAAAP/aOAT/2jgE/9o4BP/aOAT/2jgE/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jhA/9o4QP/aOCP/2jhjAAAAAP/aOID/2jiO/9o4tP/aOEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BP/aOID/2jiO/9o4gP/aOID/2jiA/9o4Y//aOJz/2ji0/9o4nP/aOLT/2jiA/9o40v/aONL/2jjS/9o4nP/aOID/2jiO/9o4gP/aOGP/2jiA/9o4jv/aOGMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgE/9o4jv/aOJz/2jiO/9o4nP/aOOv/2jj+/9o4tP/aOID/2jiO/9o4jv/aOI7/2jjS/9o40v/aOID/2jic/9o4jv/aOID/2ji0/9o4/v/aOOv/2jic/9o4gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BP/aOID/2jjS/9o4YwAAAAD/2jgEAAAAAP/aOID/2jicAAAAAP/aOAQAAAAA/9o4Y//aONL/2jiA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAT/2jgE/9o4BP/aOAQAAAAAAAAAAP/aOCP/2ji0/9o46//aOGMAAAAA/9o4gP/aOJwAAAAA/9o4Y//aOOv/2ji0/9o4IwAAAAAAAAAA/9o4BP/aOAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgEAAAAAAAAAAD/2jhA/9o40v/aOLT/2jic/9o4tP/aOLT/2jjr/9o4QAAAAAAAAAAA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BAAAAAAAAAAA/9o4Y//aOOv/2jjr/9o4gP/aOAQAAAAA/9o4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4BP/aOAQAAAAA/9o4gP/aOJwAAAAA/9o4BP/aOAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgj/9o4IwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA///DP///AA///AAP//wAD//8AA///AAP//wAD//+AA///gAf//xAH//8Ad///AEf//wBH//8AR///AMf//wDH//8Ax///B8f//wfH//8AxMf///zHx//+CABP/gAAB//zzz///Mz///8Dx8///8oAAAAEAAAACAAAAABACAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jh4/9o4rv/aODX/2jhW/9o4HwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOGj/2jjL/9o4/f/aOP3/2jj9/9o4/f/aOOL/2jgMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aODX/2jj9/9o4/f/aOP3/2jj9/9o4/f/aOP3/2jj9/9o4NQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgf/9o44v/aOP3/2jj9/9o4/f/aOP3/2jj9/9o44v/aOAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOFb/2jjL/9o4/f/aOP3/2jj9/9o4/f/aOK4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOAz/2jji/9o4y//aOK7/2jjL/9o4eP/aOJL/2jhoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jg1/9o4/f/aOP3/2jj9/9o4/f/aOHj/2jh4/9o4eAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4DP/aOMv/2jjL/9o4y//aOOL/2jg1/9o4hv/aOHgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jh4/9o4hv/aOIb/2jiS/9o4H//aOIb/2jh4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/9o4kv/aOK7/2jiG/9o4Nf/aOAz/2jiG/9o4eAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOFb/2jhW/9o4aP/aOJL/2jgf/9o4hv/aOHgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/aOB//2jho/9o4H//aOIb/2jh4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jg1/9o4eP/aOJL/2jiu/9o4eP/aOHj/2jjL/9o4kv/aOGj/2jiu/9o4kv/aODUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgM/9o4aP/aOIb/2jgf/9o4aP/aOB//2jiG/9o4aP/aOAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jgf/9o4eP/aOMv/2jh4/9o4HwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2jhWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP3/AADwPwAA4D8AAOA/AADwPwAA4X8AAOH/AADhfwAA8X8AAON/AAD9fwAA/38AAPMnAAD93wAA/38AAP//AAA=" />
    <meta name="msapplication-config" content="4806ce9049e1e082dd3da4063ceb0eea.xml" />
    <meta name="theme-color" content="#ffffff" />
  <link href="vendor.1.css" rel="stylesheet"><link href="main.a22eefad1e7c2ab4e884.css" rel="stylesheet"></head>

  <body ng-controller="MainController">
    <div
      id="page-wrapper"
      ng-class="{
        open: toggle && ['portainer.auth', 'portainer.init.admin', 'portainer.init.endpoint'].indexOf($state.current.name) === -1,
        nopadding: ['portainer.auth', 'portainer.init.admin', 'portainer.init.endpoint', 'portainer.logout'].indexOf($state.current.name) > -1 || applicationState.loading
      }"
      ng-cloak
    >
      <div id="sideview" ui-view="sidebar" ng-if="!applicationState.loading"></div>

      <div id="content-wrapper">
        <div class="page-content">
          <div class="page-wrapper" ng-if="applicationState.loading">
            <!-- loading box -->
            <div class="container simple-box">
              <div class="col-md-6 col-md-offset-3 col-sm-6 col-sm-offset-3">
                <!-- loading box logo -->
                <div class="row">
                  <img ng-if="logo" ng-src="{{ logo }}" class="simple-box-logo" />
                  <img ng-if="!logo" src="5da83cfb4883a59354abeff852cb7394.png" class="simple-box-logo" alt="Portainer" />
                </div>
                <!-- !loading box logo -->
                <!-- panel -->
                <div class="row" style="text-align: center;">
                  Loading Portainer...
                  <i class="fa fa-cog fa-spin" style="margin-left: 5px;"></i>
                </div>
                <!-- !panel -->
              </div>
            </div>
            <!-- !loading box -->
          </div>

          <!-- Main Content -->
          <div id="view" ui-view="content" ng-if="!applicationState.loading"></div> </div
        ><!-- End Page Content --> </div
      ><!-- End Content Wrapper --> </div
    ><!-- End Page Wrapper -->
  <script type="text/javascript" src="vendor.a22eefad1e7c2ab4e884.js"></script><script type="text/javascript" src="main.a22eefad1e7c2ab4e884.js"></script></body></html
>

3.5 通过互联网访问

由于本文网络拓扑是nat+maas+k8s，故需要在nat上添加ip重定向放可访问内网node。

可以参考Ubuntu 20.04防火墙 UFW做NAT转换，IP伪装，端口重定向，端口映射配置，将公网地址的某端口，重定向到10.0.4.102:30777端口。类似的，也可重定向到10.0.4.106:30777或10.0.4.108:30777

3.6首先配置用户名密码。

![在这里插入图片描述](https://img-blog.csdnimg.cn/6c4da28c704c4a108a379426c6b4fe43.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBAaW5qZXhlbmdnZQ==,size_20,color_FFFFFF,t_70,g_se,x_16

3.7 下一步是将您的 Portainer 实例连接到您的 Kubernetes 集群：

在这里插入图片描述
3.8 最后一步是通过启用指标服务器功能在 Portainer 上配置集群的 Kubernetes 功能：

3.9 并通过 RWO、RWX 和卷扩展启用 openebs-jiva-default 存储类：

3.10 单击“保存配置”，您的 Charmed Kubernetes 集群就可以使用 Portainer 进行管理了！！

在这里插入图片描述

卸载openebs

Uninstalling OpenEBS

failed calling webhook “admission-webhook.openebs.io”: Post https://admission-server-svc.openebs.svc:443/validate?timeout=5s: context deadline exceeded #3046

卸载openebs，当进行到kubectl delete namespace openebs时，
输出如下：

warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Error from server (InternalError): Internal error occurred: failed calling webhook "admission-webhook.openebs.io": Post "https://admission-server-svc.openebs.svc:443/validate?timeout=5s": service "admission-server-svc" not found

使用kubectl get namespace验证时，

 kubectl get namespace
NAME                              STATUS   AGE
admin                             Active   25h
controller-my-controller          Active   6h46m
default                           Active   4d4h
ingress-nginx-kubernetes-worker   Active   4d4h
kube-node-lease                   Active   4d4h
kube-public                       Active   4d4h
kube-system                       Active   4d4h
kubernetes-dashboard              Active   4d4h
openebs                           Active   3d10h
portainer                         Active   3d10h

经过查找文档，发现需要编辑配置文件如下：

kubectl edit validatingwebhookconfigurations  openebs-validation-webhook-cfg

将此项变更为：failurePolicy: Ignore

编辑后，再执行

kubectl delete namespace openebs --force --grace-period=0

输出为：

 warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
namespace "openebs" force deleted

验证：

kubectl get namespace
NAME                              STATUS   AGE
admin                             Active   25h
controller-my-controller          Active   6h50m
default                           Active   4d4h
ingress-nginx-kubernetes-worker   Active   4d4h
kube-node-lease                   Active   4d4h
kube-public                       Active   4d4h
kube-system                       Active   4d4h
kubernetes-dashboard              Active   4d4h
portainer                         Active   3d10h

injexengge

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
ubuntu20.04下使用juju+maas环境部署k8s-11-在charmed k8s上部署portainer

参考文档：Deploy Portainer on Charmed KubernetesUsing Portainer to manage Canonical Charmed Kubernetes原文：1 概述如果您对 Kubernetes 的复杂性感到害怕，但仍需要使用容器对您的业务应用程序进行现代化改造，请放心，您不是唯一一个。容器编排平台解决了许多问题，但也带来了新问题，因此请继续阅读以了解一种可以帮助您获得收益的新方法。什么是Charmed Kubernetes？Charmed Kube
复制链接

扫一扫

专栏目录