香橙派4和树莓派4B构建K8S集群实践之四：BuildKit与LNMP

bennybi

已于 2023-07-08 09:58:16 修改

阅读量928

点赞数 2

分类专栏： K8s 文章标签： kubernetes lnmp nginx buildkit 云原生

于 2023-05-11 22:36:50 首次发布

本文链接：https://blog.csdn.net/bennybi/article/details/130561998

版权

K8s 专栏收录该内容

9 篇文章 0 订阅

订阅专栏

3.1 申请一个pvc存储区 (wwwroot-pvc.yaml)

3.2 Nginx

3.3 PHP-FPM

3.3.1 构建并推送镜像

3.3.2 定义入口(ingress 方式)

1. 说明

k8s带来的灵活性，使我们在部署上面有了很多花样选择，这里我希望做到以下方案：

Plan:

- namespace为iot-age, nginx 与 php分别部署到不同的pods, 它们共用相同的pvc存储（即wwwroot目录相同，应用部署在里面）, php pods 访问相同的数据库。

- 使用buildkit作为image构建工具，通过指定镜像部署定制化的php pods。

PlanB: namespace为ia2, nginx与php及其应用联合到一个deployment.yaml部署，有自己对应的存储目录, pods 共用相同的数据库。

2. 开始前的准备工作

2.1 docker 验证用户信息设置

因为将使用BuildKit来访问docker仓库（buildkit使用~/.docker/config.json）, 需要验证用户信息，先行进行设置，以免push时出现验证错误：

# 生成base64编码的auth值, 其中user, pw为hub.docker.com的账密

echo -n "{user}:{pw}" | base64

vi ~/.docker/config.json
# example:
{
  "auths": {
          "https://index.docker.io/v1/": {
               "auth": "{auth}"
          }
  }
}

2.2 安装BuildKit

法1：在指定主机上安装使用

# amd64用这个
wget https://github.com/moby/buildkit/releases/download/v0.9.0/buildkit-v0.9.0.linux-amd64.tar.gz
tar xvf buildkit-v0.9.0.linux-amd64.tar.gz -C /usr/local/

# arm64用这个
wget https://github.com/moby/buildkit/releases/download/v0.9.0/buildkit-v0.9.0.linux-arm64.tar.gz
tar xvf buildkit-v0.9.0.linux-arm64.tar.gz -C /usr/local/

# 创建服务配置
cat >/etc/systemd/system/buildkitd.service <<EOF
[Unit]
Description=BuildKit
Documentation=https://github.com/moby/buildkit
After=containerd.service

[Service]
Type=notify
ExecStart=/usr/local/bin/buildkitd  --oci-worker=false --containerd-worker=true 

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload && systemctl restart buildkitd 
systemctl enable buildkitd
systemctl status buildkitd

法2：创建 buildkitd和buildkit-cli pods，并在buildkit-cli里调用服务构建，需要说明的是: 我尚未能成功构建并推送上去仓库，可参考这篇文章Building Docker Images with BuildKit | Kubernetes Course Labs

# 安装buildkit server
kubectl apply -f https://kubernetes.courselabs.co/labs/buildkit/specs/buildkitd/buildkitd.yaml
kubectl logs -l app=buildkitd
# 删除buildkit
kubectl delete deploy,svc,pod -l kubernetes.courselabs.co=buildkit

kubectl create secret docker-registry registry-creds --docker-server=docker.io --docker-username={user} --docker-password={pw}
kubectl create configmap build-config --from-literal=REGISTRY=docker.io  --from-literal=REPOSITORY=bennybi

kubectl apply -f https://kubernetes.courselabs.co/labs/buildkit/specs/buildkit-cli/buildkit-cli.yaml

# 进入buildkit cli pod
kubectl exec -it buildkit-cli -- sh

# 在buildkit-cli pod内调用
buildctl --addr tcp://buildkitd:1234 build --frontend=dockerfile.v0 --local context=. --local dockerfile=. --output type=image,name=${REGISTRY}/${REPOSITORY}/simple,push=true

3. 安装步骤

3.1 申请一个pvc存储区 (wwwroot-pvc.yaml)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wwwroot
  namespace: iot-age
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2G

3.2 Nginx

文件清单: nginx.yaml, nginx-configmap.yaml

iot-age中所有的nginx pods将使用统一的配置，ConfigMap (nginx-configmap.yaml)

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
  namespace: iot-age
data:
  default.conf: |
    server {
        listen       80;
        listen  [::]:80;
        server_name  localhost;
        
        index index.php index.html;
        error_log  /var/log/nginx/error.log;
        access_log /var/log/nginx/access.log;
        root /var/www/html;
        location / {
            try_files $uri $uri/ /index.php?$query_string;
        }
        
        location ~ \.php$ {
            try_files $uri =404;
            fastcgi_split_path_info ^(.+\.php)(/.+)$;
            fastcgi_pass   php:9000;
            fastcgi_index  index.php;
            include        fastcgi_params;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_param PATH_INFO $fastcgi_path_info;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }
    }

kubectl apply -f nginx-configmap.yaml

nginx.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx
  namespace: iot-age
spec:
  selector:
    matchLabels:
      app: nginx
  serviceName: "nginx"
  replicas: 2
  #volumeClaimTemplates:
  #  - kind: PersistentVolumeClaim
  #    apiVersion: v1
  #    metadata:
  #      name: wwwroot
  #    spec:
  #      accessModes:
  #        - ReadWriteOnce
  #      resources:
  #        requests:
  #          storage: 2Gi
  #      volumeMode: Filesystem
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
        volumeMounts:
        - name: wwwroot
          mountPath: /var/www/html
        - name: nginx-conf
          mountPath: /etc/nginx/conf.d/
      volumes:
      - name: wwwroot
        persistentVolumeClaim:
          claimName: wwwroot
      - name: nginx-conf
        configMap:
         name: nginx-conf
status: {}

---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: iot-age
spec:
  type: ClusterIP
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800
  ports:
  - name: nginx
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx

#---
#apiVersion: v1
#kind: Service
#metadata:
#  name: nginx
#  namespace: iot-age
#spec:
#  type: NodePort
#  ports:
#  - name: nginx
#    port: 80
#    protocol: TCP
#    targetPort: 80
#    nodePort: 30010
#  selector:
#    app: nginx

kubectl apply -f nginx.yaml

3.3 PHP-FPM

文件清单: Dockerfile, php.yaml

Dockerfile，该文件参考了php/Dockerfile at master · khs1994-docker/php · GitHub

# syntax=docker/dockerfile:labs
ARG PHP_VERSION=8.2.5

FROM --platform=$TARGETPLATFORM php:${PHP_VERSION}-fpm-alpine3.17 as php

LABEL maintainer="khs1994-docker/lnmp <khs1994@khs1994.com>"

ARG PHP_EXTENSION_EXTRA

ARG PECL_EXTENSION_EXTRA

ARG APK_EXTRA

ARG APK_DEV_EXTRA

ENV TZ=Asia/Shanghai \
    APP_ENV=development

ENV PHP_EXTENSION \
      bcmath \
      bz2 \
      calendar \
      enchant \
      exif \
      gd \
      gettext \
      gmp \
      imap \
      intl \
      mysqli \
      pcntl \
      pdo_pgsql \
      pdo_mysql \
      pgsql \
      sockets \
      sysvmsg \
      sysvsem \
      sysvshm \
      # tidy \
      # xsl \
      zip \
      shmop \
      ffi \
      ${PHP_EXTENSION_EXTRA:-}

ENV PECL_EXTENSION \
      # mongodb \
      # https://github.com/mongodb/mongo-php-driver/archive/master.tar.gz \
      # igbinary \
      https://github.com/igbinary/igbinary/archive/master.tar.gz \
      # redis \
      https://github.com/phpredis/phpredis/archive/develop.tar.gz \
      memcached \
# 安装测试版的扩展，可以在扩展名后加 -beta
      # yaml \
      # https://github.com/php/pecl-file_formats-yaml/archive/php7.tar.gz \
      https://github.com/kjdev/php-ext-zstd/archive/0.11.0.tar.gz \
      ${PECL_EXTENSION_EXTRA:-}

ARG ALPINE_URL=dl-cdn.alpinelinux.org

RUN sed -i "s/dl-cdn.alpinelinux.org/${ALPINE_URL}/g" /etc/apk/repositories \
      && set -xe \
      \
      # && apk add --no-cache patch \
      # && cd / \
      # && { \
      #   echo "--- a/usr/local/lib/php/build/php.m4"; \
      #    } | tee 1.diff \
      # && patch -p1 < 1.diff \
      # && apk del --no-network patch \
      # && rm -rf 1.diff \
      \
# 不要删除
      && PHP_FPM_RUN_DEPS=" \
                         bash \
                         tzdata \
                         libmemcached-libs \
                         libpq \
                         libzip \
                         zlib \
                         libpng \
                         freetype \
                         libjpeg-turbo \
                         libxpm \
                         libwebp \
                         libbz2 \
                         libexif \
                         gmp \
                         enchant2 \
                         c-client \
                         icu-libs \
                         zstd-libs \
                         linux-headers \
                         libavif \
                         ${APK_EXTRA:-} \
                         " \
                         # tidyhtml-libs \
                         # libxslt \
                         # yaml \
# *-dev 编译之后删除
      && PHP_FPM_BUILD_DEPS=" \
                         openssl-dev \
                         libmemcached-dev \
                         cyrus-sasl-dev \
                         postgresql-dev \
                         libzip-dev \
                         zlib-dev \
                         libpng-dev \
                         freetype-dev \
                         libjpeg-turbo-dev \
                         libxpm-dev \
                         libwebp-dev \
                         libexif-dev \
                         gmp-dev \
                         bzip2-dev \
                         enchant2-dev \
                         imap-dev \
                         gettext-dev \
                         libwebp-dev \
                         icu-dev \
                         zstd-dev \
                         libavif-dev \
                         ${APK_DEV_EXTRA:-} \
                         " \
                         # tidyhtml-dev \
                         # libxslt-dev \
                         # yaml-dev \
        && apk add --no-cache --virtual .php-fpm-run-deps $PHP_FPM_RUN_DEPS \
        && apk add --no-cache --virtual .php-fpm-build-deps $PHP_FPM_BUILD_DEPS \
        && apk add --no-cache --virtual .phpize-deps $PHPIZE_DEPS \
        && curl -fsSL -o /usr/local/bin/pickle \
           https://github.com/khs1994-php/pickle/releases/download/nightly/pickle-debug.phar \
        && chmod +x /usr/local/bin/pickle \
# 安装内置扩展
        && docker-php-source extract \
        && docker-php-ext-configure zip \
                                    --with-zip \
        && docker-php-ext-install zip \
        && strip --strip-all $(php-config --extension-dir)/zip.so \
        # && docker-php-ext-configure gd \
        && echo " \
                                        --disable-gd-jis-conv \
                                        --with-freetype \
                                        --with-jpeg \
                                        --with-webp \
                                        --with-xpm \
                                        --with-avif" > /tmp/gd.configure.options \
        # && docker-php-ext-install $PHP_EXTENSION \
        && pickle install $PHP_EXTENSION -n --defaults --strip \
        && docker-php-source delete \
# 安装 PECL 扩展
        && echo "--enable-redis-igbinary --enable-redis-zstd" > /tmp/redis.configure.options \
        && echo "--enable-memcached-igbinary" > /tmp/memcached.configure.options \
        && echo "--with-libzstd" > /tmp/zstd.configure.options \
        && pickle install $PECL_EXTENSION -n --defaults \
           --strip --cleanup \
# 默认不启用的扩展
        && pickle install \
# https://github.com/tideways/php-xhprof-extension.git
             https://github.com/tideways/php-xhprof-extension/archive/master.tar.gz \
             -n --defaults --strip --cleanup --no-write \
        && pickle install opcache \
        # && docker-php-ext-enable opcache \
        && apk del --no-network .phpize-deps .php-fpm-build-deps \
        && rm -rf /tmp/* \
# 创建日志文件夹
        && mkdir -p /var/log/php-fpm \
        && ln -sf /dev/stdout /var/log/php-fpm/access.log \
        && ln -sf /dev/stderr /var/log/php-fpm/error.log \
        && chmod -R 777 /var/log/php-fpm \
        && rm -rf /usr/local/lib/php/.registry/.channel.pecl.php.net/* \
        && php -m \
        && ls -la $(php-config --extension-dir) \
        && php -d error_reporting=22527 -d display_errors=1 -r 'var_dump(iconv("UTF-8", "UTF-8//IGNORE", "This is the Euro symbol '\''€'\''."));' \
# smoke test
        && php --version

WORKDIR /app

ARG VCS_REF="unknow"

LABEL org.opencontainers.image.revision=$VCS_REF \
      org.opencontainers.image.source="https://github.com/khs1994-docker/php"

3.3.1 构建并推送镜像

buildctl build \
--frontend=dockerfile.v0 \
--local context=. \
--local dockerfile=. \
--export-cache type=inline \
--output type=image,name=docker.io/bennybi/php-fpm:v1,push=true

php.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: php
  namespace: iot-age
spec:
  selector:
    matchLabels:
      app: php
      tier: backend
  replicas: 2
  template:
    metadata:
      labels:
        app: php
        tier: backend
    spec:
      containers:
      - name: php
        #image: php:8.2-fpm
        image: bennybi/php-fpm:v1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9000
        volumeMounts:
        - name: wwwroot
          mountPath: /var/www/html/
 
      volumes:
      - name: wwwroot
        persistentVolumeClaim:
          claimName: wwwroot
 
---
apiVersion: v1
kind: Service
metadata:
  name: php
  namespace: iot-age
spec:
  type: ClusterIP
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800
  ports:
  - name: php
    port: 9000
    protocol: TCP
    targetPort: 9000
  selector:
    app: php

kubectl apply -f php.yaml

3.3.2 定义入口(ingress 方式)

编写ingres-nginx.yaml，注意填写对应的 ingress.class是哪个

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ia-web-service1
  namespace: iot-age
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/session-cookie-name: stickounet
    nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
spec:
  rules:
  - host: "*.k8s-t1.com"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80
  ingressClassName: nginx

安装效果如图：

访问 http://t1.k8s-t1.com/

3.4 Maria Galera

用helm工具来安装maria-galera集群是相当方便的，但遇到的重启机器后crash问题解决却要找好久(见遇到的问题集合)。

3.4.1 安装相关命令

# 用在线源安装，如果有现存数据库，须加上现用的用密，否则重装会产生新的密码字典，造成pod内部root登录问题，不能启动!
helm install my-release oci://registry-1.docker.io/bitnamicharts/mariadb-galera \
    --set galera.mariabackup.password=aaa\
    --set rootUser.password=bbb\
    --set db.password=ccc\
    --set galera.bootstrap.forceSafeToBootstrap=true \
    --set galera.bootstrap.bootstrapFromNode=0

# 下载安装
helm list
helm search repo mariadb-galera
helm fetch bitnami/mariadb-galera --version 8.1.1
tar -zxvf mariadb-galera-8.1.1.tgz

# To uninstall/delete the my-release release:
helm delete my-release

# 临时建立个client pod来访问db
kubectl run my-release-mariadb-galera-client --rm --tty -i --restart='Never' --namespace default --image docker.io/bitnami/mariadb-galera:10.11.3-debian-11-r0 --command \
      -- mysql -h my-release-mariadb-galera -P 3306 -uroot -p$(kubectl get secret --namespace default my-release-mariadb-galera -o jsonpath="{.data.mariadb-root-password}" | base64 -d) my_database


设置本地端口转发远程pod端口
kubectl port-forward [portName] [LocalPort:RemotePort]

3.4.2 特别注意

在关停宿主机或K8s集群服务的时候，要先停掉mariadb-galera相关的pods, 不然重启后会出现一堆莫名错误（靠修改safe_to_bootstrap:1也不保证能顺利启动），执行：

kubectl scale --replicas=0 statefulset/my-release-mariadb-galera 暂停
kubectl scale --replicas=3 statefulset/my-release-mariadb-galera 恢复

如图：

或者，k8s启动完毕后，再重建pods (mariadb-galera-start.sh)

#!/bin/bash

export MARIADB_PASSWORD=$(kubectl get secret --namespace "default" my-release-mariadb-galera -o jsonpath="{.data.mariadb-password}" | base64 -d) && \
export MARIADB_ROOT_PASSWORD=$(kubectl get secret --namespace "default" my-release-mariadb-galera -o jsonpath="{.data.mariadb-root-password}" | base64 -d) && \
export MARIADB_GALERA_MARIABACKUP_PASSWORD=$(kubectl get secret --namespace "default" my-release-mariadb-galera -o jsonpath="{.data.mariadb-galera-mariabackup-password}" | base64 -d)

helm upgrade my-release \
    --set galera.mariabackup.password=$MARIADB_GALERA_MARIABACKUP_PASSWORD \
    --set rootUser.password=$MARIADB_ROOT_PASSWORD \
    --set db.password=$MARIADB_PASSWORD \
    --set galera.bootstrap.forceSafeToBootstrap=true \
    --set galera.bootstrap.bootstrapFromNode=0 ../mariadb-galera (values.yaml所在目录)

3.4.3 外部连接数据库

法一：临时

# 在192.168.0.106设置转发端口 => 服务端口
kubectl port-forward --namespace default svc/my-release-mariadb-galera 3306:3306 --address 127.0.0.1,192.168.0.106 &

法二：screen 方式

screen -S mariadb-galera
kubectl port-forward --namespace default svc/my-release-mariadb-galera 3306:3306 --address 127.0.0.1,192.168.0.106

# 离开会话并让程序断续运行：ctrl a d (按住ctrl不放，分别按 a 和 d)

法三: 配置ingress-nginx开放转发3306端口，另外更进一步可同时配置ingress-mariadb用子域名访问，详见香橙派4和树莓派4B构建K8S集群实践之五：端口公开访问配置_bennybi的博客-CSDN博客。

3.3.4 内部跨域访问服务

通过 {SERVICE_NAME}.{NAMESPACE_NAME}.svc.cluster.local 这样的格式，访问目标 namespace 下的服务。

如上述的db服务：my-release-mariadb-galera.default

4. 遇到的问题

buildctl build 并 push上去时，出现 "authorization status: 401: authorization failed"错误

解决办法：确认 ~/.docker/config.json已正确设置auth信息

重启k8s集群后遇到maria-galera不能启动的状况：

[ERROR] WSREP: It may not be safe to bootstrap the cluster from this node. It was not the last one to leave the cluster and may not contain all the updates. To force cluster bootstrap with this node, edit the grastate.dat file manually and set safe_to_bootstrap to 1 .

解决办法：

- 具体原因可参考: Unclean shutdown of mariadb-galera clusters leaves pod in CrashLoopBackOff state · Issue #2260 · bitnami/charts · GitHub

- 分别去db pods对应的pvc部署目录，找到grastate.dat文件，修改seqno最大的文件(如果都为0，就修改my-release-mariadb-galera-0)，把safe_to_bootstrap设为1，删除该pod后将自动重建。

# GALERA saved state
version: 2.1
uuid:    a550ccd5-f31f-11ed-8130-0ecae2f1ac87
seqno:   16
safe_to_bootstrap: 1

5. 相关命令

# 查看本地就绪镜像
ctr -n buildkit images list 或 ctr -n buildkit i ls

# 检查证书过期时间
kubeadm certs check-expiration

6. 参考

GitHub - moby/buildkit: concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit

Building Docker Images with BuildKit | Kubernetes Course Labs

Deploy a PHP Application on Kubernetes Cluster with Ubuntu 18.04

K8s---Pod搭建LNMP_k8s搭建lnmp_我不满意的博客-CSDN博客

k8s之StatefulSet详解_最美dee时光的博客-CSDN博客

buildkit的简单学习与使用

Helm Charts to deploy MariaDB Galera in Kubernetes

Unclean shutdown of mariadb-galera clusters leaves pod in CrashLoopBackOff state · Issue #2260 · bitnami/charts · GitHub

charts/bitnami/mariadb-galera at main · bitnami/charts · GitHub

bennybi

关注

2
点赞
踩
1

收藏

觉得还不错? 一键收藏
打赏
7
评论
香橙派4和树莓派4B构建K8S集群实践之四：BuildKit与LNMP

namespace为iot-age, nginx 与 php分别部署到不同的pods, 它们共用相同的pvc存储（即wwwroot目录相同，应用部署在里面）, php pods 访问相同的数据库。- 使用buildkit作为image构建工具，通过指定镜像部署定制化的php pods。
复制链接

扫一扫