1.环境准备
1.1、准备好3台服务器，这里我以两台服务器搭建的k8s集群，还有一台用来搭建好harbor远程仓库
1.2、准备好构建fdfs所需的安装包
1.3、准备好fastdfs中相关配置文件模板
2、编写创建fastdfs镜像的Dockerfile脚本
内容如下：
FROM centos:7 MAINTAINER kgf <kgf@123.com> ENV FASTDFS_PATH=/opt/fdfs \ FASTDFS_BASE_PATH=/opt/fdfs \ PORT= \ GROUP_NAME= \ TRACKER_SERVER= #install all the dependences RUN yum install -y net-tools \ && yum install -y gcc gcc-c++ libevent libevent-devel pcre pcre-devel zlib zlib-devel openssl openssl-devel wget vim automake autoconf libtool make perl unzip #create the dirs to store the files downloaded from internet RUN mkdir -p ${FASTDFS_PATH} \ && mkdir -p ${FASTDFS_PATH}/client \ && mkdir -p ${FASTDFS_PATH}/files \ && mkdir -p ${FASTDFS_PATH}/tracker \ && mkdir -p ${FASTDFS_PATH}/nginx_mod #add libfastcommon,fastdfs,fastdfs-nginx-module,nginx to image ADD libfastcommon-1.0.36.tar.gz ${FASTDFS_PATH} ADD fastdfs-5.11.tar.gz ${FASTDFS_PATH} ADD fastdfs-nginx-module-1.20.tar.gz ${FASTDFS_PATH} ADD nginx-1.18.0.tar.gz ${FASTDFS_PATH} #compile the libfastcommon RUN cd ${FASTDFS_PATH}/libfastcommon-1.0.36/ \ && ./make.sh \ && ./make.sh install \ && rm -rf ${FASTDFS_PATH}/libfastcommon-1.0.36 # compile the fastdfs RUN cd ${FASTDFS_PATH}/fastdfs-5.11/ \ && ./make.sh \ && ./make.sh install \ && rm -rf ${FASTDFS_PATH}/fastdfs-5.11 #解决/usr/local/include/fastdfs/fdfs_define.h:15:27: 致命错误：common_define.h：没有那个文件或目录 RUN cd ${FASTDFS_PATH}/fastdfs-nginx-module-1.20/src/ \ && sed -i 's|ngx_module_incs="/usr/local/include"|ngx_module_incs="/usr/include/fastdfs /usr/include/fastcommon/"|g' ${FASTDFS_PATH}/fastdfs-nginx-module-1.20/src/config \ && sed -i 's|CORE_INCS="$CORE_INCS /usr/local/include"|CORE_INCS="$CORE_INCS /usr/include/fastdfs /usr/include/fastcommon/"|g' ${FASTDFS_PATH}/fastdfs-nginx-module-1.20/src/config #copy config file COPY conf/*.* /etc/fdfs/ RUN rm -rf /etc/fdfs/nginx.conf #compile the nginx RUN cd ${FASTDFS_PATH}/nginx-1.18.0/ \ && ./configure --prefix=/usr/local/nginx_fdfs --add-module=${FASTDFS_PATH}/fastdfs-nginx-module-1.20/src/ \ && make \ && make install \ && rm -rf ${FASTDFS_PATH}/nginx-1.18.0 #copy config file RUN rm -rf /usr/local/nginx_fdfs/conf/nginx.conf COPY conf/nginx.conf /usr/local/nginx_fdfs/conf # 默认nginx端口 ENV IP="localhost" # 默认fastdfs端口 EXPOSE 22122 23000 8080 8888 80 VOLUME ["$FASTDFS_BASE_PATH", "/etc/fdfs"] COPY conf/*.* /etc/fdfs/ COPY start.sh /usr/bin/ #make the start.sh executable RUN chmod 777 /usr/bin/start.sh ENTRYPOINT ["/usr/bin/start.sh"] CMD ["tracker"]
3、编写start.sh脚本
内容如下：
#!/bin/bash #set -e if [ "$1" = "monitor" ] ; then if [ -n "$TRACKER_SERVER" ] ; then sed -i "s|tracker_server=.*$|tracker_server=${TRACKER_SERVER}|g" /etc/fdfs/client.conf fi fdfs_monitor /etc/fdfs/client.conf exit 0 elif [ "$1" = "storage" ] ; then FASTDFS_MODE="storage" sed -i "s|store_path0.*$|store_path0=/opt/fdfs/files|g" /etc/fdfs/mod_fastdfs.conf sed -i "s|url_have_group_name =.*$|url_have_group_name = true|g" /etc/fdfs/mod_fastdfs.conf /usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf else FASTDFS_MODE="tracker" fi if [ -n "$PORT" ] ; then sed -i "s|^port=.*$|port=${PORT}|g" /etc/fdfs/"$FASTDFS_MODE".conf fi if [ -n "$TRACKER_SERVER" ] ; then sed -i "s|tracker_server=.*$|tracker_server=${TRACKER_SERVER}|g" /etc/fdfs/storage.conf sed -i "s|tracker_server=.*$|tracker_server=${TRACKER_SERVER}|g" /etc/fdfs/client.conf sed -i "s|tracker_server=.*$|tracker_server=${TRACKER_SERVER}|g" /etc/fdfs/mod_fastdfs.conf fi if [ -n "$GROUP_NAME" ] ; then sed -i "s|group_name=.*$|group_name=${GROUP_NAME}|g" /etc/fdfs/storage.conf fi FASTDFS_LOG_FILE="${FASTDFS_BASE_PATH}/logs/${FASTDFS_MODE}d.log" PID_NUMBER="${FASTDFS_BASE_PATH}/data/fdfs_${FASTDFS_MODE}d.pid" echo "try to start the $FASTDFS_MODE node..." if [ -f "$FASTDFS_LOG_FILE" ]; then rm "$FASTDFS_LOG_FILE" fi # start the fastdfs node. fdfs_${FASTDFS_MODE}d /etc/fdfs/${FASTDFS_MODE}.conf start # wait for pid file(important!),the max start time is 5 seconds,if the pid number does not appear in 5 seconds,start failed. TIMES=5 while [ ! -f "$PID_NUMBER" -a $TIMES -gt 0 ] do sleep 1s TIMES=`expr $TIMES - 1` done # if the storage node start successfully, print the started time. # if [ $TIMES -gt 0 ]; then # echo "the ${FASTDFS_MODE} node started successfully at $(date +%Y-%m-%d_%H:%M)" # # give the detail log address # echo "please have a look at the log detail at $FASTDFS_LOG_FILE" # # leave balnk lines to differ from next log. # echo # echo # # make the container have foreground process(primary commond!) # tail -F --pid=`cat $PID_NUMBER` /dev/null # # else print the error. # else # echo "the ${FASTDFS_MODE} node started failed at $(date +%Y-%m-%d_%H:%M)" # echo "please have a look at the log detail at $FASTDFS_LOG_FILE" # echo # echo # fi tail -f "$FASTDFS_LOG_FILE"
4、使用docker命令运行Dockerfile脚本打成镜像并且推送到harbor仓库
4.1、命令如下：
docker build -t hub.harbor.com/library/kgf-fdfs:v1 .
4.2、推送harbor
命令：docker push hub.harbor.com/library/kgf-fdfs:v1
5、fdfs部署分为两部分，一个是tracker一个是storage,下面我们先部署tracker
5.1、创建tracker存储pvc和pv,这里我们使用的是阿里云的oss,详细配置参考k8s存储之挂载阿里云oss对象存储
tracker.pvc.yaml内容如下：apiVersion: v1 kind: PersistentVolumeClaim metadata: namespace: kgf-dev name: tracker-oss-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi selector: matchLabels: alicloud-pvname: oss-tracker-pvtracker.pv.yaml内容如下：
apiVersion: v1 kind: PersistentVolume metadata: namespace: kgf-dev name: oss-tracker-pv labels: alicloud-pvname: oss-tracker-pv spec: capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain csi: driver: ossplugin.csi.alibabacloud.com volumeHandle: oss-tracker-pv volumeAttributes: bucket: "kgf" url: "oss-cn-shanghai.aliyuncs.com" otherOpts: "-o max_stat_cache_size=0 -o allow_other" akId: "*******" akSecret: "********" path: "/"5.2、创建tracker的yaml配置文件，因为pod ip可能会发生变化本次使用有状态statefulset控制器，tracker-statefulset.yaml内容如下：
apiVersion: apps/v1 kind: StatefulSet metadata: namespace: kgf-dev labels: app: tracker #statefulset名为tracker创建的pod名以该名为基础按顺序为tacker-0 1 2 3... name: tracker spec: replicas: 1 selector: matchLabels: app: tracker #serviceName名为tracker不要与statefulset名混淆,pod的解析需要在pod名后加该名例如tracker-0.tracker.default即$pod名.$serviceName.$命名空间 serviceName: "tracker" template: metadata: labels: app: tracker spec: containers: - image: hub.harbor.com/library/kgf-fdfs:v1 name: fastdfs #容器启动命令本次为使用tracker command: ["/bin/bash", "/usr/bin/start.sh","tracker"] volumeMounts: - name: tracker-oss-pvc mountPath: "/dhnasfile" volumes: - name: tracker-oss-pvc persistentVolumeClaim: claimName: tracker-oss-pvc5.3、创建svc因为是有状态服务所以不需要创建ClusterIP，tracker-cluster-svc.yaml内容如下
apiVersion: v1 kind: Service metadata: namespace: kgf-dev labels: app: tracker name: tracker spec: ports: - port: 22122 protocol: TCP targetPort: 22122 clusterIP: None selector: app: tracker5.4、执行上面的tracker相关脚本
命令：
kubectl apply -f tracker-pv.yaml
kubectl apply -f tracker-pvc.yaml
kubectl apply -f tracker-cluster-svc.yaml
kubectl apply -f tracker-statefulset.yaml
5.5、查看创建的pvc pv pod svc
查看pv和pvc命令：
kubectl get pv -n kgf-dev
kubectl get pvc-n kgf-dev
查看svc和pod命令：
kubectl get svc -n kgf-dev
kubectl get pods -n kgf-dev
5.6、登录tracker容器查看
命令：kubectl exec -it tracker-0 -n kgf-dev /bin/sh
5.7、在容器内部的/dhnasfile目录下创建一个测试文件，看有没有挂载到OSS
6、创建storage服务
6.1、创建storage存储pvc和pv
storage-pv.yaml内容如下：
apiVersion: v1 kind: PersistentVolume metadata: namespace: kgf-dev name: oss-storage-pv labels: alicloud-pvname: oss-storage-pv spec: capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain csi: driver: ossplugin.csi.alibabacloud.com volumeHandle: oss-storage-pv volumeAttributes: bucket: "kgf" url: "oss-cn-shanghai.aliyuncs.com" otherOpts: "-o max_stat_cache_size=0 -o allow_other" akId: "*****" akSecret: "*****" path: "/"storage-pvc.yaml内容如下：
apiVersion: v1 kind: PersistentVolumeClaim metadata: namespace: kgf-dev name: storage-oss-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi selector: matchLabels: alicloud-pvname: oss-storage-pv6.2、定义storage的statefulset，storage-statefulset.yaml内容如下
apiVersion: apps/v1 kind: StatefulSet metadata: namespace: kgf-dev labels: app: storage name: storage spec: replicas: 1 selector: matchLabels: app: storage serviceName: "storage" template: metadata: labels: app: storage spec: containers: - image: hub.harbor.com/library/kgf-fdfs:v1 name: fastdfs #该容器启动为storage服务 command: ["/bin/bash", "/usr/bin/start.sh","storage"] env: #定义环境变量 - name: TRACKER_SERVER #tracker服务器地址及IP DNS解析地址为$pod名.$serviceName.$命名空间 value: tracker-0.tracker.kgf-dev:22122 #定义存储组名 - name: GROUP_NAME value: group1 volumeMounts: - name: storage-oss-pvc mountPath: "/dhnasfile" volumes: - name: storage-oss-pvc persistentVolumeClaim: claimName: storage-oss-pvc6.3、定义storage的svc，storage-cluster-svc.yaml内容如下
apiVersion: v1 kind: Service metadata: namespace: kgf-dev labels: app: storage name: storage spec: ports: - port: 23000 name: storage # 指定端口的名称 protocol: TCP targetPort: 23000 - port: 80 name: nginx # 指定端口的名称 protocol: TCP targetPort: 80 clusterIP: None selector: app: storage type: ClusterIP6.4、执行上面的脚本
命令：
kubectl apply -f storage-pv.yaml
kubectl apply -f storage-pvc.yaml
kubectl apply -f storage-cluster-svc.yaml
kubectl apply -f storage-statefulset.yaml
6.5、查看创建的pv pvc svc 已经pod
6.6、进入容器内部查看应用
命令：kubectl exec -it storage-0 -n kgf-dev /bin/sh
7.6、查看storage是否和tracker通信
命令：fdfs_monitor /etc/fdfs/storage.conf
7.7、文件上传测试
访问默认nginx端口是80所以加端口访问：
7、创建Ingress服务，在外部访问，k8s-storage-nginx.yaml内容如下
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: k8s-storage-nginx
namespace: kgf-dev #命名空间 接下来的部署都会使用这个命名空间
spec:
rules:
- host: kgf.storage-nginx.com #外部访问的域名
http:
paths:
- path: /group1
backend:
serviceName: storage #服务对应的service名称
servicePort: 80 #对应service的端口
命令执行：kubectl apply -f k8s-storage-nginx.yaml
在外部配置hosts:
访问刚刚上传的文件：
2376
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
