python项目微服务化并部署在k8s上

已于 2022-04-27 10:59:47 修改
阅读量5.1k 收藏 12
点赞数 3
分类专栏： k8s docker flask 文章标签： docker kubernetes
于 2022-04-25 11:04:14 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_47058489/article/details/122111549
版权
docker 同时被 3 个专栏收录
28 篇文章 2 订阅
订阅专栏
k8s
25 篇文章 5 订阅
订阅专栏
flask
2 篇文章 0 订阅
订阅专栏

0 前提

创建tensorflow-gpu虚拟环境，参考这篇博客

1 尝试运行demo

# 进入之前创建好的tf虚拟环境
root@master:/home/hqc# source activate tf

# 进入含源码所在地址
(tf) root@master:/home/hqc# cd 自然基金项目/Federated/
(tf) root@master:/home/hqc/自然基金项目/Federated# 

# 运行程序
(tf) root@master:/home/hqc/自然基金项目/Federated# python main.py
# 会报错，依次conda install 对应包 即可！

# conda install 对应包 
(tf) root@master:/home/hqc/自然基金项目/Federated# conda install matplotlib
(tf) root@master:/home/hqc/自然基金项目/Federated# conda install numpy

# 再次运行程序
(tf) root@master:/home/hqc/自然基金项目/Federated# python main.py
	...
	...
	32/32 [==============================] - 0s 805us/step - loss: 0.0610 - accuracy: 0.9840
	Sever: 轮次: 99,准确率: 0.9840，共测试了10000张图片 
	Epoch 1/3
	7/7 [==============================] - 0s 1ms/step - loss: 0.0515 - accuracy: 0.9799
	Epoch 2/3
	7/7 [==============================] - 0s 2ms/step - loss: 0.0153 - accuracy: 1.0000
	Epoch 3/3
	7/7 [==============================] - 0s 1ms/step - loss: 0.0073 - accuracy: 1.0000
	Epoch 1/3
	6/6 [==============================] - 0s 1ms/step - loss: 0.0122 - accuracy: 1.0000
	Epoch 2/3
	6/6 [==============================] - 0s 2ms/step - loss: 0.0053 - accuracy: 1.0000
	Epoch 3/3
	6/6 [==============================] - 0s 1ms/step - loss: 0.0038 - accuracy: 1.0000
	Epoch 1/3
	16/16 [==============================] - 0s 1ms/step - loss: 0.0101 - accuracy: 0.9980
	Epoch 2/3
	16/16 [==============================] - 0s 1ms/step - loss: 0.0037 - accuracy: 1.0000
	Epoch 3/3
	16/16 [==============================] - 0s 1ms/step - loss: 0.0017 - accuracy: 1.0000
	Epoch 1/3
	1/1 [==============================] - 0s 2ms/step - loss: 0.0122 - accuracy: 1.0000
	Epoch 2/3
	1/1 [==============================] - 0s 2ms/step - loss: 0.0065 - accuracy: 1.0000
	Epoch 3/3
	1/1 [==============================] - 0s 1ms/step - loss: 0.0038 - accuracy: 1.0000
	32/32 [==============================] - 0s 775us/step - loss: 0.0595 - accuracy: 0.9800
	Sever: 轮次: 100,准确率: 0.9800，共测试了10000张图片 
	QStandardPaths: wrong ownership on runtime directory /run/user/1000, 1000 instead of 0

# 成功！

最终结果图

2 使用flask+html将demo微服务化

from flask import Flask,render_template
from flask import jsonify
import random

import matplotlib.pyplot as plt
# this module is used to draw a picture
# usually connected to 'numpy' module
# its usage just like matlab

import numpy as np
# this module is used to calculate or transform arrays and lists

from tensorflow.keras import datasets, layers, models
# tensorflow.keras is a high lever module for python API
# 'from tensorflow.keras import datasets' is used to download datasets
# 'import layers' is used to customize the layers of neural network
# 'import models' is used to customize the whole model of neural network

app = Flask(__name__)

# set number of rounds
BATCH = 100

# the images source
class DataSource(object):
    def __init__(self):
        (train_images, train_labels), (test_images, test_labels) = datasets.mnist.load_data()

        # 6万张训练图片，1万张测试图片
        train_images = train_images.reshape((60000, 28, 28, 1))
        test_images = test_images.reshape((10000, 28, 28, 1))

        # 像素值映射到 0 - 1 之间
        train_images, test_images = train_images / 255.0, test_images / 255.0
        self.train_images, self.train_labels = train_images[0:15000], train_labels[0:15000]
        self.test_images, self.test_labels = test_images[0:10000], test_labels[0:10000]


def random_num_with_fix_total(maxvalue, num):
    """生成总和固定的整数序列
    maxvalue: 序列总和
    num：要生成的整数个数"""
    a = random.sample(range(1, maxvalue), k=num - 1)  # 在1~99之间，采集20个数据
    a.append(0)  # 加上数据开头
    a.append(maxvalue)
    a = sorted(a)
    b = [a[count] - a[count - 1] for count in range(1, len(a))]  # 列表推导式，计算列表中每两个数之间的间隔
    return b


class DataSource1(object):
    def __init__(self):
        (train_images, train_labels), (test_images, test_labels) = datasets.mnist.load_data()
        # 6万张训练图片，1万张测试图片
        train_images = train_images.reshape((60000, 28, 28, 1))
        test_images = test_images.reshape((10000, 28, 28, 1))
        # 像素值映射到 0 - 1 之间
        train_images, test_images = train_images / 255.0, test_images / 255.0
        self.TI, self.TL = train_images[0:15000], train_labels[0:15000]
        self.train_images = np.empty(BATCH, dtype=object)
        self.train_labels = np.empty(BATCH, dtype=object)
        begin = 0
        rand_count = random_num_with_fix_total(15000, BATCH)
        for count in range(100):
            self.train_images[count] = self.TI[begin:(begin + rand_count[count])]
            self.train_labels[count] = self.TL[begin:(begin + rand_count[count])]
            begin = begin + rand_count[count]
        self.test_images, self.test_labels = test_images[0:10000], test_labels[0:10000]


class DataSource2(object):
    def __init__(self):
        (train_images, train_labels), (test_images, test_labels) = datasets.mnist.load_data()
        # 6万张训练图片，1万张测试图片
        train_images = train_images.reshape((60000, 28, 28, 1))
        test_images = test_images.reshape((10000, 28, 28, 1))
        # 像素值映射到 0 - 1 之间
        train_images, test_images = train_images / 255.0, test_images / 255.0
        self.TI, self.TL = train_images[15000:30000], train_labels[15000:30000]
        self.train_images = np.empty(BATCH, dtype=object)
        self.train_labels = np.empty(BATCH, dtype=object)
        begin = 0
        rand_count = random_num_with_fix_total(15000, BATCH)
        for count in range(100):
            self.train_images[count] = self.TI[begin:(begin + rand_count[count])]
            self.train_labels[count] = self.TL[begin:(begin + rand_count[count])]
            begin = begin + rand_count[count]


class DataSource3(object):
    def __init__(self):
        (train_images, train_labels), (test_images, test_labels) = datasets.mnist.load_data()
        # 6万张训练图片，1万张测试图片
        train_images = train_images.reshape((60000, 28, 28, 1))
        test_images = test_images.reshape((10000, 28, 28, 1))
        # 像素值映射到 0 - 1 之间
        train_images, test_images = train_images / 255.0, test_images / 255.0
        self.TI, self.TL = train_images[15000:30000], train_labels[15000:30000]
        self.train_images = np.empty(BATCH, dtype=object)
        self.train_labels = np.empty(BATCH, dtype=object)
        begin = 0
        rand_count = random_num_with_fix_total(15000, BATCH)
        for count in range(100):
            self.train_images[count] = self.TI[begin:(begin + rand_count[count])]
            self.train_labels[count] = self.TL[begin:(begin + rand_count[count])]
            begin = begin + rand_count[count]


class DataSource4(object):
    def __init__(self):
        (train_images, train_labels), (test_images, test_labels) = datasets.mnist.load_data()
        # 6万张训练图片，1万张测试图片
        train_images = train_images.reshape((60000, 28, 28, 1))
        test_images = test_images.reshape((10000, 28, 28, 1))
        # 像素值映射到 0 - 1 之间
        train_images, test_images = train_images / 255.0, test_images / 255.0
        self.TI, self.TL = train_images[15000:30000], train_labels[15000:30000]
        self.train_images = np.empty(BATCH, dtype=object)
        self.train_labels = np.empty(BATCH, dtype=object)
        begin = 0
        rand_count = random_num_with_fix_total(15000, BATCH)
        for count in range(100):
            self.train_images[count] = self.TI[begin:(begin + rand_count[count])]
            self.train_labels[count] = self.TL[begin:(begin + rand_count[count])]
            begin = begin + rand_count[count]


# Define as LeNet
class CNN(object):
    def __init__(self):
        model = models.Sequential()
        model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))
        model.add(layers.MaxPool2D((2, 2)))
        model.add(layers.Conv2D(64, (3, 3), activation='relu'))
        model.add(layers.MaxPool2D((2, 2)))
        model.add(layers.Conv2D(64, (3, 3), activation='relu'))
        model.add(layers.Flatten())
        model.add(layers.Dense(64, activation='relu'))
        model.add(layers.Dense(10, activation='softmax'))
        #        model.summary()  #打印网络结构
        self.model = model


# FedAvg Function
def FedAvg():
    weight_CNN_1 = np.load("Client1Weight.npy", allow_pickle=True)
    weight_CNN_2 = np.load("Client2Weight.npy", allow_pickle=True)
    weight_CNN_3 = np.load("Client3Weight.npy", allow_pickle=True)
    weight_CNN_4 = np.load("Client4Weight.npy", allow_pickle=True)
    weight_array = (weight_CNN_1 + weight_CNN_2 + weight_CNN_3 + weight_CNN_4) / 4
    weight_out = np.array(weight_array)
    return weight_out


# EKF Function
def EKF(cnn, weight_in):
    cnn.model.set_weights(weight_in)
    return cnn


# Create Models:LeNet
cnn_sever = CNN()
cnn1 = CNN()
cnn2 = CNN()
cnn3 = CNN()
cnn4 = CNN()
# Prepare Client Data
data_sever = DataSource()
data1 = DataSource1()
data2 = DataSource2()
data3 = DataSource3()
data4 = DataSource4()

# Compile Client and Sever Model
cnn_sever.model.compile(optimizer='adam',
                        loss='sparse_categorical_crossentropy',
                        metrics=['accuracy'])
cnn1.model.compile(optimizer='adam',
                   loss='sparse_categorical_crossentropy',
                   metrics=['accuracy'])
cnn2.model.compile(optimizer='adam',
                   loss='sparse_categorical_crossentropy',
                   metrics=['accuracy'])
cnn3.model.compile(optimizer='adam',
                   loss='sparse_categorical_crossentropy',
                   metrics=['accuracy'])
cnn4.model.compile(optimizer='adam',
                   loss='sparse_categorical_crossentropy',
                   metrics=['accuracy'])
storage_acc = []
weight = cnn_sever.model.get_weights()
np.save("SeverWeight", weight)
# All Clint Train
for i in range(BATCH):
    # Client Model Update(Downloads From Sever)
    weight = np.load("SeverWeight.npy", allow_pickle=True)
    #    cnn1.model.set_weights(weight)
    #    cnn2.model.set_weights(weight)
    #    cnn3.model.set_weights(weight)
    #    cnn4.model.set_weights(weight)
    cnn1 = EKF(cnn1, weight)
    cnn2 = EKF(cnn2, weight)
    cnn3 = EKF(cnn3, weight)
    cnn4 = EKF(cnn4, weight)
    # Client Model Fit
    cnn1.model.fit(data1.train_images[i], data1.train_labels[i], epochs=3)
    cnn2.model.fit(data2.train_images[i], data2.train_labels[i], epochs=3)
    cnn3.model.fit(data3.train_images[i], data3.train_labels[i], epochs=3)
    cnn4.model.fit(data4.train_images[i], data4.train_labels[i], epochs=3)
    # FedAvg

    weight_CNN1 = np.array(cnn1.model.get_weights())
    weight_CNN2 = np.array(cnn2.model.get_weights())
    weight_CNN3 = np.array(cnn3.model.get_weights())
    weight_CNN4 = np.array(cnn4.model.get_weights())


    np.save("Client1Weight", weight_CNN1)
    np.save("Client2Weight", weight_CNN2)
    np.save("Client3Weight", weight_CNN3)
    np.save("Client4Weight", weight_CNN4)
    weight = FedAvg()
    # Uploads to Sever
    cnn_sever.model.set_weights(weight)
    np.save("SeverWeight", weight)
    test_loss, test_acc = cnn_sever.model.evaluate(data_sever.test_images[0:1000], data_sever.test_labels[0:1000])
    print("Sever: 轮次: %d,准确率: %.4f，共测试了%d张图片 " % (i + 1, test_acc, len(data_sever.test_labels)))
    storage_acc = np.append(storage_acc, test_acc)
# Show Acc
x = np.array(range(100))
plt.plot(x, storage_acc)
plt.savefig('./static/acc.png')

@app.route('/')
def index():
    return render_template('index.html', weight = str(weight))

if __name__ == '__main__':
    app.run(host = '0.0.0.0')

关联的HTML文件：

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>RESULT</title>
</head>
<body>
    the picture of accuracy is here :
    <br>
    <br>
    <img src="/static/acc.png" width="1080px" height="720px">
    <br>
    <br>
    the sever's weight is :
    <br>
    <br>
    {{weight}}

</body>
</html>

注意：

  1. 导入Flask,render_template模块
  2. 一定要注意文件结构：图片要放在static文件夹下，html文件要放在templates文件夹下，不能都放在一个文件夹下（至于为啥，不知道，反正是不能运行）
  3. 最后的host必须为host = '0.0.0.0'否则docker run 之后不能在公网进行访问

3 容器镜像化

Dockerfile文件：

FROM python
RUN mkdir -p /Federated \
    && mkdir -p /Federated/templates \
    && mkdir -p /Federated/static \
    && pip install tensorflow \
    && pip install matplotlib \
    && pip install flask
COPY ./templates/index.html /Federated/templates
COPY main.py /Federated/
WORKDIR /Federated
EXPOSE 5000
RUN /bin/bash -c 'echo init ok'
CMD ["python", "main.py"]

注意：得自己pip安装所需要的包

4 创建deployment和service

federated-deployment.yaml文件：

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: federated-deployment
  name: federated-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: federated-deployment
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: federated-deployment
    spec:
      containers:
      - image: registry.cn-beijing.aliyuncs.com/hqc-k8s/federated:v1.0
        name: federated
        resources: {}
        ports:
        - containerPort: 5000
        imagePullPolicy: IfNotPresent
status: {}

federated-service.yaml文件：

apiVersion: v1 # 注意此处不能和deployment一样为‘apps/v1’
kind: Service
metadata:
  name: federated-deployment
  labels:
    app: federated-deployment
spec:
  ports:
  - port: 80
    targetPort: 5000
    nodePort: 30001
    protocol: TCP
  selector:
    app: federated-deployment
  type: NodePort

5 结果

1 命令行结果

root@master:/home/hqc/自然基金项目/Federated# kubectl get all
NAME                                         READY   STATUS        RESTARTS   AGE
pod/federated-deployment-5d5cfb4c7c-5bq9r    1/1     Running       0          12m
pod/federated-deployment-5d5cfb4c7c-mcfg2    1/1     Running       0          12m

NAME                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/federated-deployment    NodePort    10.107.96.50     <none>        80:30001/TCP   8m41s

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/federated-deployment    2/2     2            2           12m

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/federated-deployment-5d5cfb4c7c    2         2         2       12m

2 dashboard结果

在这里插入图片描述在这里插入图片描述在这里插入图片描述在这里插入图片描述

3 运行结果

在这里插入图片描述

6 升级微服务（添加选择下载成果文件功能）

1 main.py

  1. 添加了下载功能
  2. bootstrap优化html模块
  3. flash闪现消息模块
from flask import Flask,render_template
from flask import jsonify
import random

import matplotlib.pyplot as plt
# this module is used to draw a picture
# usually connected to 'numpy' module
# its usage just like matlab

import numpy as np
# this module is used to calculate or transform arrays and lists

from tensorflow.keras import datasets, layers, models
# tensorflow.keras is a high lever module for python API
# 'from tensorflow.keras import datasets' is used to download datasets
# 'import layers' is used to customize the layers of neural network
# 'import models' is used to customize the whole model of neural network

from flask_bootstrap import Bootstrap # 继承"bootstrap/base.html"模板
import os # 处理路径相关

from flask import flash # 用于提示
from flask import request # 访问请求需要用到
#from flask import redirect # 重定向
#from flask import url_for # 获取url
from flask import send_from_directory # 用于下载文件

#from werkzeug.utils import secure_filename # 当用户输入恶意字符时，对服务器进行保护

app = Flask(__name__)
bootstrap = Bootstrap(app)
# 编写html要想继承"bootstrap/base.html"文件就必须加上这一句

app.secret_key = 'sdafdsdfdasaf'
# 使用flash后需要设置秘钥，否则会报错，秘钥随机即可

# set number of rounds
BATCH = 100

# the images source
class DataSource(object):
    def __init__(self):
        (train_images, train_labels), (test_images, test_labels) = datasets.mnist.load_data()

        # 6万张训练图片，1万张测试图片
        train_images = train_images.reshape((60000, 28, 28, 1))
        test_images = test_images.reshape((10000, 28, 28, 1))

        # 像素值映射到 0 - 1 之间
        train_images, test_images = train_images / 255.0, test_images / 255.0
        self.train_images, self.train_labels = train_images[0:15000], train_labels[0:15000]
        self.test_images, self.test_labels = test_images[0:10000], test_labels[0:10000]


def random_num_with_fix_total(maxvalue, num):
    """生成总和固定的整数序列
    maxvalue: 序列总和
    num：要生成的整数个数"""
    a = random.sample(range(1, maxvalue), k=num - 1)  # 在1~99之间，采集20个数据
    a.append(0)  # 加上数据开头
    a.append(maxvalue)
    a = sorted(a)
    b = [a[count] - a[count - 1] for count in range(1, len(a))]  # 列表推导式，计算列表中每两个数之间的间隔
    return b


class DataSource1(object):
    def __init__(self):
        (train_images, train_labels), (test_images, test_labels) = datasets.mnist.load_data()
        # 6万张训练图片，1万张测试图片
        train_images = train_images.reshape((60000, 28, 28, 1))
        test_images = test_images.reshape((10000, 28, 28, 1))
        # 像素值映射到 0 - 1 之间
        train_images, test_images = train_images / 255.0, test_images / 255.0
        self.TI, self.TL = train_images[0:15000], train_labels[0:15000]
        self.train_images = np.empty(BATCH, dtype=object)
        self.train_labels = np.empty(BATCH, dtype=object)
        begin = 0
        rand_count = random_num_with_fix_total(15000, BATCH)
        for count in range(100):
            self.train_images[count] = self.TI[begin:(begin + rand_count[count])]
            self.train_labels[count] = self.TL[begin:(begin + rand_count[count])]
            begin = begin + rand_count[count]
        self.test_images, self.test_labels = test_images[0:10000], test_labels[0:10000]


class DataSource2(object):
    def __init__(self):
        (train_images, train_labels), (test_images, test_labels) = datasets.mnist.load_data()
        # 6万张训练图片，1万张测试图片
        train_images = train_images.reshape((60000, 28, 28, 1))
        test_images = test_images.reshape((10000, 28, 28, 1))
        # 像素值映射到 0 - 1 之间
        train_images, test_images = train_images / 255.0, test_images / 255.0
        self.TI, self.TL = train_images[15000:30000], train_labels[15000:30000]
        self.train_images = np.empty(BATCH, dtype=object)
        self.train_labels = np.empty(BATCH, dtype=object)
        begin = 0
        rand_count = random_num_with_fix_total(15000, BATCH)
        for count in range(100):
            self.train_images[count] = self.TI[begin:(begin + rand_count[count])]
            self.train_labels[count] = self.TL[begin:(begin + rand_count[count])]
            begin = begin + rand_count[count]


class DataSource3(object):
    def __init__(self):
        (train_images, train_labels), (test_images, test_labels) = datasets.mnist.load_data()
        # 6万张训练图片，1万张测试图片
        train_images = train_images.reshape((60000, 28, 28, 1))
        test_images = test_images.reshape((10000, 28, 28, 1))
        # 像素值映射到 0 - 1 之间
        train_images, test_images = train_images / 255.0, test_images / 255.0
        self.TI, self.TL = train_images[15000:30000], train_labels[15000:30000]
        self.train_images = np.empty(BATCH, dtype=object)
        self.train_labels = np.empty(BATCH, dtype=object)
        begin = 0
        rand_count = random_num_with_fix_total(15000, BATCH)
        for count in range(100):
            self.train_images[count] = self.TI[begin:(begin + rand_count[count])]
            self.train_labels[count] = self.TL[begin:(begin + rand_count[count])]
            begin = begin + rand_count[count]


class DataSource4(object):
    def __init__(self):
        (train_images, train_labels), (test_images, test_labels) = datasets.mnist.load_data()
        # 6万张训练图片，1万张测试图片
        train_images = train_images.reshape((60000, 28, 28, 1))
        test_images = test_images.reshape((10000, 28, 28, 1))
        # 像素值映射到 0 - 1 之间
        train_images, test_images = train_images / 255.0, test_images / 255.0
        self.TI, self.TL = train_images[15000:30000], train_labels[15000:30000]
        self.train_images = np.empty(BATCH, dtype=object)
        self.train_labels = np.empty(BATCH, dtype=object)
        begin = 0
        rand_count = random_num_with_fix_total(15000, BATCH)
        for count in range(100):
            self.train_images[count] = self.TI[begin:(begin + rand_count[count])]
            self.train_labels[count] = self.TL[begin:(begin + rand_count[count])]
            begin = begin + rand_count[count]


# Define as LeNet
class CNN(object):
    def __init__(self):
        model = models.Sequential()
        model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))
        model.add(layers.MaxPool2D((2, 2)))
        model.add(layers.Conv2D(64, (3, 3), activation='relu'))
        model.add(layers.MaxPool2D((2, 2)))
        model.add(layers.Conv2D(64, (3, 3), activation='relu'))
        model.add(layers.Flatten())
        model.add(layers.Dense(64, activation='relu'))
        model.add(layers.Dense(10, activation='softmax'))
        #        model.summary()  #打印网络结构
        self.model = model


# FedAvg Function
def FedAvg():
    weight_CNN_1 = np.load("Client1Weight.npy", allow_pickle=True)
    weight_CNN_2 = np.load("Client2Weight.npy", allow_pickle=True)
    weight_CNN_3 = np.load("Client3Weight.npy", allow_pickle=True)
    weight_CNN_4 = np.load("Client4Weight.npy", allow_pickle=True)
    weight_array = (weight_CNN_1 + weight_CNN_2 + weight_CNN_3 + weight_CNN_4) / 4
    weight_out = np.array(weight_array)
    return weight_out


# EKF Function
def EKF(cnn, weight_in):
    cnn.model.set_weights(weight_in)
    return cnn


# Create Models:LeNet
cnn_sever = CNN()
cnn1 = CNN()
cnn2 = CNN()
cnn3 = CNN()
cnn4 = CNN()
# Prepare Client Data
data_sever = DataSource()
data1 = DataSource1()
data2 = DataSource2()
data3 = DataSource3()
data4 = DataSource4()

# Compile Client and Sever Model
cnn_sever.model.compile(optimizer='adam',
                        loss='sparse_categorical_crossentropy',
                        metrics=['accuracy'])
cnn1.model.compile(optimizer='adam',
                   loss='sparse_categorical_crossentropy',
                   metrics=['accuracy'])
cnn2.model.compile(optimizer='adam',
                   loss='sparse_categorical_crossentropy',
                   metrics=['accuracy'])
cnn3.model.compile(optimizer='adam',
                   loss='sparse_categorical_crossentropy',
                   metrics=['accuracy'])
cnn4.model.compile(optimizer='adam',
                   loss='sparse_categorical_crossentropy',
                   metrics=['accuracy'])
storage_acc = []
weight = cnn_sever.model.get_weights()
np.save("SeverWeight", weight)
# All Clint Train
for i in range(BATCH):
    # Client Model Update(Downloads From Sever)
    weight = np.load("SeverWeight.npy", allow_pickle=True)
    #    cnn1.model.set_weights(weight)
    #    cnn2.model.set_weights(weight)
    #    cnn3.model.set_weights(weight)
    #    cnn4.model.set_weights(weight)
    cnn1 = EKF(cnn1, weight)
    cnn2 = EKF(cnn2, weight)
    cnn3 = EKF(cnn3, weight)
    cnn4 = EKF(cnn4, weight)
    # Client Model Fit
    cnn1.model.fit(data1.train_images[i], data1.train_labels[i], epochs=3)
    cnn2.model.fit(data2.train_images[i], data2.train_labels[i], epochs=3)
    cnn3.model.fit(data3.train_images[i], data3.train_labels[i], epochs=3)
    cnn4.model.fit(data4.train_images[i], data4.train_labels[i], epochs=3)
    # FedAvg

    weight_CNN1 = np.array(cnn1.model.get_weights())
    weight_CNN2 = np.array(cnn2.model.get_weights())
    weight_CNN3 = np.array(cnn3.model.get_weights())
    weight_CNN4 = np.array(cnn4.model.get_weights())


    np.save("Client1Weight", weight_CNN1)
    np.save("Client2Weight", weight_CNN2)
    np.save("Client3Weight", weight_CNN3)
    np.save("Client4Weight", weight_CNN4)
    weight = FedAvg()
    # Uploads to Sever
    cnn_sever.model.set_weights(weight)
    np.save("SeverWeight", weight)
    test_loss, test_acc = cnn_sever.model.evaluate(data_sever.test_images[0:1000], data_sever.test_labels[0:1000])
    print("Sever: 轮次: %d,准确率: %.4f，共测试了%d张图片 " % (i + 1, test_acc, len(data_sever.test_labels)))
    storage_acc = np.append(storage_acc, test_acc)

# @app.route('/c1')
# def weight_CNN1():
#     return str(weight_CNN1)
#
# @app.route('/c2')
# def weight_CNN2():
#     return ('the CNN2 client\'s weight is : \n' , str(weight_CNN2))
#
# @app.route('/c3')
# def weight_CNN3():
#     return ('the CNN3 client\'s weight is : \n' , str(weight_CNN3))
#
# @app.route('/c4')
# def weight_CNN4():
#     return ('the CNN4 client\'s weight is : \n' , str(weight_CNN4))

# Show Acc
x = np.array(range(100))
plt.plot(x, storage_acc)
plt.savefig('./acc.png')

# @app.route('/')
# def index():
#     return render_template('index.html', weight = str(weight))

@app.route('/', methods=['GET', 'POST'])
# GET只是对服务器获取数据
# POST会对服务器产生改变，请求上传就一定需要POST

def download_file():
    Path = os.listdir('.') # 列出指定路径下的所有文件和文件夹
    #print(Path)

		# 用于限定文件后缀名进行展示
    entries = []
    for path in Path:
        if os.path.splitext(path)[1] == '.png' or os.path.splitext(path)[1] == '.npy':
            entries.append(path)
            #print(entries)
            
    return render_template('download.html',entries = entries)


@app.route('/downloads/<filename>', methods=['GET', 'POST'])
def downloaded_file(filename):
    #if request.method == 'GET':
        #filename = request.url # 此处'files'对应的是html中的name="file"
        #print(filename)
    flash('Download Successfully ! ! !', 'success')
    return send_from_directory('./',filename,as_attachment = True)
# 若没有as_attachment = True，则会新开一个标签页显示上传文件的内容；有此参数则会直接下载


if __name__ == '__main__':
    app.run(host = '0.0.0.0')
    # from livereload import Server  # 用于再次刷新本页面
    #
    # server = Server(app.wsgi_app)
    # server.watch('**/*.*')
    # server.serve()

2 download.html

{% extends "bootstrap/base.html" %}
{% block title %}DOWNLOAD PAGE{% endblock %}

{% block content %}
    <div class="container">
        {% with messages = get_flashed_messages(with_categories=true) %}
            {% if messages %}
                {% for category, message in messages %}
                    <div class="alert alert-{{ category }}">
                        <button type="button" class="close" data-dismiss="alert" aria-hidden="true">×</button>
                        {{ message }}
                    </div>
                {% endfor %}
            {% endif %}
        {% endwith %}
        <h2>You Can Download The Following Files</h2>
        <h5 style="color:red">(just click it ☝ ~)</h5>
        <ol>
        {% for entry in entries %}
        <li><a href="{{url_for('downloaded_file',filename = entry)}}">{{entry}}</a>
        {% endfor %}
        </ol>
    </div>
{% endblock %}

3 Dockerfile 和 requirement.txt

FROM federated:v1.2
COPY . . # 将当前文件夹的所有文件复制到容器中
WORKDIR . # 设置当前工作目录
RUN pip install -r requirement.txt
EXPOSE 5000
RUN /bin/bash -c 'echo init ok'
CMD ["python", "main.py"]

requirement.txt：
可用tensorflow==2.4.1的形式指定版本进行pip install

Flask
tensorflow
matplotlib
flask_bootstrap
numpy

4 deployment 和 service

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: federated-deployment
  name: federated-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: federated-deployment
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: federated-deployment
    spec:
      containers:
      - image: registry.cn-beijing.aliyuncs.com/hqc-k8s/federated:v1.3
        name: federated
        resources: {}
        ports:
        - containerPort: 5000
        imagePullPolicy: IfNotPresent
status: {}

service.yaml:

apiVersion: v1 # 注意此处不能和deployment一样为‘apps/v1’
kind: Service
metadata:
  name: federated-service
  labels:
    app: federated-service
spec:
  ports:
  - port: 80
    targetPort: 5000
    nodePort: 30000
    protocol: TCP
  selector:
    app: federated-service
  type: NodePort

5 结果

控制器和服务等全部正常
在这里插入图片描述

5.1 出错

但是发现只是一瞬间running，过一会就一直重启。ip+port的方式更是没法成功。
查看日志发现：

root@master:/home/hqc/自然基金项目/Federated# kubectl logs federated-deployment-68f5c7fb8d-2zqpf -n default
# federated-deployment-68f5c7fb8d-2zqpf 是容器名

在这里插入图片描述发现是内部程序的原因，不是集群的问题，程序无法访问网址下载所需数据集。
为啥本地docker run可以成功运行，创建deployment后不能访问呢？？
在这里插入图片描述
以为的原因：

  1. 看这报错好像是外网的问题，但实际上不是。
  2. 肯定是service.yaml文件出错了，修改后仍然不行。

5.2 解决

最后发现不知为啥master的IP地址变成了192.168.43.49，好离谱。
想设置静态IP，按照网上大家都行的方法尝试发现没法固定IP，遂作罢。这个工作以后再弄。
直接修改hosts文件为192.168.43.49。
vim /etc/hosts
重启节点，惊喜发现所有都running。
在这里插入图片描述
创建deployment和service之后要等一段时间运行结束，再进行ip+port方式访问。
在这里插入图片描述
成功！！！

ERROR_LESS
关注
专栏目录
python 微服务开发实例-python 微服务方案
weixin_37988176的博客
10-29 1755
介绍使用python做web开发面临的一个最大的问题就是性能，在解决C10K问题上显的有点吃力。有些异步框架Tornado、Twisted、Gevent 等就是为了解决性能问题。这些框架在性能上有些提升，但是也出现了各种古怪的问题难以解决。在python3.6中，官方的异步协程库asyncio正式成为标准。在保留便捷性的同时对性能有了很大的提升,已经出现许多的异步框架使用asyncio。使用较早的...
Python应用程序部署至K8s——循序渐进版
wmy2002718的博客
02-05 1424
（1）修改admin.conf等配置文件，并且应用配置，但是在其中也会报错。⚠注：在工作节点中，很可能一开始就不存在admin.conf文件，需要将主节点中的文件复制到工作节点上的对应位置；问题描述：盒子联网时地IP并不总是固定的，如果在配置文件没有同步更改IP的情况下，就会出现集群无法使用的情况。解决方法：将程序设置成能够常驻的，并且在创建容器时使用交互式，即可看到输出以及Pod运行正常。原因：python不能够常驻，只能运行一次，且是后台运行情况，则会出错。Ⅰ.将主节点（master）中的“/etc/
Python代码部署，用Kubernetes实现自动化
最新发布
涛哥聊Python
08-16 1036
更多Python学习内容：ipengtao.com在现代软件开发中，容器化和微服务架构逐渐成为主流，而Kubernetes则是管理容器化应用的事实标准。Kubernetes（简称K8s）提供了自动化部署、扩展和管理容器化应用的强大功能。本文将详细介绍如何使用Kubernetes来部署Python代码，包括Kubernetes的基本概念、部署流程，以及通过示例代码演示完整的部署过程。Kubernet...
使用python脚本部署k8s集群
qq_56776641的博客
12-14 968
# coding=UTF-8# ping主机# 1.本地创建ssh公钥# 向被控主机添加公钥的方法# 测试ssh连接的方法# 本地的密钥开始加入被控制主机# 2.配置防火墙和selinux的方法# 配置防火墙和selinux# 3.配置域名映射# 首次域名ssh连接的方法# 4.配置主机名（主机名即域名）# 5.配置yum源# 6.安装必要工具# 7.配置chrony服务器# 8.禁用swap分区，修改linux的内核参数，配置ipvs功能# 9.重启主机并判断可以ssh登陆# 10.安装Docker和k8
k8s部署python3项目,并使用gitlab-ci实现自动化发布
07-29 1290
代码】k8s部署python3项目。
k8s构建python应用(django)
qingdao666666的博客
08-02 874
#k8s构建python应用(django) ######k8s构建python应用(django) mkdir -p pydemo && cd pydemo cat >Dockerfile <<EOF FROM centos:6.8 RUN mkdir -p /data/soft && cd /data/soft RUN yum install -y vim net-tools zip wget curl python-setuptools
Docker+Kubernetes(k8s)微服务容器化实践1
08-08
【Docker+Kubernetes(k8s)微服务容器化实践】是关于现代软件开发和部署方式的探讨，其中重点介绍了Docker和Kubernetes在构建微服务架构中的作用。本实践分为多个章节，从基础概念到具体实践，全面解析了微服务架构的...
Python微服务开发.rar
07-26
Python微服务开发是一种现代软件开发方法，它基于面向服务的架构（SOA）理念，但更强调服务的小型化、独立部署和自主性。在这个范例程序中，我们将深入探讨如何利用Python语言来构建和管理微服务。 首先，Python...
微服务：在Kubernetes上部署微服务（演示）
02-05
在Kubernetes上部署微服务 此包含我的有关上教程的源代码。 它包含两个微服务：检测器和查看器。 检测器服务是一个Python / Flask应用程序，它使用预训练的 v3模型来检测图像中的例如笔记本电脑和椅子。 它返回到...
Docker+k8s的微服务实战课程
09-26
Docker+k8s的微服务实战课程 课程目录 一、初识微服务 1 微服务-导学 2 软件架构的进化 3 什么是微服务 4 画出微服务架构图 5 微服务架构的优势和不足 二、微服务带来的问题及解决方案分析 1 微服务架构带来的问题 ...
微服务系列文章之 Springboot应用在k8s集群中配置的使用
Coder_Boy_的博客
07-16 1184
比方说不同项目中yml的层级不同，可能配置项目的名称也不一样，就到只在k8s管理员那里，不太好通过统一的yaml模板部署项目，每次都需要跟研发人员去核对。2. 将环境变量的列表提供给k8s运维人员，由他按照列表，配置k8s部署的yml文件，通过spec下env增加对应环境变量设置，改为k8s环境下设置。其实还有一个办法，经测试，deploy的env的配置，会复写springboot中yml对应key的值。k8s会把env配置下的参数，输出到容器的环境变量中，服务直接读取就会生效。那其实就是环境变量。
部署Python3开发的REST服务到K8S
weixin_45357522的博客
08-20 583
部署Python3开发的REST服务到K8S1. 创建gunicorn.conf.py2. 创建Dockerfile3. 构建Docker镜像4. 导出镜像到K8S的NODE5. 部署到Kubenates上6. 测试服务 为了方便管理，我们需要把Python3开发的REST服务部署到K8S中，再通过ingress暴露对外访问。下面就一步步来完成部署： 1. 创建gunicorn.conf.py 添...
python操作k8s
wuxingge的博客
09-22 3695
pip安装kubernetes库 pip3 install kubernetes 连接k8s配置文件~/.kube/config 拷贝到 kubeconfig.yaml python连接k8s from kubernetes import client, config config.kube_config.load_kube_config(config_file="kubeconfig.yaml") class Kubernetes: def __init__(self): self.Co
python 简单操作k8s
weixin_42562106的博客
04-24 4642
需要注意的是，重启 Pod 操作将会杀死该 Pod 中的容器，并重新启动一个新的容器。因此，在执行重启操作之前，请确保你已经备份了相关数据，并且已经确认该操作不会对应用程序产生不良影响。方法创建 Kubernetes API 客户端。接着，指定要重启的 Pod 的命名空间和名称，并使用。中指定要重启的容器名称和重启次数，以确保重启操作能够正确执行。方法加载 Kubernetes 配置文件，然后使用。方法来重启该 Pod。
gitlab ci/cd+harbor+k8s实现一键部署（python项目）
tangsiqi130的博客
07-06 1127
gitlab ci/cd+harbor+k8s实现一键部署（python项目）
dockerfile构建一个（python+flask+html）镜像 + 上传到阿里云私有仓库 + 部署到k8s---全过程
error的博客
12-13 2550
0 第一次构建 # 创建项目文件夹 root@master:/home/hqc/docker_learning# mkdir plus-of-arrays # 创建模板文件夹用于存放html文件 root@master:/home/hqc/docker_learning/plus-of-arrays# mkdir templates # 编写html文件 root@master:/home/hqc/docker_learning/plus-of-arrays# vim plus.html <!D
使用Python构建微服务
ChatGPTer
12-30 1779
什么是基于微服务的架构？它是一种架构风格，将复杂的软件系统构建为一组松散耦合的服务，并通过预定义的标准(API) 相互通信。这种架构风格具有敏捷性、可扩展性以及适应不断变化的业务需求的能力等优点。
使用Python进行微服务架构的设计与实现
一键难忘的博客
03-29 2970
在当今软件开发领域中，微服务架构已经成为了一种流行的设计范式。它通过将应用程序拆分为一系列小型、自治的服务，每个服务都围绕着特定的业务功能进行构建，从而实现了更高的灵活性、可扩展性和可维护性。Python作为一种简单易用且功能强大的编程语言，能够很好地支持微服务架构的设计与实现。本文将介绍如何使用Python语言来设计和实现微服务架构，并通过案例代码进行说明。
python 微服务_python微服务方案
weixin_39609620的博客
12-13 380
介绍使用python做web开发面临的一个最大的问题就是性能，在解决C10K问题上显的有点吃力。有些异步框架Tornado、Twisted、Gevent 等就是为了解决性能问题。这些框架在性能上有些提升，但是也出现了各种古怪的问题难以解决。在python3.6中，官方的异步协程库asyncio正式成为标准。在保留便捷性的同时对性能有了很大的提升,已经出现许多的异步框架使用asyncio。使用较早的...
Python3.6(uwsgi+Flask)在k8s的Docker化部署实践
总结来说，这个Python3应用的K8s容器化部署方案通过uwsgi和Flask提供了高性能的后端服务，并利用Kubernetes的强大功能实现灵活的部署和扩展。通过合理的配置管理和容器化，开发者和运维团队可以更高效地协作，确保...
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值