Mask-RCNN配合Flask使用的时候出现错误，已解决！！！

MASK-RCNN用keras训练好模型，再在Flask初始化加载模型，这个过程没有问题，但是在requests post请求调用到模型执行model.predict()的时候就报错：

[2020-02-29 15:57:02,308] ERROR in app: Exception on /predict [POST]
Traceback (most recent call last):
  File "D:\Anaconda3\envs\SafeHat\lib\site-packages\flask\app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "D:\Anaconda3\envs\SafeHat\lib\site-packages\flask\app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "D:\Anaconda3\envs\SafeHat\lib\site-packages\flask\app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "D:\Anaconda3\envs\SafeHat\lib\site-packages\flask\_compat.py", line 39, in reraise
    raise value
  File "D:\Anaconda3\envs\SafeHat\lib\site-packages\flask\app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "D:\Anaconda3\envs\SafeHat\lib\site-packages\flask\app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "D:/PycharmProjects/安全帽目标检测/run_keras_server.py", line 85, in predict
    yhat = model.detect(im_array, verbose=0)[0]
  File "D:\PycharmProjects\安全帽目标检测\Mask_RCNN\mrcnn\model.py", line 2526, in detect
    self.keras_model.predict([molded_images, image_metas, anchors], verbose=0)
  File "D:\Anaconda3\envs\SafeHat\lib\site-packages\keras\engine\training.py", line 1456, in predict
    self._make_predict_function()
  File "D:\Anaconda3\envs\SafeHat\lib\site-packages\keras\engine\training.py", line 378, in _make_predict_function
    **kwargs)
  File "D:\Anaconda3\envs\SafeHat\lib\site-packages\keras\backend\tensorflow_backend.py", line 3009, in function
    **kwargs)
  File "D:\Anaconda3\envs\SafeHat\lib\site-packages\tensorflow\python\keras\backend.py", line 3479, in function
    return GraphExecutionFunction(inputs, outputs, updates=updates, **kwargs)
  File "D:\Anaconda3\envs\SafeHat\lib\site-packages\tensorflow\python\keras\backend.py", line 3142, in __init__
    with ops.control_dependencies([self.outputs[0]]):
  File "D:\Anaconda3\envs\SafeHat\lib\site-packages\tensorflow\python\framework\ops.py", line 5426, in control_dependencies
    return get_default_graph().control_dependencies(control_inputs)
  File "D:\Anaconda3\envs\SafeHat\lib\site-packages\tensorflow\python\framework\ops.py", line 4867, in control_dependencies
    c = self.as_graph_element(c)
  File "D:\Anaconda3\envs\SafeHat\lib\site-packages\tensorflow\python\framework\ops.py", line 3796, in as_graph_element
    return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
  File "D:\Anaconda3\envs\SafeHat\lib\site-packages\tensorflow\python\framework\ops.py", line 3875, in _as_graph_element_locked
    raise ValueError("Tensor %s is not an element of this graph." % obj)
ValueError: Tensor Tensor("mrcnn_detection/Reshape_1:0", shape=(1, 100, 6), dtype=float32) is not an element of this graph.

问题原因：

load model是在实例化我的调用mask rcnn的类的时候进行的，然而inference进行预测是在接收到request的时候才进行，显然不在一个进程里，所以抛出了错误。

Mask-RCNN，他们是在同一个进程里的。Mask-RCNN的图片不断的写入一个类成员变量里，这里利用了python多线程中互斥锁确保不会同时读写这个变量，然后就可以让model对当前的图片进行inference了，所以单独调用Mask-RCNN不会出错，部署模型就出错。

言而总之：只能在同一个进程，不然就出错

未成功的办法一：

网上一般说：

在Flask中加载模型后，先执行一次model.predict()操作，参数的大小和真实大小一致之后的调用就不会出问题了

def load_model():
     global model
    # 创建配置
    cfg = PredictionConfig()
    # 定义模型
    model = MaskRCNN(mode='inference', model_dir='./', config=cfg)
    # 加载模型权重
    model_path = 'mask_rcnn_SafetyHat_cfg_site.h5'
    model.load_weights(model_path, by_name=True)
    #加载模型后，先执行一次model.predict()操作
    demo_img = np.ones((1, 224,224,3))
    model.detect(demo_img)

但是我重新requests post请求调用加载好的模型时，又出了一个新Bug：

tensorflow.python.framework.errors_impl.InvalidArgumentError: Tensor input_image:0, specified in either feed_devices or fetch_devices was not found in the Graph

网上搜到的原因：

1、在于K.tensorflow_backend.set_session(session)这句，import keras.backend as K，并将原来的改为K.set_session(session)即可

然而我找了半天也找不到K.tensorflow_backend.set_session(session)这句，但是我记住了session

2、同时加载多个Keras Model发生异常

我从始至终只定义了一个模型，哪来的多个。

未成功的方法二、

尝试在加载模型后保存tensorflow进程，在使用时，调用全局graph

# Right after loading or constructing your model, save the TensorFlow graph:

graph = tf.get_default_graph()

# In the other thread (or perhaps in an asynchronous event handler), do:

global graph
with graph.as_default():
    (... do inference here ...)

出现Bug：

tensorflow.python.framework.errors_impl.FailedPreconditionError: 2 root error(s) found.
  (0) Failed precondition: Error while reading resource variable res2c_branch2a/kernel from Container: localhost. This could mean that the variable was uninitialized. Not found: Container localhost does not exist. (Could not find resource: localhost/res2c_branch2a/kernel)
  [[{{node res2c_branch2a/convolution/ReadVariableOp}}]]
  [[mrcnn_detection/map/while/GatherV2/_91]]
  (1) Failed precondition: Error while reading resource variable res2c_branch2a/kernel from Container: localhost. This could mean that the variable was uninitialized. Not found: Container localhost does not exist. (Could not find resource: localhost/res2c_branch2a/kernel)
  [[{{node res2c_branch2a/convolution/ReadVariableOp}}]]
0 successful operations.
0 derived errors ignored.

这个错误呢，也是TensorFlow和Flask结合使用时的常见错误，最后通过这个Bug找到了真正的解决办法

成功的解决办法

session和graph同时使用

添加代码：

# 程序开始时声明
from tensorflow.python.keras.backend import set_session
sess = tf.Session()
graph = tf.get_default_graph()

# 在model加载前添加set_session
set_session(sess)
model = models.load_model(…………)

# 每次使用有关TensorFlow的请求时
# in each request (i.e. in each thread):
global sess
global graph
with graph.as_default():
    set_session(sess)
    model.predict(...)

注意：这里Debug要设置False，flask 开启 debug 模式，启动服务时，dubug 模式会开启一个 tensorflow 的线程，导致调用 tensorflow 的时候，graph 产生了错位。也会报错

具体代码如下：

run_keras_sever.py

# USAGE
# Start the server:
#  python run_keras_server.py
# Submit a request via cURL:
#  curl -X POST -F image=@dog.jpg 'http://localhost:5000/predict'
# Submita a request via Python:
#  python simple_request.py

# import the necessary packages

from keras.preprocessing.image import img_to_array
from PIL import Image
import flask
import io
import json
from model import *
# 程序开始时声明
from tensorflow.python.keras.backend import set_session
sess = tf.Session()
graph = tf.get_default_graph()
# initialize our Flask application and the Keras model
app = flask.Flask(__name__)
model = None


def load_model():
    # load the pre-trained Keras model (here we are using a model
    # pre-trained on ImageNet and provided by Keras, but you can
    # substitute in your own networks just as easily)
    global model
    global sess
    global cfg
    # 创建配置
    set_session(sess)
    cfg = PredictionConfig()
    # 定义模型
    model = MaskRCNN(mode='inference', model_dir='./', config=cfg)
    # 加载模型权重
    model_path = 'mask_rcnn_SafetyHat_cfg_site.h5'
    model.load_weights(model_path, by_name=True)

    # print(r)


def prepare_image(image):
    # if the image mode is not RGB, convert it
    if image.mode != "RGB":
        image = image.convert("RGB")
    # resize the input image and preprocess it
    scaled_image = img_to_array(image)
    # 转换像素值（例如居中）

    # scaled_image = mold_image(scaled_image, cfg)
    image = expand_dims(scaled_image, 0)
    # image = np.expand_dims(scaled_image, axis=0)
    # return the processed image
    return image


@app.route("/predict", methods=["POST"])
def predict():
    # initialize the data dictionary that will be returned from the
    # view

    if flask.request.method == "GET":
        return 'GET ERROR'
    # ensure an image was properly uploaded to our endpoint
    if flask.request.method == "POST":
        if flask.request.files.get("image"):
            # read the image in PIL format
            image = flask.request.files["image"].read()
            image = Image.open(io.BytesIO(image))
            im_array = prepare_image(image)
            # preprocess the image and prepare it for classification
            # cv2.cvtColor(im_array[0],cv2.COLOR_RGB2BGR)
            # cv2.imwrite('w.jpg',im_array[0])
            # classify the input image and then initialize the list
            # of predictions to return to the client

            global sess
            global graph
            with graph.as_default():
                set_session(sess)
                yhat = model.detect(im_array, verbose=0)[0]
                # yhat = json.dumps(yhat)

                result = {
                    'rois': yhat['rois'].tolist(),
                    'class_ids': yhat['class_ids'].tolist(),
                    'scores': yhat['scores'].tolist(),
                    'masks': yhat['masks'].tolist(),
                          }
            # results = imagenet_utils.decode_predictions(preds)
            # data["predictions"] = []
            #
            # # loop over the results and add them to the list of
            # # returned predictions
            # for (imagenetID, label, prob) in results[0]:
            #     r = {"label": label, "probability": float(prob)}
            #     data["predictions"].append(r)

            # indicate that the request was a success
    # return the data dictionary as a JSON response
    return flask.jsonify(result)

# if this is the main thread of execution first load the model and
# then start the server
if __name__ == "__main__":
    print(("* Loading Keras model and Flask starting server..."
           "please wait until server has fully started"))
    load_model()
    app.run(host='0.0.0.0',port=80)

simple_request.py

# USAGE
# python simple_request.py

# import the necessary packages
import requests

import io
from matplotlib import pyplot
from matplotlib.patches import Rectangle
# initialize the Keras REST API endpoint URL along with the input
# image path
KERAS_REST_API_URL = "http://192.168.10.107:80/predict"
IMAGE_PATH = "request.jpg"


# load the input image and construct the payload for the request
image = open(IMAGE_PATH, "rb").read()
payload = {"image": image}

# submit the request
yhat = requests.post(KERAS_REST_API_URL, files=payload).json()

image = pyplot.imread(IMAGE_PATH)

pyplot.imshow(image)
pyplot.title('Predicted')
ax = pyplot.gca()
# 绘制每个绘图框
class_ids = yhat['class_ids']
index = 0
for box in yhat['rois']:
    # 获取坐标
    y1, x1, y2, x2 = box
    # 计算绘图框的宽度和高度
    width, height = x2 - x1, y2 - y1
    # 创建形状对象
    if class_ids[index] == 1:
        rect = Rectangle((x1, y1), width, height, fill=False, color='green')
    if class_ids[index] == 2:
        rect = Rectangle((x1, y1), width, height, fill=False, color='red')
    # 绘制绘图框
    ax.add_patch(rect)
    index += 1
# 显示绘制结果
pyplot.savefig('response.jpg')
pyplot.clf()

参考文章：
ValueError: Tensor Tensor(“mrcnn_detection/PyFunc:0”, dtype=float32) is not an element of this graph
os.listdir以及keras如何使用GPU的两个坑
在Flask使用TensorFlow的几个常见错误