黑得五彩斑斓：基于PaddleHub实现系列漫画生成器

0.题解

世界上有一种黑，虽然颜色单调，但却映射出五彩斑斓的世界。

那就是——

漫画书！

这次我们就来自己做个用视频生成系列漫画的小玩意。

经过实验，使用stylepro_artistic可以得到比较接近的效果，过程是：先提取视频中发生场景切换的帧（用统计直方图的差异来判断），再使用小人书的图片作为风格集，用stylepro_artistic将视频关键帧转换为黑白漫画风格。

不废话，上代码——

1.准备阶段

1.1. 安装paddleHub和预训练模型

# 安装 paddlehub 和 stylepro_artistic
!pip install paddlehub
!hub install stylepro_artistic==1.0.1

1.2. 准备给预训练模型用的风格图片

风格数据使用“漫画风格数据集-人美-水浒传”

地址是： https://aistudio.baidu.com/aistudio/datasetdetail/93870

其中 index.zip是压缩包，解开后会有26个文件夹，每个文件夹里是百余张png格式文件，2k * 1.5k 的分辨率（尺寸根据每本书稍有差别）。

index.txt里面是文件夹对应的分册书名，用处不大。

# 首先解压：
!rm -dfr /home/aistudio/work/style/93870/;mkdir /home/aistudio/work/style/;mkdir /home/aistudio/work/style/93870/;cd /home/aistudio/work/style/93870/ &&unzip /home/aistudio/data/data93870/index.zip

数据文件解压后，会给每本书建立一个文件夹。

#显示一张看看
import cv2
import matplotlib.pyplot as plt
#用open cv读出来，注意要调整3原色的顺序
t= cv2.cvtColor(cv2.imread('/home/aistudio/work/style/93870/05/0013.png'),cv2.COLOR_BGR2RGB )
plt.imshow( t )
plt.axis('off') #不显示坐标轴
plt.show()

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-IeRaO7YL-1641602244866)(output_6_0.png)]

由于原图带文字和边框，所以用作风格训练前要只抠出图画部分。

import cv2
def load_comic_photo(src=''):
    img = cv2.imread(src)
    (h,w,_) = img.shape
    #img = img[56:912, 386:1453] #y0,y1  x0,x1
    img = img[int(h*0.06):int(h*0.95), int(w*0.25):int(w*0.95)] 
    return img

#显示一张看看
import cv2
import matplotlib.pyplot as plt
t=load_comic_photo('/home/aistudio/work/style/93870/05/0013.png')
t = cv2.cvtColor(t,cv2.COLOR_BGR2RGB)
plt.imshow( t , cmap='gray')
plt.axis('off')
plt.show()

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-2aSa05eG-1641602244866)(output_9_0.png)]

1.3. 区分关键帧

基本思路是如果一帧图像与它上一帧发生很大变化，就可以认为切换了场景。那么就需要一个算法来计算2个图片的差异大小，这里简单的统计B通道明暗统计分布。

#定义函数计算单通道的直方图的相似值，越小差异越大
def calculate_delta(image1=False, image2=False):
    if type(image1)!=type(image2) :
        degree = [1e-10]
    else:
        # 灰度直方图算法
        # 计算单通道的直方图的相似值
        hist1 = cv2.calcHist([image1], [0], None, [256], [0.0, 255.0])
        hist2 = cv2.calcHist([image2], [0], None, [256], [0.0, 255.0])
        # 计算直方图的重合度
        degree = 0
        for i in range(len(hist1)):
            if hist1[i] != hist2[i]:
                degree = degree + \
                    (1 - abs(hist1[i] - hist2[i]) / max(hist1[i], hist2[i]))
            else:
                degree = degree + 1
        degree = degree / len(hist1)
        while type(degree)==type([]):
            degree=degree[0]
        degree=[degree]
    return degree

#找2张图试试
img1 = load_comic_photo('/home/aistudio/work/style/93870/06/0012.png')
plt.imshow(  cv2.cvtColor( img1 ,cv2.COLOR_BGR2RGB) )
plt.axis('off')
plt.show()

img2 = load_comic_photo('/home/aistudio/work/style/93870/05/0013.png')
plt.imshow(  cv2.cvtColor( img2 ,cv2.COLOR_BGR2RGB) )
plt.axis('off')
plt.show()

print( calculate_delta(img1, img2 )  )

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-e6OpnMEH-1641602244867)(output_11_0.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-5KecAbN8-1641602244867)(output_11_1.png)]

[array([0.7653695], dtype=float32)]

1.4. 预训练模型出场

stylepro_artistic 很容易使用：用 hub.Module(name=“stylepro_artistic”) 就可以加载。

加载后可以调用.style_transfer方法，参数images是元素为dict的list，dict必需的参数有2个：

【1】 ‘content’: 用cv读出来的准备转换的图片；

【2】 ‘styles’: 用cv读出来的用于训练风格的图片list。

import paddlehub as hub

#图片转为漫画
stylepro_artistic = hub.Module(name="stylepro_artistic") #加载预训练模型

def get_cartoon_img(src_img_data):
    result = stylepro_artistic.style_transfer(
        images=[{
            'content': src_img_data ,
            'styles': [load_comic_photo('/home/aistudio/work/style/93870/05/0013.png')
                ,load_comic_photo('/home/aistudio/work/style/93870/07/0015.png')
            ]
        }]
        #,use_gpu=True
        )
    t = result[0]['data']
    t = cv2.cvtColor( t,cv2.COLOR_RGB2GRAY) # 先要转换为灰度图片
    #ret, t = cv2.threshold(t, 140, 255,cv2.THRESH_BINARY)
    t = cv2.medianBlur(t,3)
    return t

[2021-12-19 01:50:10,530] [ WARNING] - The _initialize method in HubModule will soon be deprecated, you can use the __init__() to handle the initialization of the object

1.5. 准备处理这个视频：

https://vfx.mtime.cn/Video/2020/11/23/mp4/201123101722987047.mp4

# 下载：
!rm -dfr /home/aistudio/work/mov/;mkdir /home/aistudio/work/mov/;cd /home/aistudio/work/mov/ &&wget https://vfx.mtime.cn/Video/2020/11/23/mp4/201123101722987047.mp4

--2021-12-19 00:51:40--  https://vfx.mtime.cn/Video/2020/11/23/mp4/201123101722987047.mp4
Resolving vfx.mtime.cn (vfx.mtime.cn)... 222.85.26.239
Connecting to vfx.mtime.cn (vfx.mtime.cn)|222.85.26.239|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3021817 (2.9M) [video/mp4]
Saving to: ‘201123101722987047.mp4’

201123101722987047. 100%[===================>]   2.88M  11.0MB/s    in 0.3s    

2021-12-19 00:51:41 (11.0 MB/s) - ‘201123101722987047.mp4’ saved [3021817/3021817]

1.6.干活的函数

具体操作时，先把视频拆成连续的图片序列，利用前面的calculate_delta计算相邻两帧的差异，如果差异很大且间隔时间不是过分的短，那么把切换场景后的第一帧漫画化。

get_video_key_frame的定义包括4个参数

【1】video_src：视频文件路径；

【2】frame_path ：处理后漫画化的关键帧存放的文件夹；

【3】show_key_frame：是否显示处理前的关键帧，默认不显示；

【4】show_comic_key_frame：是否显示处理后的关键帧，默认不显示。

## 将视频转换为图片,抽取关键帧
import os
import cv2

#显示图片
def showimg(img, isgray=False):
  plt.axis("off")
  if isgray == True:
    plt.imshow(img, cmap='gray')
  else: 
    plt.imshow(img)
  plt.show()

def get_video_key_frame(video_src,frame_path='',show_key_frame=False,show_comic_key_frame=False):
    if frame_path and not os.path.exists(frame_path):
        os.mkdir(frame_path)
    vc = cv2.VideoCapture(video_src)
    i_imp =-1000 #上一次保存的帧号
    (frame_last,delta_last) =(False,0)#上一帧图、上一帧的差异指数
    if vc.isOpened():
        rval, frame = vc.read()
    else:
        rval = False

    i = 0 #当前帧号
    while rval:
        rval, frame = vc.read()
        if  rval:
            #cv2.imwrite(frame_path+'{}.jpg'.format(i), frame)
            delta = calculate_delta(frame_last, frame )[0] # 计算差异程度
            pct = abs(delta - delta_last)/(delta+0.000001) #差异程度的变化量
            #print( (i,delta,pct)  )
            #if delta_last<0.9 and pct>0.5:
            if pct>0.2 and i-i_imp>10: #差异程度变化很大，且间隔半秒以上
                #print( (i,delta,pct)  )
                if show_key_frame:
                    showimg( cv2.cvtColor( frame ,cv2.COLOR_BGR2RGB) , isgray=True)
                if frame_path:
                    cv2.imwrite(frame_path+'{}.jpg'.format(i), frame)
                if show_comic_key_frame:
                    frame_cartoon = get_cartoon_img(frame)
                    showimg( frame_cartoon, isgray=True)
                    if frame_path:
                        cv2.imwrite(frame_path+'{}_c.jpg'.format(i), frame_cartoon)
                i_imp = i
            (frame_last,delta_last) = ( frame , delta )
            i += 1
    print('视频帧数：{}'.format(i))
    FPS = vc.get(cv2.CAP_PROP_FPS)
    SIZE = (int(vc.get(cv2.CAP_PROP_FRAME_WIDTH)), 
            int(vc.get(cv2.CAP_PROP_FRAME_HEIGHT)))
    print('FPS:{}, SIZE:{}'.format(FPS,SIZE))

2.正式开始

2.1. 先看一下抽取关键帧的成效：

# 可以试试了
video = 'work/mov/201123101722987047.mp4'
frame_path = 'work/mov/picture/'

# 抽取关键帧
get_video_key_frame(video,frame_path,show_key_frame=True,show_comic_key_frame=False)

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-up19Awts-1641602244867)(output_19_0.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-6Ps2qxsK-1641602244867)(output_19_1.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-w17AoOhN-1641602244868)(output_19_2.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-sPaQpUax-1641602244868)(output_19_3.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-HKxnRxby-1641602244868)(output_19_4.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-09zHnuXK-1641602244868)(output_19_5.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-DgDdL4vq-1641602244869)(output_19_6.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-1W6ZbkWl-1641602244869)(output_19_7.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-eF6qukjE-1641602244869)(output_19_8.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-fmhi5EGy-1641602244869)(output_19_9.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-G5Zm7Wyp-1641602244869)(output_19_10.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ygOV3mvd-1641602244870)(output_19_11.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-lbWvqOVL-1641602244870)(output_19_12.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-U2kP4UJr-1641602244870)(output_19_13.png)]

视频帧数：359
FPS:24.0, SIZE:(1280, 720)

2.2. 再看看转成漫画的效果：

# 关键帧转为漫画
get_video_key_frame(video,frame_path,show_key_frame=False,show_comic_key_frame=True)

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-BvYGg4fo-1641602244870)(output_21_0.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-olstMfEz-1641602244871)(output_21_1.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-0wU8vcK9-1641602244871)(output_21_2.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-1jYtbiSe-1641602244871)(output_21_3.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-QLj0Db5f-1641602244871)(output_21_4.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-fGbgiKYG-1641602244871)(output_21_5.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-HBYMgWG8-1641602244872)(output_21_6.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Yg9CyPK5-1641602244872)(output_21_7.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-qysNwsNd-1641602244872)(output_21_8.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-O17D5Fvc-1641602244872)(output_21_9.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-UwGhiX0K-1641602244872)(output_21_10.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-j94XHY9T-1641602244873)(output_21_11.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-HdBc5GLv-1641602244873)(output_21_12.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-um1QX1ff-1641602244873)(output_21_13.png)]

视频帧数：359
FPS:24.0, SIZE:(1280, 720)

3.写在下一版之前

可以看出，抽取的关键帧图片转成漫画风格虽然成功，但是还有一些问题，比如虽然线条转换后有毛笔起落的感觉，但是对大块颜色的处理，采用了用很多小笔触模拟阴影，显得画面混乱。据高手分析，可能是模型网络中存在对细节不大友好的核，导致诡影。另外，也怀疑由于喂给模型的风格数据量过小、次数却很多，有过拟合的可能。结合整个处理过程，总结有3个方向值得改进：

1 现有的关键帧抽取算法简单粗暴，可以改用深度学习，让选择出的帧更接近人类理解的“关键帧”。

2 对于转换效果，可以调整模型网络，增加风格训练数据，达到减少残影和冗余笔划的效果，让输出更接近“小人书”风格。

3 可以使用短视频预测标签的形式，生成图片对应的文字说明，输出完整的“小人书”。

============= 我是安静的华丽分割线，你看不见我，看不见我 ===========================

以下为各种实验，请忽略

!ls /home/aistudio/data
#https://aistudio.baidu.com/aistudio/projectdetail/3260649

data93870  data94144

!ls /home/aistudio/work

!mkdir /home/aistudio/external-libraries
!pip install beautifulsoup4 -t /home/aistudio/external-libraries
# -t /home/aistudio/external-libraries

!hub install stylepro_artistic==1.0.1

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/__init__.py:107: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import MutableMapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/rcsetup.py:20: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Iterable, Mapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/colors.py:53: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Sized
[2021-12-18 21:44:19,362] [    INFO] - Module stylepro_artistic-1.0.1 already installed in /home/aistudio/.paddlehub/modules/stylepro_artistic

import sys 
sys.path.append('/home/aistudio/external-libraries')

import paddlehub as hub
import cv2

stylepro_artistic = hub.Module(name="stylepro_artistic")

def load_comic_photo(src=''):
    #src = '/home/aistudio/work/s1/07/0011.png'
    img = cv2.imread(src)
    (h,w,_) = img.shape
    #img =  cv2.cvtColor(img,cv2.COLOR_RGB2GRAY)
    #img =  cv2.cvtColor(img,cv2.COLOR_GRAY2RGB)
    #img = img[56:912, 386:1453] #y0,y1  x0,x1
    img = img[int(h*0.06):int(h*0.95), int(w*0.25):int(w*0.95)] 
    return img

def get_caton_img(src_img_data):
    result = stylepro_artistic.style_transfer(
        images=[{
            'content': src_img_data ,
            'styles': [load_comic_photo('/home/aistudio/work/s1/05/0013.png')
                ,load_comic_photo('/home/aistudio/work/s1/07/0015.png')
            ]
        }])
    #print(result)
    #import matplotlib.pyplot as plt
    #from matplotlib.image import imread
    #plt.imshow(imread(picture))
    t = result[0]['data']
    t = cv2.cvtColor( t,cv2.COLOR_RGB2GRAY) # 先要转换为灰度图片
    #ret, t = cv2.threshold(t, 140, 255,cv2.THRESH_BINARY) # 这里的第二个参数要调，是阈值！！
    t = cv2.medianBlur(t,3)
    #plt.imshow( t , cmap='gray')
    #plt.axis('off')
    #plt.show()
    #cv2.imwrite('out.jpg',result[0]['data'])
    return t

#################
src = '/home/aistudio/work/pexels-photo-8055848.jpeg'
src = '/home/aistudio/output/0_1639814440.jpg'
#src = '/home/aistudio/work/8a02601e-24a7-428f-9fe4-12b5f0cc2909.jpg'
#src = '/home/aistudio/work/1f81fa79-b46e-4120-9bf4-91714a8a93f9.jpg'
#src = '/home/aistudio/work/work/1274696563185580257.jpg'
src = 'work/v1/pics/1.jpg'
src = 'work/v1/pics/130.jpg'
src = 'work/v1/pics/218.jpg'
src = 'work/v1/f4/1.jpg'
src = 'work/v1/f4/197.jpg'
img = cv2.imread(src)
#img =  cv2.cvtColor(img,cv2.COLOR_RGB2GRAY)
#img =  cv2.cvtColor(img,cv2.COLOR_GRAY2RGB)
result = stylepro_artistic.style_transfer(
    images=[{
        'content': img ,
        'styles': [load_comic_photo('/home/aistudio/work/s1/05/0013.png')
            ,load_comic_photo('/home/aistudio/work/s1/07/0015.png')
        ]
    }])

#print(result)
import matplotlib.pyplot as plt
from matplotlib.image import imread
#plt.imshow(imread(picture))
t = result[0]['data']
t = cv2.cvtColor( t,cv2.COLOR_RGB2GRAY) # 先要转换为灰度图片
#ret, t = cv2.threshold(t, 140, 255,cv2.THRESH_BINARY) # 这里的第二个参数要调，是阈值！！
t = cv2.medianBlur(t,3)
plt.imshow( t , cmap='gray')
plt.axis('off')
plt.show()
cv2.imwrite('out.jpg',result[0]['data'])

[2021-12-18 23:18:16,073] [ WARNING] - The _initialize method in HubModule will soon be deprecated, you can use the __init__() to handle the initialization of the object

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-EiIAkyOK-1641602244873)(output_29_1.png)]

True

#dir(stylepro_artistic)
stylepro_artistic.save_inference_model(dirname='work/v1/pics/m/', model_filename=None, params_filename=None, combined=False)
#sa = hub.Module(directory='work/v1/pics/m/')
sa = hub.Module(name="stylepro_artistic")
sa.load( 'work/v1/pics/m/' )

[2021-12-18 22:10:53,644] [ WARNING] - The _initialize method in HubModule will soon be deprecated, you can use the __init__() to handle the initialization of the object



---------------------------------------------------------------------------

ModuleNotFoundError                       Traceback (most recent call last)

/tmp/ipykernel_100/3943064399.py in <module>
      3 #sa = hub.Module(directory='work/v1/pics/m/')
      4 sa = hub.Module(name="stylepro_artistic")
----> 5 sa.load( 'work/v1/pics/m/' )


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddlehub/compat/paddle_utils.py in runner(*args, **kwargs)
    218     def runner(*args, **kwargs):
    219         with static_mode_guard():
--> 220             return func(*args, **kwargs)
    221 
    222     return runner


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddlehub/module/module.py in load(cls, directory)
    193         basename = os.path.split(directory)[-1]
    194         dirname = os.path.join(*list(os.path.split(directory)[:-1]))
--> 195         py_module = utils.load_py_module(dirname, '{}.module'.format(basename))
    196 
    197         for _item, _cls in inspect.getmembers(py_module, inspect.isclass):


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddlehub/utils/utils.py in load_py_module(python_path, py_module_name)
    247         sys.modules.pop(py_module_name)
    248 
--> 249     py_module = importlib.import_module(py_module_name)
    250     sys.path.pop(0)
    251 


/opt/conda/envs/python35-paddle120-env/lib/python3.7/importlib/__init__.py in import_module(name, package)
    125                 break
    126             level += 1
--> 127     return _bootstrap._gcd_import(name[level:], package, level)
    128 
    129 


/opt/conda/envs/python35-paddle120-env/lib/python3.7/importlib/_bootstrap.py in _gcd_import(name, package, level)


/opt/conda/envs/python35-paddle120-env/lib/python3.7/importlib/_bootstrap.py in _find_and_load(name, import_)


/opt/conda/envs/python35-paddle120-env/lib/python3.7/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)


ModuleNotFoundError: No module named 'm.module'

import paddlehub as hub
import cv2

def load_comic_photo(src=''):
    #src = '/home/aistudio/work/s1/07/0011.png'
    img = cv2.imread(src)
    (h,w,_) = img.shape
    #img =  cv2.cvtColor(img,cv2.COLOR_RGB2GRAY)
    #img =  cv2.cvtColor(img,cv2.COLOR_GRAY2RGB)
    #img = img[56:912, 386:1453] #y0,y1  x0,x1
    img = img[int(h*0.06):int(h*0.95), int(w*0.25):int(w*0.95)] 
    return img

import matplotlib.pyplot as plt
t=load_comic_photo('/home/aistudio/work/s1/05/0013.png')
t = cv2.cvtColor(t,cv2.COLOR_RGB2GRAY) # 先要转换为灰度图片
ret, t = cv2.threshold(t, 140, 205,cv2.THRESH_BINARY) # 这里的第二个参数要调，是阈值！！
plt.imshow( t , cmap='gray')
plt.axis('off')
plt.show()

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-jhguNJol-1641602244873)(output_31_0.png)]

#print(result)
import matplotlib.pyplot as plt
from matplotlib.image import imread
#plt.imshow(imread(picture))
#.convert('L')
t = result[0]['data']
#ret, t = cv2.threshold(t, 160, 255,cv2.THRESH_BINARY) # 这里的第二个参数要调，是阈值！！
t = cv2.medianBlur(t,3)
t = cv2.cvtColor(t,cv2.COLOR_RGB2GRAY) # 先要转换为灰度图片
plt.imshow( t , cmap='gray')
#plt.axis('off')
plt.show()
# 1 只转灰度，之前去噪（不去也可） 2 二值化，略高些才能留下轮廓，但是需要去噪 3，先2值化，会有杂色，转成灰度去噪

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-2uBX1ygL-1641602244874)(output_32_0.png)]

from PIL import Image
import matplotlib.pyplot as plt
import numpy as np

#获取图片
def getimg():
  return Image.open('/home/aistudio/work/pexels-photo-8055848.jpeg')
  
#显示图片
def showimg(img, isgray=False):
  plt.axis("off")
  if isgray == True:
    plt.imshow(img, cmap='gray')
  else: 
    plt.imshow(img)
  plt.show()

#showimg(getimg(), True)


im = getimg()
im_gray = im.convert('L')
showimg(im_gray, True)

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-R9ZdrpQh-1641602244874)(output_33_0.png)]

from cv2 import cv2
import os

def resize(image, width=None, height=None, inter=cv2.INTER_AREA):
	dim = None
	(h, w) = image.shape[:2]
	if width is None and height is None:
		return image
	if width is None:
		r = height / float(h)
		dim = (int(w * r), height)
	else:
		r = width / float(w)
		dim = (width, int(h * r))
	resized = cv2.resize(image, dim, interpolation=inter)
	return resized

def rgb_to_sketch(pic_path):
	img_rgb = cv2.imread(pic_path)
	img_gray = cv2.cvtColor(img_rgb, cv2.COLOR_BGR2GRAY)##转为灰度图
	##可改变图片的尺寸，只设置h或者只设置w就会按照等比例放大或缩小；若都指定，则按照指定的大小变换
	img_gray=resize(img_gray, height = 500)
	img_blur = cv2.GaussianBlur(img_gray, ksize=(21, 21), sigmaX=0, sigmaY=0)##高斯滤波
	##可改变scale的值来改变生成效果，scale越小，会让一些原本比较亮的区域变得更加清晰
	img_blend = cv2.divide(img_gray, img_blur, scale=225)##图像相除实现素描效果
	return img_blend

f= r'work/1274696563185580257.jpg'#r'work/1f81fa79-b46e-4120-9bf4-91714a8a93f9.jpg'#r'work/8a02601e-24a7-428f-9fe4-12b5f0cc2909.jpg'#r'work/pexels-photo-8055848.jpeg'
t = rgb_to_sketch(f)
ret, t = cv2.threshold(t, int(t.mean()), 255,cv2.THRESH_BINARY)
t = cv2.medianBlur(t,3)
cv2.imwrite(f+r'.output.jpg', t)

True

t.mean()

198.83404

import paddlehub as hub
import cv2

m7 = hub.Module(name='animegan_v1_hayao_60')

# 模型预测
result = m7.style_transfer(images=[cv2.imread('work/pexels-photo-8055848.jpeg')],visualization=True)

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/__init__.py:107: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import MutableMapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/rcsetup.py:20: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Iterable, Mapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/colors.py:53: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Sized
[2021-12-18 15:59:20,648] [ WARNING] - The _initialize method in HubModule will soon be deprecated, you can use the __init__() to handle the initialization of the object
W1218 15:59:20.654083  9756 analysis_predictor.cc:1353] Deprecated. Please use CreatePredictor instead.

!rm -f work/v1/pics/*.*
!rm -f work/v2/08/*.*
## 将视频转换为图片
import os
import cv2

# 原视频读取
video = 'work/v1/201123101722987047.mp4'#'work/v1/v.f42905.mp4'#'work/v1/211217084858556132.mp4'#'work/v1/d1.mp4'#'work/v1/m2.mp4'#'work/v1/f5.mp4'#'work/v1/olds.mp4'#'work/v2/MTV_08.mp4'#
frame_path = 'work/v1/201123101722987047/'#'work/v1/v.f42905/'#'work/v1/211217084858556132/'#'work/v1/d1/'#'work/v1/m2/'#'work/v1/f5/'#'work/v1/pics/' # 'work/v2/08/' #
dog = 1
if not os.path.exists(frame_path):
    os.mkdir(frame_path)
vc = cv2.VideoCapture(video)

i_imp =-1000
(frame_last,delta_last) =(False,0)
if vc.isOpened():
    rval, frame = vc.read()
else:
    rval = False

i = 0
while rval:
    rval, frame = vc.read()
    if  rval:
        #cv2.imwrite(frame_path+'{}.jpg'.format(i), frame)
        delta = calculate(frame_last, frame )[0]
        pct = abs(delta - delta_last)/(delta+0.000001)
        #print( (i,delta,pct)  )
        #if delta_last<0.9 and pct>0.5:
        if pct>0.2 and i-i_imp>10:
            print( (i,delta,pct)  )
            showimg(frame, isgray=True)
            cv2.imwrite(frame_path+'{}.jpg'.format(i), frame)
            if dog>0:
                frame_caton = get_caton_img(frame)
                showimg( frame_caton, isgray=True)
                cv2.imwrite(frame_path+'{}_c.jpg'.format(i), frame_caton)
            i_imp = i
        (frame_last,delta_last) = ( frame , delta )
        i += 1
print('视频帧数：{}'.format(i))
FPS = vc.get(cv2.CAP_PROP_FPS)
SIZE = (int(vc.get(cv2.CAP_PROP_FRAME_WIDTH)), 
        int(vc.get(cv2.CAP_PROP_FRAME_HEIGHT)))
print('FPS:{}, SIZE:{}'
e)
                cv2.imwrite(frame_path+'{}_c.jpg'.format(i), frame_caton)
            i_imp = i
        (frame_last,delta_last) = ( frame , delta )
        i += 1
print('视频帧数：{}'.format(i))
FPS = vc.get(cv2.CAP_PROP_FPS)
SIZE = (int(vc.get(cv2.CAP_PROP_FRAME_WIDTH)), 
        int(vc.get(cv2.CAP_PROP_FRAME_HEIGHT)))
print('FPS:{}, SIZE:{}'
    .format(FPS,SIZE))

(1, array([0.9369093], dtype=float32), array([0.9999989], dtype=float32))

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-7EuznbhE-1641602244874)(output_37_1.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-pnynssML-1641602244874)(output_37_2.png)]

(23, array([0.16196954], dtype=float32), array([4.8128796], dtype=float32))

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Fuy9mLcQ-1641602244874)(output_37_4.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-FGAfVKI8-1641602244875)(output_37_5.png)]

(58, array([0.24561134], dtype=float32), array([1.8473916], dtype=float32))

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-pwdXz0vk-1641602244875)(output_37_7.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Xx3dxiIE-1641602244875)(output_37_8.png)]

(90, array([0.48886403], dtype=float32), array([0.8467305], dtype=float32))

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Kj8oM5ei-1641602244875)(output_37_10.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-8XP4KU1B-1641602244875)(output_37_11.png)]

(122, array([0.32180133], dtype=float32), array([1.6550589], dtype=float32))

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-0aSQZw3p-1641602244876)(output_37_13.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-S8YLwb7Q-1641602244876)(output_37_14.png)]

(143, array([0.3552464], dtype=float32), array([1.4872069], dtype=float32))

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-28CTZjSP-1641602244876)(output_37_16.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ahc4VZiH-1641602244876)(output_37_17.png)]

(173, array([0.32388538], dtype=float32), array([1.9440843], dtype=float32))

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-U0RXkMYy-1641602244876)(output_37_19.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-IHEXKy9d-1641602244877)(output_37_20.png)]

(195, array([0.3770919], dtype=float32), array([1.3326628], dtype=float32))

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-fzI5iZDO-1641602244877)(output_37_22.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-8qEFKv8l-1641602244877)(output_37_23.png)]

(218, array([0.48482153], dtype=float32), array([0.7061973], dtype=float32))

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-P6xNRUgy-1641602244877)(output_37_25.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-nJKbuzVc-1641602244877)(output_37_26.png)]

(234, array([0.34026635], dtype=float32), array([1.32831], dtype=float32))

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-5BAEFchw-1641602244877)(output_37_28.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-buT6rjlI-1641602244878)(output_37_29.png)]

(245, array([0.8021379], dtype=float32), array([0.79289573], dtype=float32))

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-szUZoMf5-1641602244878)(output_37_31.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-C5ox4rpm-1641602244878)(output_37_32.png)]

(259, array([0.3227935], dtype=float32), array([1.5863717], dtype=float32))

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-L7T2Y5t5-1641602244878)(output_37_34.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-HYwVjnhe-1641602244878)(output_37_35.png)]

(318, array([0.50448614], dtype=float32), array([0.8998087], dtype=float32))

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-WDlyLTk5-1641602244879)(output_37_37.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ALX0bCG1-1641602244879)(output_37_38.png)]

(346, array([0.41244707], dtype=float32), array([1.2286121], dtype=float32))

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ibi3x4Za-1641602244879)(output_37_40.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-9Xmdqw2H-1641602244879)(output_37_41.png)]

视频帧数：359
FPS:24.0, SIZE:(1280, 720)

from tqdm import tqdm

#import paddlevideo as pv

import paddlehub as hub

videotag = hub.Module(name="videotag_tsn_lstm")

# execute predict and print the result
results = videotag.classify(paths=["work/mov/201123101722987047.mp4"], use_gpu=False)
print(results)