cannot import name 'ReparseException' & tf模型参数提取

原创 2018年04月16日 19:45:06

Tensorboard ImportError: cannot import name ‘ReparseException’


环境:

  • python 3.5.4
  • Tensorflow==1.4.1
  • html5lib==1.0.1

错误信息:

ImportError: cannot import name 'ReparseException'

定位咋:

~/anaconda3/envs/tf3/lib/python3.5/site-packages/bleach/sanitizer.py

找到对应的文件,在

from html5lib.constants import (
    entities,
    ReparseException,
    namespaces,
    prefixes,
    tokenTypes,
)

中找到出问题的地方,将ReparseException改为_ReparseException即可解决问题。

tf模型参数提取


通过TensorFlow saver 保存的模型是有v1和v2两种格式,在tf.train.Saver中有所说明:

  • write_version: controls what format to use when saving checkpoints. It also affects certain filepath matching logic. The V2 format is the recommended choice: it is much more optimized than V1 in terms of memory required and latency incurred during restore. Regardless of this flag, the Saver is able to restore from both V2 and V1 checkpoints.

实际上通过对比,v1 500M的模型到了v2需要 1.5G。。

在实际的参数提取过程中,如何把模型中的variable变成可直接操作的numpy数组是一个比较麻烦的问题。搜索后大部分答案是先构建图,然后在运行的session中进行eval。(难道就不能直接通过模型拿到参数吗?)

针对v2的模型,可以使用tensorpack包进行快速的参数提取,得到npz的数据格式。

安装

tensorpack
首先需要有git,然后命令行输入:

 pip install -U git+https://github.com/ppwwyyxx/tensorpack.git   

脚本

在仓库中的./scripts/dump-model-params.py为模型参数提取的脚本文件。

使用示例为:

./scripts/dump-model-params.py --meta train_log/alexnet-dorefa/graphxxxx.meta train_log/alexnet-dorefa/model-xxx output.npz

源码

#!/usr/bin/env python
# -*- coding: UTF-8 -*-
# File: dump-model-params.py
# Author: Yuxin Wu <ppwwyyxx@gmail.com>

import numpy as np
import six
import argparse
import os
import tensorflow as tf

from tensorpack.tfutils import varmanip
from tensorpack.tfutils.common import get_op_tensor_name

if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description='Keep only TRAINABLE and MODEL variables in a checkpoint.')
    parser.add_argument('--meta', help='metagraph file', required=True)
    parser.add_argument(dest='input', help='input model file, has to be a TF checkpoint')
    parser.add_argument(dest='output', help='output model file, can be npz or TF checkpoint')
    args = parser.parse_args()

    # this script does not need GPU
    os.environ['CUDA_VISIBLE_DEVICES'] = ''

    tf.train.import_meta_graph(args.meta, clear_devices=True)

    # loading...
    if args.input.endswith('.npz'):
        dic = np.load(args.input)
    else:
        dic = varmanip.load_chkpt_vars(args.input)
    dic = {get_op_tensor_name(k)[1]: v for k, v in six.iteritems(dic)}

    # save variables that are GLOBAL, and either TRAINABLE or MODEL
    var_to_dump = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)
    var_to_dump.extend(tf.get_collection(tf.GraphKeys.MODEL_VARIABLES))
    assert len(set(var_to_dump)) == len(var_to_dump), "TRAINABLE and MODEL variables have duplication!"
    globvarname = [k.name for k in tf.global_variables()]
    var_to_dump = set([k.name for k in var_to_dump if k.name in globvarname])

    for name in var_to_dump:
        assert name in dic, "Variable {} not found in the model!".format(name)

    dic_to_dump = {k: v for k, v in six.iteritems(dic) if k in var_to_dump}
    varmanip.save_chkpt_vars(dic_to_dump, args.output)
版权声明:本文为博主原创文章,转载请标注出处。 https://blog.csdn.net/Yan_Joy/article/details/79964647

tensorboard ImportError: cannot import name 'ReparseException'

环境:Mac os、Anaconda3 通过conda create -n tf36 python=3.6新建tensorflow的python环境,通过pip install tensorflow...
  • wustfrank
  • wustfrank
  • 2018-01-05 12:29:51
  • 470

tensorboard安装错误史

tensorboard安装错误1尝试阶段 Dell 32位电脑,原有安装Anaconda,conda create -n tensorflow python=3.6activate tensorflo...
  • mnwlh_
  • mnwlh_
  • 2018-03-26 14:59:08
  • 13

网络爬虫在Python 3.5下出现“ cannot import name 'HTMLParseError'”错误解决办法

升级了Python3.5之后,我使用BeautifulSoup4时候出现了ImportError: cannot import name 'HTMLParseError'的错误。在网上搜索好久资料之后...
  • yongjiutongmi53151
  • yongjiutongmi53151
  • 2017-03-13 10:11:01
  • 1910

TensorFlow - ImportError: cannot import name pywrap_tensorflow

>>> import tensorflow as tf Traceback (most recent call last):   File "", line 1, in   File "ten...
  • hanhuili
  • hanhuili
  • 2016-07-31 10:02:37
  • 9966

解决bs4在Python 3.5下出现“ImportError: cannot import name 'HTMLParseError'”错误

升级了Python3.5之后,我使用BeautifulSoup4时候出现了ImportError: cannot import name 'HTMLParseError'的错误。在网上搜索好久资料之后...
  • sinat_26599509
  • sinat_26599509
  • 2016-01-29 23:39:43
  • 11489

ImportError: cannot import name Pseudo

在scrapy中使用PyQuery出现如下错误: from lxml.cssselect import Pseudo, XPathExpr, XPathExprOr, FunImport Error:...
  • ly1046906153
  • ly1046906153
  • 2015-03-31 17:03:40
  • 2078

ImportError: cannot import name ABCMeta 问题解决

今天测试python代码,不知怎么都启动不了测试,显示“ImportError: cannot import name ABCMeta” 然后我试着只用一句“print ‘abc’”结果还是错,于是...
  • LaineGates
  • LaineGates
  • 2013-10-05 15:41:20
  • 14040

ImportError: cannot import name mkdtemp

问题: 以前是没问题的 今天突然python导入cv2和caffe却开始报错,如题。 解决办法:pip install -U numpy更新numpy即可...
  • hyqsong
  • hyqsong
  • 2016-06-28 14:28:51
  • 2015

ImportError: cannot import name __check_build

问题描述—–这个错误,不是我代码的问题,因为该下载的包我也下载并安装了,使用的pip安装的,路径也应该不会有问题,而且这个代码我直接Run的话就是无误的,但是如果在python console中一行一...
  • a787264137
  • a787264137
  • 2017-09-04 09:37:06
  • 1015
收藏助手
不良信息举报
您举报文章:cannot import name 'ReparseException' & tf模型参数提取
举报原因:
原因补充:

(最多只允许输入30个字)