tensorflow 2.2 tensorboard error

最新推荐文章于 2023-04-09 23:21:15 发布

SoraCaptain

最新推荐文章于 2023-04-09 23:21:15 发布

阅读量2.2k

点赞数 2

分类专栏： debug 文章标签： tensorflow gpu

本文链接：https://blog.csdn.net/lzycnan/article/details/109772122

版权

debug 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

使用TF2.2运行tensorboard callback，希望存些profile进行分析，报以下错误：

Error during training

E tensorflow /core/profiler/internal/ gpu / cupti_tracer .cc:1408] function cupti_interface _->Subscribe( &subscriber_, ( CUpti_CallbackFunc ) ApiCallback , this)failed with error CUPTI_ERROR_NOT_INITIALIZED

function cupti_interface _-> ActivityRegisterCallbacks ( AllocCuptiActivityBuffer , FreeCuptiActivityBuffer )failed with error CUPTI_ERROR_NOT_INITIALIZED

function cupti_interface _-> EnableCallback ( 0 , subscriber_, CUPTI_CB_DOMAIN_DRIVER_API, cbid )failed with error CUPTI_ERROR_INVALID_PARAMETER

function cupti_interface _->Subscribe( &subscriber_, ( CUpti_CallbackFunc ) ApiCallback , this)failed with error CUPTI_ERROR_INSUFFICIENT_PRIVILEGES

function cupti_interface _-> ActivityRegisterCallbacks ( AllocCuptiActivityBuffer , FreeCuptiActivityBuffer )failed with error CUPTI_ERROR_INSUFFICIENT_PRIVILEGES

function cupti_interface _-> EnableCallback ( 0 , subscriber_, CUPTI_CB_DOMAIN_DRIVER_API, cbid )failed with error CUPTI_ERROR_INVALID_PARAMETER

tensorboard 打开没有kernel的信息

运行环境：

Linux Ubuntu 16.04

TensorFlow 2.2.0 (使用conda install 安装）

python 3.6.12

CUDA 10.1

cudnn 7.6.5

查到一些帖子

https://github.com/tensorflow/tensorflow/issues/39143

https://github.com/tensorflow/tensorflow/issues/35860

https://github.com/tensorflow/tensorboard/issues/3256

https://forums.developer.nvidia.com/t/nvprof-warning-the-user-does-not-have-permission-to-profile-on-the-target-device/72374

https://unix.stackexchange.com/questions/539838/unable-to-resolve-nvidia-nvprof-err-nvgpuctrperm-with-nvreg-restrictprofilingt

简单来说原因是nvidia driver 限制GPU performance的读取，以防信息被窃取. 不开放权限给 ‘user’

以下两种方法较为方便 (第一种对我没有效果，第二种可以）

1. adding options nvidia “ NVreg_RestrictProfilingToAdminUsers =0” to / etc / modprobe.d / nvidia -kernel- common.conf , and reboot should resolve the permission issue (doesn’t work)

nvidia -kernel- common.conf is a readonly file

2. run training in sudo mode

sudo /opt/anaconda3/ envs /TF2_2/bin/python ( work )