当我给pytorch的dataloader设置num_workers=4进行并行时,出现的warning。
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)
The current process just got forked. Disabling parallelism to avoid deadlocks...
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)
The current process just got forked. Disabling parallelism to avoid deadlocks...
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)
The current process just got forked. Disabling parallelism to avoid deadlocks...
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)
解决方法warning已经告诉我们了,为了避免死锁,需要我们手动显式设置该变量TOKENIZERS_PARALLELISM是否支持tokens并行。即在程序前面加上以下代码行就可以。
import os
os.environ["TOKENIZERS_PARALLELISM"] = "false"
或
import os
os.environ["TOKENIZERS_PARALLELISM"] = "true"