Colaboratory 是一个 Google 研究项目,旨在帮助传播机器学习培训和研究成果。它是一个 Jupyter 笔记本环境,不需要进行任何设置就可以使用,并且完全在云端运行。Colaboratory 笔记本存储在 Google 云端硬盘中,并且可以共享,就如同您使用 Google 文档或表格一样。Colaboratory 可免费使用。其实就是薅谷歌的羊毛。
本文主要介绍如何通过Colaboratory连接Google云端硬盘,并使用免费的Tesla K80 GPU执行Google driver中的机器学习程序。
Colaboratory介绍
首先,访问 CoLaboratory 网站,注册后接受使用该工具的邀请。注册成功后可以打开Colaboratory 简介,会出现一个「Hello, Colaboratory」文件,包含一些基本示例。
注意右上角的连接按钮,如果没有连接是红色的。Colaboratory现在已全面支持python2和python3两个版本,可以在 代码执行程序→更改运行时类型 选择python版本,并在 硬件加速器 中选择是否使用GPU加速。
连接Google driver
先注册Google云端硬盘,注册后便会有免费的15G空间,日常跑个学习的程序,无论是保存程序还是数据集都足够用了,然后通过Colaboratory连接云端硬盘。
这里以Cifar-10 GPU版本程序为例:
首先连接到Google硬盘,因为我的程序都上传在Google硬盘中,新建一个一个python记事本,将下列代码复制到记事本中。
这段代码执行后会安装很多东西
# Install a Drive FUSE wrapper.
# https://github.com/astrada/google-drive-ocamlfuse
!apt-get install -y -qq software-properties-common python-software-properties module-init-tools
!add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null
!apt-get update -qq 2>&1 > /dev/null
!apt-get -y install -qq google-drive-ocamlfuse fuse
执行下面代码,打开链接,将链接中授权的代码复制到框格中回车执行
# Generate auth tokens for Colab
from google.colab import auth
auth.authenticate_user()
# Generate creds for the Drive FUSE library.
from oauth2client.client import GoogleCredentials
creds = GoogleCredentials.get_application_default()
import getpass
!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL
vcode = getpass.getpass()
!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}
最后一步,挂载Google driver
# Create a directory and mount Google Drive using that directory.
!mkdir -p drive
!google-drive-ocamlfuse drive
print('Files in Drive:')
!ls drive/
# Create a file in Drive.
#!echo "This newly created file will appear in your Drive file list." > drive/created.txt
执行 !ls 查看虚拟机中的文件,如果有一个Driver文件夹,便是成功了
完整连接程序:
https://colab.research.google.com/drive/1AxtTSlL6CiVzkdTMoy0-4BU_bUhmcIT8
Google官方导入数据示例:
https://colab.research.google.com/notebook#fileId=/v2/external/notebooks/io.ipynb&scrollTo=eikfzi8ZT_rW
官方导入数据示例中有好几种方法,喜欢别的方法也建议去尝试。
执行driver中的程序
修改程序中数据的导入路径,执行程序:
!python3 drive/Cifar10/cifar10_multi_gpu_train.py
执行效果:
/usr/local/lib/python3.6/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
Filling queue with 20000 CIFAR images before starting to train. This will take a few minutes.
2018-03-05 04:39:34.103914: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-03-05 04:39:34.104527: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1208] Found device 0 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:00:04.0
totalMemory: 11.17GiB freeMemory: 11.11GiB
2018-03-05 04:39:34.104577: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1308] Adding visible gpu devices: 0
2018-03-05 04:39:34.381677: I tensorflow/core/common_runtime/gpu/gpu_device.cc:989] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10774 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
2018-03-05 04:39:53.097102: step 0, loss = 11.24 (7.1 examples/sec; 18.121 sec/batch)
2018-03-05 04:39:54.442389: step 10, loss = 11.12 (2234.6 examples/sec; 0.057 sec/batch)
2018-03-05 04:39:55.025589: step 20, loss = 11.01 (2083.2 examples/sec; 0.061 sec/batch)
2018-03-05 04:39:55.668362: step 30, loss = 10.85 (1894.9 examples/sec; 0.068 sec/batch)
2018-03-05 04:39:56.305252: step 40, loss = 10.72 (1816.1 examples/sec; 0.070 sec/batch)
2018-03-05 04:39:56.958798: step 50, loss = 10.72 (2370.1 examples/sec; 0.054 sec/batch)
2018-03-05 04:39:57.622728: step 60, loss = 10.60 (1597.2 examples/sec; 0.080 sec/batch)
2018-03-05 04:39:58.280594: step 70, loss = 10.50 (1932.1 examples/sec; 0.066 sec/batch)
2018-03-05 04:40:02.650865: step 80, loss = 10.33 (2322.6 examples/sec; 0.055 sec/batch)
2018-03-05 04:40:03.311098: step 90, loss = 10.35 (2116.3 examples/sec; 0.060 sec/batch)
2018-03-05 04:40:03.922531: step 100, loss = 10.22 (2136.7 examples/sec; 0.060 sec/batch)
2018-03-05 04:40:05.191579: step 110, loss = 10.02 (2078.2 examples/sec; 0.062 sec/batch)
2018-03-05 04:40:05.811269: step 120, loss = 10.02 (2396.7 examples/sec; 0.053 sec/batch)
2018-03-05 04:40:06.444760: step 130, loss = 9.92 (2304.1 examples/sec; 0.056 sec/batch)
2018-03-05 04:40:07.083557: step 140, loss = 9.95 (1968.4 examples/sec; 0.065 sec/batch)
2018-03-05 04:40:07.717866: step 150, loss = 9.88 (1941.3 examples/sec; 0.066 sec/batch)
2018-03-05 04:40:12.595326: step 160, loss = 9.78 (2032.0 examples/sec; 0.063 sec/batch)
2018-03-05 04:40:13.256459: step 170, loss = 9.66 (1801.6 examples/sec; 0.071 sec/batch)
2018-03-05 04:40:13.882327: step 180, loss = 9.57 (2418.8 examples/sec; 0.053 sec/batch)
.
.
.
执行结束后将模型压缩,并下载到本地:
!tar -zcvf tf-model.tar.gz tf-model
下载模型到本地:
from google.colab import files
files.download('tf-model.tar.gz')
最后进行测试:
!python3 drive/Cifar10/cifar10_eval.py
/usr/local/lib/python3.6/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
2018-02-04 07:54:41.306434: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-02-04 07:54:41.306951: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:00:04.0
totalMemory: 11.17GiB freeMemory: 11.11GiB
2018-02-04 07:54:41.306978: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
model_checkpoint_path: "tf-model/model.ckpt-29999"
all_model_checkpoint_paths: "tf-model/model.ckpt-26000"
all_model_checkpoint_paths: "tf-model/model.ckpt-27000"
all_model_checkpoint_paths: "tf-model/model.ckpt-28000"
all_model_checkpoint_paths: "tf-model/model.ckpt-29000"
all_model_checkpoint_paths: "tf-model/model.ckpt-29999"
2018-02-04 07:54:42.467940: precision @ 1 = 0.914
2018-02-04 07:54:49.018441: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
model_checkpoint_path: "tf-model/model.ckpt-29999"
all_model_checkpoint_paths: "tf-model/model.ckpt-26000"
all_model_checkpoint_paths: "tf-model/model.ckpt-27000"
all_model_checkpoint_paths: "tf-model/model.ckpt-28000"
all_model_checkpoint_paths: "tf-model/model.ckpt-29000"
all_model_checkpoint_paths: "tf-model/model.ckpt-29999"
2018-02-04 07:54:49.205102: precision @ 1 = 0.922
2018-02-04 07:54:55.774931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
model_checkpoint_path: "tf-model/model.ckpt-29999"
all_model_checkpoint_paths: "tf-model/model.ckpt-26000"
all_model_checkpoint_paths: "tf-model/model.ckpt-27000"
all_model_checkpoint_paths: "tf-model/model.ckpt-28000"
all_model_checkpoint_paths: "tf-model/model.ckpt-29000"
all_model_checkpoint_paths: "tf-model/model.ckpt-29999"
Colaboratory自带的框架为Tensorflow,也可以自己安装别的库。
安装Keras:
!pip install -q keras
import keras
安装PyTorch:
!pip install -q http://download.pytorch.org/whl/cu75/torch-0.2.0.post3-cp27-cp27mu-manylinux1_x86_64.whl torchvision
import torch
安装OpenCV:
!apt-get -qq install -y libsm6 libxext6 && pip install -q -U opencv-python
import cv2
安装其他库:
用!pip install或者!apt-get install命令。
总结
Colaboratory是一个在线Jupyter 笔记本环境,并且可以免费使用Google提供的GPU,对于机器配置较差或者没有GPU的小伙伴确实是一个不小的福利,更多玩法,自己去发掘吧。