(转)Google开放AI工具Colaboratory使用教程

Colaboratory 是一个 Google 研究项目，旨在帮助传播机器学习培训和研究成果。它是一个 Jupyter 笔记本环境，不需要进行任何设置就可以使用，并且完全在云端运行。Colaboratory 笔记本存储在 Google 云端硬盘中，并且可以共享，就如同您使用 Google 文档或表格一样。Colaboratory 可免费使用。其实就是薅谷歌的羊毛。

本文主要介绍如何通过Colaboratory连接Google云端硬盘，并使用免费的Tesla K80 GPU执行Google driver中的机器学习程序。

Colaboratory介绍
首先，访问 CoLaboratory 网站，注册后接受使用该工具的邀请。注册成功后可以打开Colaboratory 简介，会出现一个「Hello, Colaboratory」文件，包含一些基本示例。 

注意右上角的连接按钮，如果没有连接是红色的。Colaboratory现在已全面支持python2和python3两个版本，可以在 代码执行程序→更改运行时类型 选择python版本，并在 硬件加速器 中选择是否使用GPU加速。 

连接Google driver
先注册Google云端硬盘，注册后便会有免费的15G空间，日常跑个学习的程序，无论是保存程序还是数据集都足够用了，然后通过Colaboratory连接云端硬盘。 
这里以Cifar-10 GPU版本程序为例： 
首先连接到Google硬盘，因为我的程序都上传在Google硬盘中，新建一个一个python记事本，将下列代码复制到记事本中。

这段代码执行后会安装很多东西

# Install a Drive FUSE wrapper.
# https://github.com/astrada/google-drive-ocamlfuse
!apt-get install -y -qq software-properties-common python-software-properties module-init-tools
!add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null
!apt-get update -qq 2>&1 > /dev/null
!apt-get -y install -qq google-drive-ocamlfuse fuse

执行下面代码，打开链接，将链接中授权的代码复制到框格中回车执行

# Generate auth tokens for Colab
from google.colab import auth
auth.authenticate_user()

# Generate creds for the Drive FUSE library.
from oauth2client.client import GoogleCredentials
creds = GoogleCredentials.get_application_default()
import getpass
!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL
vcode = getpass.getpass()
!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}

最后一步，挂载Google driver

# Create a directory and mount Google Drive using that directory.
!mkdir -p drive
!google-drive-ocamlfuse drive

print('Files in Drive:')
!ls drive/

# Create a file in Drive.
#!echo "This newly created file will appear in your Drive file list." > drive/created.txt

执行 !ls 查看虚拟机中的文件，如果有一个Driver文件夹，便是成功了 

完整连接程序： 
https://colab.research.google.com/drive/1AxtTSlL6CiVzkdTMoy0-4BU_bUhmcIT8 
Google官方导入数据示例： 
https://colab.research.google.com/notebook#fileId=/v2/external/notebooks/io.ipynb&scrollTo=eikfzi8ZT_rW 
官方导入数据示例中有好几种方法，喜欢别的方法也建议去尝试。

执行driver中的程序
修改程序中数据的导入路径，执行程序：

!python3 drive/Cifar10/cifar10_multi_gpu_train.py

执行效果：

/usr/local/lib/python3.6/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Filling queue with 20000 CIFAR images before starting to train. This will take a few minutes.
2018-03-05 04:39:34.103914: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-03-05 04:39:34.104527: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1208] Found device 0 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:00:04.0
totalMemory: 11.17GiB freeMemory: 11.11GiB
2018-03-05 04:39:34.104577: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1308] Adding visible gpu devices: 0
2018-03-05 04:39:34.381677: I tensorflow/core/common_runtime/gpu/gpu_device.cc:989] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10774 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
2018-03-05 04:39:53.097102: step 0, loss = 11.24 (7.1 examples/sec; 18.121 sec/batch)
2018-03-05 04:39:54.442389: step 10, loss = 11.12 (2234.6 examples/sec; 0.057 sec/batch)
2018-03-05 04:39:55.025589: step 20, loss = 11.01 (2083.2 examples/sec; 0.061 sec/batch)
2018-03-05 04:39:55.668362: step 30, loss = 10.85 (1894.9 examples/sec; 0.068 sec/batch)
2018-03-05 04:39:56.305252: step 40, loss = 10.72 (1816.1 examples/sec; 0.070 sec/batch)
2018-03-05 04:39:56.958798: step 50, loss = 10.72 (2370.1 examples/sec; 0.054 sec/batch)
2018-03-05 04:39:57.622728: step 60, loss = 10.60 (1597.2 examples/sec; 0.080 sec/batch)
2018-03-05 04:39:58.280594: step 70, loss = 10.50 (1932.1 examples/sec; 0.066 sec/batch)
2018-03-05 04:40:02.650865: step 80, loss = 10.33 (2322.6 examples/sec; 0.055 sec/batch)
2018-03-05 04:40:03.311098: step 90, loss = 10.35 (2116.3 examples/sec; 0.060 sec/batch)
2018-03-05 04:40:03.922531: step 100, loss = 10.22 (2136.7 examples/sec; 0.060 sec/batch)
2018-03-05 04:40:05.191579: step 110, loss = 10.02 (2078.2 examples/sec; 0.062 sec/batch)
2018-03-05 04:40:05.811269: step 120, loss = 10.02 (2396.7 examples/sec; 0.053 sec/batch)
2018-03-05 04:40:06.444760: step 130, loss = 9.92 (2304.1 examples/sec; 0.056 sec/batch)
2018-03-05 04:40:07.083557: step 140, loss = 9.95 (1968.4 examples/sec; 0.065 sec/batch)
2018-03-05 04:40:07.717866: step 150, loss = 9.88 (1941.3 examples/sec; 0.066 sec/batch)
2018-03-05 04:40:12.595326: step 160, loss = 9.78 (2032.0 examples/sec; 0.063 sec/batch)
2018-03-05 04:40:13.256459: step 170, loss = 9.66 (1801.6 examples/sec; 0.071 sec/batch)
2018-03-05 04:40:13.882327: step 180, loss = 9.57 (2418.8 examples/sec; 0.053 sec/batch)
.
.
.
执行结束后将模型压缩，并下载到本地：

!tar -zcvf tf-model.tar.gz tf-model

下载模型到本地:

from google.colab import files
files.download('tf-model.tar.gz')

最后进行测试：

!python3 drive/Cifar10/cifar10_eval.py

/usr/local/lib/python3.6/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
2018-02-04 07:54:41.306434: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-02-04 07:54:41.306951: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:00:04.0
totalMemory: 11.17GiB freeMemory: 11.11GiB
2018-02-04 07:54:41.306978: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
model_checkpoint_path: "tf-model/model.ckpt-29999"
all_model_checkpoint_paths: "tf-model/model.ckpt-26000"
all_model_checkpoint_paths: "tf-model/model.ckpt-27000"
all_model_checkpoint_paths: "tf-model/model.ckpt-28000"
all_model_checkpoint_paths: "tf-model/model.ckpt-29000"
all_model_checkpoint_paths: "tf-model/model.ckpt-29999"

2018-02-04 07:54:42.467940: precision @ 1 = 0.914
2018-02-04 07:54:49.018441: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
model_checkpoint_path: "tf-model/model.ckpt-29999"
all_model_checkpoint_paths: "tf-model/model.ckpt-26000"
all_model_checkpoint_paths: "tf-model/model.ckpt-27000"
all_model_checkpoint_paths: "tf-model/model.ckpt-28000"
all_model_checkpoint_paths: "tf-model/model.ckpt-29000"
all_model_checkpoint_paths: "tf-model/model.ckpt-29999"

2018-02-04 07:54:49.205102: precision @ 1 = 0.922
2018-02-04 07:54:55.774931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
model_checkpoint_path: "tf-model/model.ckpt-29999"
all_model_checkpoint_paths: "tf-model/model.ckpt-26000"
all_model_checkpoint_paths: "tf-model/model.ckpt-27000"
all_model_checkpoint_paths: "tf-model/model.ckpt-28000"
all_model_checkpoint_paths: "tf-model/model.ckpt-29000"
all_model_checkpoint_paths: "tf-model/model.ckpt-29999"

Colaboratory自带的框架为Tensorflow，也可以自己安装别的库。 
安装Keras：

!pip install -q keras
import keras

安装PyTorch：

!pip install -q http://download.pytorch.org/whl/cu75/torch-0.2.0.post3-cp27-cp27mu-manylinux1_x86_64.whl torchvision
import torch

安装OpenCV：

!apt-get -qq install -y libsm6 libxext6 && pip install -q -U opencv-python
import cv2

安装其他库： 
用!pip install或者!apt-get install命令。
--------------------- 
作者：Junyuan12 
原文：https://blog.csdn.net/qq_29592829/article/details/79444466