gcp设置root登录_我在gcp上进行数据工程的本地环境设置

gcp设置root登录

This guide is intended to be a handy reference for myself when i’m looking for a specific link or command, or setting up a new workstation. I’m expecting this guide to grow with more commands over time.

当我在寻找特定的链接或命令或设置新的工作站时，本指南旨在为我自己提供方便的参考。 我希望随着时间的推移，本指南将包含更多命令。

I’ll be covering the following areas in this guide:

我将在本指南中涵盖以下领域：

• Homebrew
  家酿
• Pyenv
  平耶夫
• Virtual env
  虚拟环境
• Google Cloud Python Client
  Google Cloud Python客户端
• Google Auth Library
  Google Auth库
• Authentication from environment variables
  通过环境变量进行身份验证
• Handy ~/.bashrc or ~/.zshrc lines
  方便的〜/ .bashrc或〜/ .zshrc行

自制，Pyenv和虚拟环境 (Homebrew, Pyenv & Virtual env)

This holy trinity should be taught in all tutorials. If you want control and understanding of your local environment, the combination of homebrew, pyenv & venv is the only way to go.

在所有教程中都应该教导这个三位一体。 如果您想控制和了解您的本地环境，那么将自制，pyenv和venv结合使用是唯一的方法。

家酿 (Homebrew)

Homebrew is the de-facto package manager for mac/linux. Find more information about how to install it here.

Homebrew是mac / linux的事实上的软件包管理器。 在此处找到有关如何安装它的更多信息。

Commands to know:

要知道的命令：

brew doctor              #Performs a health-check on your install
brew install *name*      #For installing cli-based applications
brew cask install *name* #For installing gui-based applications

平耶夫 (Pyenv)

Forgot about ‘installing python’. Install pyenv (with homebrew) and use it to install and select python versions. Pyenv basically intercepts the command ‘python’ in your terminal and makes it point to the specific python version you want. More information on pyenv can be found here.

忘记了“安装python”。 安装pyenv(使用自制软件)，并使用它来安装和选择python版本。 Pyenv基本上在终端中拦截命令“ python”，并使其指向您想要的特定python版本。 可以在这里找到有关pyenv的更多信息。

brew install pyenv

To make sure every terminal session has pyenv initiated you need to put an init-script in your .bash_profile or .zshrc. Information on how to do this can be found after point at point 3 of ‘Basic GitHub Checkout’ here.

为了确保每个终端会话都启动了pyenv，您需要在.bash_profile或.zshrc中放入一个初始化脚本。 如何做到这一点的信息可以在“基本GitHub的结账”的点3点之后找到这里 。

When you’ve got pyenv working you can install your preferred python version like so:

pyenv运行后，您可以安装首选的python版本，如下所示：

pyenv install 3.7.8   #Installs python 3.7.8 pyenv global 3.7.8    #Sets python 3.7.8 to be your global version

Type ‘python’ in your terminal and watch the magic of pyenv.

在终端中输入“ python”，然后观看pyenv的魔力。

python --version
> Python 3.7.8

Venv (Venv)

Packages in Python are by default installed to a global packages folder. If you want to ensure your code performs the same in the cloud as on your local computer, this is not ideal.

默认情况下，Python中的软件包会安装到全局软件包文件夹中。 如果要确保您的代码在云中的性能与本地计算机上的性能相同，那是不理想的。

Venv solves this problem by creating virtual environments that are project specific. Packages can be installed to this virtual environment instead of the global scope. By utilizing a requirements.txt file to keep track of packages you want installed for a specific project you can ensure consistency between the cloud and your local development environment.

Venv通过创建特定于项目的虚拟环境来解决此问题。 可以将软件包安装到此虚拟环境，而不是全局范围。 通过利用requirements.txt文件来跟踪要为特定项目安装的软件包，可以确保云与本地开发环境之间的一致性。

The venv-module is bundled with python since 3.5.

从3.5开始，venv-module与python捆绑在一起。

To create a venv-config folder run the following in your terminal:

要创建venv-config文件夹，请在终端中运行以下命令：

python -m venv venv-config      #Creates the venv-config folder
source venv-config/bin/activate #Takes you inside the virtual env

You are now inside the virtual environment. Feel free to install any packages you want. Preferably these are listed in a requirements.txt file and can be installed with this command:

您现在位于虚拟环境中。 随意安装所需的任何软件包。 最好将这些列在requirements.txt文件中，并可以使用以下命令安装：

pip install -r requirements.txt

When you want to leave the virtual environment, type:

当您想离开虚拟环境时，输入：

deactivate

GCP客户端库 (GCP client libraries)

To be able to access googles services you need a client library that be ‘imported’ from and used as a module in your Python code. This comes in two forms (one for all discovery-based APIs and one for interacting with services on GCP). There’s a slight overlap from the first one to the second one, my advise is to use the second one when available

为了能够访问googles服务，您需要一个客户端库，该客户端库是从Python代码“导入”并用作模块的。 这有两种形式(一种用于所有基于发现的API，一种用于与GCP上的服务进行交互)。 第一个和第二个之间有一点重叠，我的建议是在可用时使用第二个

Google Cloud Python客户端 (Google Cloud Python Client)

Supports all GCP services. Please note that this is just a container repo, all specific clients have their own specific libraries. For example the Cloud Storage API Client can be found here

支持所有GCP服务。 请注意，这只是一个容器存储库，所有特定的客户端都有自己的特定库。 例如，可以在此处找到Cloud Storage API客户端

All client libraries can be pip installed (or put in requirements.txt) on the format:

所有客户端库都可以采用以下格式进行pip安装(或放置在requirements.txt中)：

google-cloud-*service*

And imported in your code (main.py) like so:

并以如下方式导入您的代码(main.py)中：

from google.cloud import *service*

Please note that there’s also a Google API Python Client. This library is intended to be used for the discovery-based APIs, that is to say Googles products outside of GCP. For example the Google Analytics API. I found this rather confusing at first.

请注意，还有一个Google API Python客户端 。 该库旨在用于基于发现的API，即Google的GCP以外的产品。 例如Google Analytics(分析)API。 起初我觉得这很混乱。

Google Auth库 (Google Auth Library)

This library contains the authorization-layer depended upon by all the google-cloud-*service* libraries. For clarity i like putting this in the requirements.txt when i’m specifically using it:

该库包含所有google-cloud- * service *库所依赖的授权层。 为了清楚起见，我喜欢在专门使用它时将其放在requirements.txt中：

google-auth

And this is how you go about creating a client with specific credentials:

这就是使用特定凭据创建客户端的方式：

from google.oauth2 import service_account
from google.cloud import storagecredentials = service_account.Credentials.from_service_account_file(
    'path_to_service_account_key.json',
    scopes=['https://www.googleapis.com/auth/devstorage.read_only']
)storage_client = storage.Client(credentials=credentials)

A full list of available scopes can be found here.

可用范围的完整列表可以在这里找到。

通过环境变量进行认证〜 (Authentication from environment variables~)

〜/ .bashrc或〜/ .zshrc (~/.bashrc or ~/.zshrc)

Depending on your shell driver, one of these files is executed when you start a new shell session. By placing initializiation calls and the definition of handy environment variables here you…

根据您的Shell驱动程序，当您启动新的Shell会话时，将执行以下文件之一。 通过在此处放置初始化调用和方便的环境变量的定义，您可以…

GOOGLE_APPLICATION_CREDENTIALS (GOOGLE_APPLICATION_CREDENTIALS)

When deploying functions or apps to GCP, GCP automatically injects a service account to be used by all clients created in the code that It does this by setting the environment variable GOOGLE_APPLICATION_CREDENTIALS to point to a path with the service account json key.

将功能或应用程序部署到GCP时，GCP会通过设置环境变量GOOGLE_APPLICATION_CREDENTIALS指向带有服务帐户json密钥的路径，来自动注入一个服务帐户，供所有在此代码中创建的客户端使用。

In order to get the same feature to work when developing and testing code locally you need to set up the environment variable GOOGLE_APPLICATION_CREDENTIALS to have an absolute path to the service account json key that you want to use for local development.

为了使相同的功能在本地开发和测试代码时起作用，您需要设置环境变量GOOGLE_APPLICATION_CREDENTIALS来获取要用于本地开发的服务帐户json密钥的绝对路径。

export GOOGLE_APPLICATION_CREDENTIALS= "/path_to_key.json"

With this you can initialize clients in this way:

这样，您可以通过以下方式初始化客户端：

from google.cloud import storage#Client with credentials from environment variable.
storage_client = storage.Client()

方便的〜/ .bashrc或〜/ .zshrc行 (Handy ~/.bashrc or ~/.zshrc lines)

Instead of memorizing venv-related commands, give them easy to remember aliases:

与其记住与venv相关的命令，不如让它们容易记住别名：

alias venv="python -m venv venv"
alias venva="source venv/bin/activate"
alias pipi="pip install -r requirements.txt"

pyenv:

pyenv：

export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"eval "$(pyenv init -)"

翻译自: https://medium.com/@maxkrog/my-local-environment-setup-for-data-engineering-on-gcp-5611e78aa28

gcp设置root登录