浏览器直接上传文件到 Cloud Storage,绕开 App Engine Request 最大 32M 限制

随着流量的增加,GCP App Engine 会自动为应用分配更多的资源,但自动分配资源仍然受到一些阀值的约束,其中一条便是:发送到应用的请求,请求体不能大于32M。而对于一些上传大文件的需求,这个限制使得那些将文件上传服务的 EndPoint 设置在 App Engine 上的应用无法正常处理请求。

考虑到 App Engine 不允许应用操作本地存储,而且我们上传的文件一般也不会保存在本地,而是存到 Cloud Storage,File Store 等地方。

这里以 Cloud Storage 为例,Cloud Storage 提供了Signed URLs(除了失效时间之外,可以不受任何约束)访问 Storage 中的资源。这种方式在进行签名时需要将Content-Type用作签名的一部分,然而对于mutlipart/formdata类型的表单,Content-Tyep 是 mutlipart/formdata ---boundaryString的形式,而 boundaryString 是动态生成的,这就导致无法为文件上传请求进行签名

在这个 questions 中也可以看到有人给出了解决方案,那就是使用 post-object#policydocument

看如下的示例:

<form action="https://storage.googleapis.com/bucket-name/file-name" method="post" enctype="multipart/form-data">
    <input type="hidden" name="GoogleAccessId" value="1234567890123@developer.gserviceaccount.com">
    <input type="hidden" name="policy" value="eyJleHBpcmF0aW9uIjogIjIwMTAtMDYtMTZUMTE6MTE6MTFaIiwNCiAi">
    <input type="hidden" name="signature" value="BSAMPLEaASAMPLE6SAMPLE+SAMPPLEqSAMPLEPSAMPLE+SAMPLEgSAMPL">
    <input type="hidden" name="success_action_status" value="200">
    <input type="file" name="file">
    <input type="submit" value="Upload">
</form>

policy 字段包含一个指明用户需要遵循的上传规则,即 policy document,经过 base64 编码后填入 policy 字段。

// policy document
{
  "expiration": "2010-06-16T11:11:11Z",
  "conditions": [
    ["content-length-range", 0, 1073741824],
    ["eq", "$success_action_status", "200"]
  ]
}

上面的 content-length-range指明用户上传文件大小限制在 0~1GB 之间,此外还有其他几种规则可以添加,需要注意的是,这种方式只支持一次上传一个文件。form 的 signature 字段是签名之后的 policy document,需要使用 service account key 进行签名,签名算法为 SHA256withRSA。go 的实现如下:


import (
	"crypto"
	"crypto/rand"
	"crypto/rsa"
	"crypto/sha256"
	"crypto/x509"
	"encoding/base64"
	"encoding/json"
	"encoding/pem"
)

func signPolicy(policy string) (string, error) {
	secretKey, err := loadSecretKey()
	if err != nil {
		return "", err
	}

	// sign base64ed policy document using RSA with SHA-256 using a secret key
	d := sha256SumMessage(base64.StdEncoding.EncodeToString([]byte(policy)))
	messageDigest, err := rsa.SignPKCS1v15(rand.Reader, secretKey, crypto.SHA256, d)
	if err != nil {
		return "", err
	}

	return base64.StdEncoding.EncodeToString(messageDigest), nil
}

func sha256SumMessage(msg string) []byte {
	h := sha256.New()
	h.Write([]byte(msg))
	d := h.Sum(nil)
	return d
}


func loadSecretKey() (priKey *rsa.PrivateKey, err error) {
	// RSA private key, extract from service account key json file, private_key filed
	blockPri, _ := pem.Decode([]byte(`-----BEGIN PRIVATE KEY-----
-----END PRIVATE KEY-----
`))

	// may returns a *rsa.PrivateKey, a *ecdsa.PrivateKey, or a ed25519.PrivateKey
	// see doc here: https://golang.org/src/crypto/x509/pkcs8.go
	prkI, err := x509.ParsePKCS8PrivateKey(blockPri.Bytes)
	if err != nil {
		return nil, err
	}

	return prkI.(*rsa.PrivateKey), err
}

在 GCP 的 IAM & admin#Service account 中可手动为 App Engine default service account 创建 RSA key,选择把私钥以 json 文件的格式下载到本地。文件格式如下:

{
  "type": "service_account",
  "project_id": "***",
  "private_key_id": "***",
  "private_key": "-----BEGIN PRIVATE KEY-----***-----END PRIVATE KEY-----\n",
  "client_email": "***",
  "client_id": "***",
  "auth_uri": "***",
  "token_uri": "***",
  "auth_provider_x509_cert_url": "***",
  "client_x509_cert_url": "***"
}

其中 private_key 的值即为 RSA 私钥。需要注意的是,把私钥直接放在代码中是很不安全的,接下来需要做的就是找个安全的地方来保存并定期更替这些私钥。KMS 是个可选的方案。

展开阅读全文

Cloud-Run进程失败,并显示500个状态代码和一个错误的gvisor错误

08-15
<div class="post-text" itemprop="text"> <p><strong>Backround</strong></p> <p>The service is a simple Go program that pipes a file from Cloud Storage to the browser.</p> <p>Everything works fine on my Macbook, but fails on Cloud-Run (managed) for some requests. Mostly large mp4 files.</p> <p><strong>Problem</strong></p> <p>The logs just show a <code>500</code> status, as does the browser. But my service doesn't log anything other than starting to copy the file. No IO errors or anything.</p> <p>This message is shown 4 seconds before the <code>500</code> status:</p> <p><code>Container Sandbox Limitation: Unsupported syscall membarrier(0x10,0x0,0x0,0x8,0x775dce0b030,0x775dce0b000). Please, refer to https://gvisor.dev/c/linux/amd64/membarrier for more information.</code></p> <p>I cannot reproduce this locally. Works fine locally with the same configuration and GCP buckets.</p> <p>The service works fine on Cloud-Run with smaller files, like images. Just not the videos I've tried.</p> <p><strong>I've tried</strong></p> <ul> <li>Logging everything up to the <code>io.Copy</code>. No errors, hangs afte <code>io.Copy</code> is called.</li> <li>Increasing the memory of the container. It's now running a 1G. No change from 512M.</li> <li>Running in a Docker container locally with the same configuration, same credentials. No problems.</li> <li>Reaching out to GCP on Twitter</li> </ul> <p><strong>Update 2019-08-16</strong></p> <p>I created a very simple service that prints 'A' to a http responsewriter. It also works perfectly locally, yet returns 500 on cloud-run with larg-ish sizes. 1MB OK, 5MB OK, 50 MB fails, 100MB fails, etc. There are no membarrier messages when this service runs.</p> <p>Code is available here: <a href="https://github.com/andrioid/reproduce-cloud-run-bug" rel="nofollow noreferrer">https://github.com/andrioid/reproduce-cloud-run-bug</a></p> <p>Reported on issue-tracker as well: <a href="https://issuetracker.google.com/issues/139511257" rel="nofollow noreferrer">https://issuetracker.google.com/issues/139511257</a></p> <p><strong>Update 2: Probable cause</strong></p> <p>Seems like there is a hard limit on response sizes to 32MB.</p> <p><a href="https://cloud.google.com/run/quotas" rel="nofollow noreferrer">https://cloud.google.com/run/quotas</a></p> <p>Very disappointing that this cannot be increased and that the error doesn't mention this limit, neither does the log file.</p> </div>
©️2020 CSDN 皮肤主题: 大白 设计师: CSDN官方博客 返回首页
实付0元
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值