[掌握Google BigQuery：高效加载和查询你的数据]

最新推荐文章于 2024-10-30 19:27:31 发布

qq_37836323

最新推荐文章于 2024-10-30 19:27:31 发布

阅读量415

点赞数 4

文章标签： python

本文链接：https://blog.csdn.net/qq_29929123/article/details/142676748

版权

Google BigQuery：高效加载和查询你的数据

引言

在现代数据驱动的世界中，处理海量数据的能力至关重要。Google BigQuery作为一款无服务器的企业级数据仓库，为开发者提供了强大的工具来跨云处理和分析数据。本篇文章将介绍如何使用BigQuery进行数据加载和查询操作，帮助您快速上手。

主要内容

BigQuery基础概念

Google BigQuery是Google Cloud Platform的一部分，它提供了一个高效的数据分析平台，支持大规模数据处理。由于其无服务器的特性，开发者不必担心基础设施的管理。

使用BigQueryLoader加载数据

为了简化数据加载，BigQueryLoader库提供了便捷的接口。我们可以通过简单的SQL查询来获取并处理数据。

基本用法

以下代码展示了如何使用BigQueryLoader加载数据。

%pip install --upgrade --quiet langchain-google-community[bigquery]

from langchain_google_community import BigQueryLoader

BASE_QUERY = """
SELECT
  id,
  dna_sequence,
  organism
FROM (
  SELECT
    ARRAY (
    SELECT
      AS STRUCT 1 AS id, "ATTCGA" AS dna_sequence, "Lokiarchaeum sp. (strain GC14_75)." AS organism
    UNION ALL
    SELECT
      AS STRUCT 2 AS id, "AGGCGA" AS dna_sequence, "Heimdallarchaeota archaeon (strain LC_2)." AS organism
    UNION ALL
    SELECT
      AS STRUCT 3 AS id, "TCCGGA" AS dna_sequence, "Acidianus hospitalis (strain W1)." AS organism) AS new_array),
  UNNEST(new_array)
"""

loader = BigQueryLoader(BASE_QUERY)
data = loader.load()
print(data)

指定内容与元数据

通过指定数据列和元数据列，您可以更灵活地定义数据结构：

loader = BigQueryLoader(
    BASE_QUERY,
    page_content_columns=["dna_sequence", "organism"],
    metadata_columns=["id"],
)

data = loader.load()
print(data)

添加元数据来源

有时，我们需要为数据添加来源标签，这是通过在SQL查询中使用别名实现的：

ALIASED_QUERY = """
SELECT
  id,
  dna_sequence,
  organism,
  id as source
FROM (
  SELECT
    ARRAY (
    SELECT
      AS STRUCT 1 AS id, "ATTCGA" AS dna_sequence, "Lokiarchaeum sp. (strain GC14_75)." AS organism
    UNION ALL
    SELECT
      AS STRUCT 2 AS id, "AGGCGA" AS dna_sequence, "Heimdallarchaeota archaeon (strain LC_2)." AS organism
    UNION ALL
    SELECT
      AS STRUCT 3 AS id, "TCCGGA" AS dna_sequence, "Acidianus hospitalis (strain W1)." AS organism) AS new_array),
  UNNEST(new_array)
"""

loader = BigQueryLoader(ALIASED_QUERY, metadata_columns=["source"])
data = loader.load()
print(data)