探索Google BigQuery：利用Langchain高效加载大数据

最新推荐文章于 2024-10-05 11:27:00 发布

afTFODguAKBF

最新推荐文章于 2024-10-05 11:27:00 发布

阅读量259

点赞数 5

文章标签： langchain 大数据 python

本文链接：https://blog.csdn.net/afTFODguAKBF/article/details/142364193

版权

引言

Google BigQuery 是 Google Cloud Platform 的一部分，它提供了无服务器、经济高效的企业级数据仓库解决方案，可以跨云处理和扩展数据。本文旨在介绍如何使用 Langchain 提供的工具，通过 BigQueryLoader 简化在 BigQuery 中加载和处理查询结果。

主要内容

BigQueryLoader 的概述

BigQueryLoader 是一个用于从 BigQuery 加载数据的工具，它的使用可以显著简化从大规模数据集中提取和处理数据的步骤。

安装 Langchain Google Community 包

首先，你需要安装相应的 Python 包：

%pip install --upgrade --quiet langchain-google-community[bigquery]

使用 BigQueryLoader 加载数据

你可以编写一个基本的 SQL 查询，然后通过 BigQueryLoader 来加载数据。

基础用法

from langchain_google_community import BigQueryLoader

BASE_QUERY = """
SELECT
  id,
  dna_sequence,
  organism
FROM (
  SELECT
    ARRAY (
    SELECT
      AS STRUCT 1 AS id, "ATTCGA" AS dna_sequence, "Lokiarchaeum sp. (strain GC14_75)." AS organism
    UNION ALL
    SELECT
      AS STRUCT 2 AS id, "AGGCGA" AS dna_sequence, "Heimdallarchaeota archaeon (strain LC_2)." AS organism
    UNION ALL
    SELECT
      AS STRUCT 3 AS id, "TCCGGA" AS dna_sequence, "Acidianus hospitalis (strain W1)." AS organism) AS new_array),
  UNNEST(new_array)
"""

loader = BigQueryLoader(BASE_QUERY)
data = loader.load()

print(data)

指定内容与元数据列

有时你可能需要分开指定内容和元数据列，这可以通过传入参数来实现：

loader = BigQueryLoader(
    BASE_QUERY,
    page_content_columns=["dna_sequence", "organism"],
    metadata_columns=["id"],
)

data = loader.load()
print(data)

添加元数据来源

在某些情况下，你可能需要将一列数据作为元数据来源进行标记：

ALIASED_QUERY = """
SELECT
  id,
  dna_sequence,
  organism,
  id as source
FROM (
  SELECT
    ARRAY (
    SELECT
      AS STRUCT 1 AS id, "ATTCGA" AS dna_sequence, "Lokiarchaeum sp. (strain GC14_75)." AS organism
    UNION ALL
    SELECT
      AS STRUCT 2 AS id, "AGGCGA" AS dna_sequence, "Heimdallarchaeota archaeon (strain LC_2)." AS organism
    UNION ALL
    SELECT
      AS STRUCT 3 AS id, "TCCGGA" AS dna_sequence, "Acidianus hospitalis (strain W1)." AS organism) AS new_array),
  UNNEST(new_array)
"""

loader = BigQueryLoader(ALIASED_QUERY, metadata_columns=["source"])
data = loader.load()
print(data)