StarRocks+Seatunnel+Hive 实战

StarRocks+Seatunnel+Hive

1 StarRocks 部署

参考 https://docs.starrocks.com/zh-cn/main/quick_start/Deploy

CREATE TABLE IF NOT EXISTS example_db.ads_product_sale_d (
    platform_id   INT           NOT NULL COMMENT "",
    cat_id     VARCHAR(500)   NOT NULL COMMENT ""
) ENGINE=OLAP
DUPLICATE KEY(platform_id, cat_id)
DISTRIBUTED BY HASH(platform_id) BUCKETS 8
PROPERTIES("replication_num" = "1");

2 Seatunnel 部署

SPARK_HOME="/opt/cloudera/parcels/CDH/lib/spark"

3 Hive数据导入StarRocks

cat ads_product_sale_d.sh

sh /data/apache-seatunnel-incubating-2.1.2/bin/start-seatunnel-spark.sh -c ads_product_sale_d.conf  -e client -m yarn