1、启动hadoop集群(hadoop-3.3.3)
2、启动hivemetestore的thriftserver服务(apache-hive-3.1.3-bin)
3、启动spark sql(iceberg)
spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.3_2.12:0.14.1\
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.spark_catalog.type=hive
iceberg示例(on Spark3) :
1、创建iceberg表
spark-sql (default)> CREATE TABLE icebergtable1 (id bigint, data string) USING iceberg;
Response code
Time taken: 1.93 seconds
2、insert into数据:
spark-sql (default)> INSERT INTO icebergtable1 VALUES (1, 'a'), (2, 'b'), (3, 'c');
Response code
Time taken: 7.124 seconds
3、查询表的数据
spark-sql (default)> select * from default.icebergtable1;
i