Geotrellis学(踩)习(坑)笔记（四）——ETL工具

 

看官方文档的时候看到geotrellis工具有一个看上去很强大的ETL工具，但是看了好几次都看不明白他到底是干什么的，理解和跑通ETL是一个超级曲折艰难的过程。这篇文章就来记录一下ETL的曲折过程~

首先ETL是个啥？？

参考资料：https://docs.microsoft.com/en-us ... relational-data/etl

Extract, transform, and load (ETL) is a data pipeline used to collect data from various sources, transform the data according to business rules, and load it into a destination data store. The transformation work in ETL takes place in a specialized engine, and often involves using staging tables to temporarily hold data as it is being transformed and ultimately loaded to its destination.

The data transformation that takes place usually involves various operations, such as filtering, sorting, aggregating, joining data, cleaning data, deduplicating, and validating data.

ETL工具就是一个从数据仓库里抽取数据、转换数据和加载数据到目的端的过程。

这样说感觉还是很抽象，没有一个很具象的概念，那就先直接上手做吧~

首先我在geotrellis的geotrellis-chatta-demo里面把data全都拷贝到我的工程里，作为测试数据。

import geotrellis.raster.Tile
import geotrellis.spark.SpatialKey
import geotrellis.spark.etl.{Etl}
import geotrellis.spark.util.SparkUtils
import geotrellis.vector.ProjectedExtent
import org.apache.spark.{SparkConf}

object GeoTrellisETL {
  def main(args: Array[String]): Unit = {
    var args=new Array[String](6)
    args(0)="--input"
    args(1)="D:\\IdeaProjects\\GeotrellisETL\\data\\input.json"
    args(2)="--output"
    args(3)="D:\\IdeaProjects\\GeotrellisETL\\data\\output.json"
    args(4)="--backend-profiles"
    args(5)="D:\\IdeaProjects\\GeotrellisETL\\data\\backend-profiles.json"
    goETL(args)
}
  def goETL(args: Array[String]): Unit = {
    implicit val sc = SparkUtils.createSparkContext("GeoTrellis ETL SinglebandIngest", new SparkConf(true).setMaster("local"))
    try {
      Etl.ingest[ProjectedExtent, SpatialKey, Tile](args)
      println("finished!")