elasticsearch学习笔记

1.Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot determine write shards for

在config/analysis下创建 synonym.txt文件即可

The index operation automatically creates an index if it has not been created before (check out thecreate index API for manually creating an index), and also automatically creates a dynamic type mapping for the specific type if one has not yet been created (check out the put mapping API for manually creating a type mapping)

es 会自动创建index和type如果不存在的话

2.

//利用隐式转换，native的支持

  implicit class ESConfig(conf: SparkConf) {

    def build(index: String, typeTable: String, mapId: Option[String] = None): SparkConf = {
      val sc = conf
        //        .set("spark.buffer.pageSize", "8m")
        .set("es.nodes", config.nodes)
        .set("es.port", config.port)
        .set("es.scroll.size", "2000")
        .set("es.resource", s"$index/$typeTable")
        .set("es.index.auto.create", "true")
        .set("es.write.operation", "upsert")
        .set("es.batch.size.entries", config.batchSize)
      mapId match {
        case Some(id) => sc.set("es.mapping.id", id)
        case None => sc
      }
    }


  }