本文介绍了如何使用elasticsearch-spark连接器从Spark读取Elasticsearch数据，遇到的问题是即使指定了查询中只获取部分字段，结果仍然包含所有字段。解决方案是利用DataFrame而非RDD，并启用pushdown predicate，通过`select()`来指定字段，用`limit()`替换`size`参数以限制返回记录数。
I've done some experiments in the spark-shell with the elasticsearch-spark connector. Invoking spark:
] $SPARK_HOME/bin/spark-shell --master local[2] --jars ~/spark/jars/elasticsearch-spark-20_2.11-5.1.2.jar
In the scala shell:
scala> import org.elasticsearch.spark._
scala> val es_rdd = sc.esRDD("myindex/mytype",query="myquery")
It works well, the result contains the good records as specified in myquery. The only thing is that I get all the fields, even if I specify a subset of these fields in the query. Example:
myquery = """{"query":..., "fields":["a","b"], "size":10}"""
returns all the fields, not only a and b (BTW, I noticed that size parameter is not taken in account neither : result contains more than 10 records). Maybe it's important
最低0.47元/天 解锁文章
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
