Spark 1.6.0 (Scala 2.11)版本的编译与安装部署

2016年元月4号, spark 在其官网上公开了1.6.0版本,于是进行下载和编译.

有了前面的编译经验和之前下载好的java类包，花了大概一分钟就编译妥当，于是重新部署配置一下，马上OK。简直是高效率。

对于scala的编译，还是只需要一条语句。build/sbt -Dscala=2.11 -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver assembly。

对spark 1.6中的新特性进行测试: (DataSet)

其中1.6的新特性还包括:

Spark Core/SQL

• API Updates
  • SPARK-9999  Dataset API - A new Spark API, similar to RDDs, that allows users to work with custom objects and lambda functions while still gaining the benefits of the Spark SQL execution engine.
  • SPARK-10810 Session Management - Different users can share a cluster while having different configuration and temporary tables.
  • SPARK-11197 SQL Queries on Files - Concise syntax for running SQL queries over files of any supported format without registering a table.
  • SPARK-11745 Reading non-standard JSON files - Added options to read non-standard JSON files (e.g. single-quotes, unquoted attributes)
  • SPARK-10412 Per-operator Metrics for SQL Execution - Display statistics on a per-operator basis for memory usage and spilled data size.
  • SPARK-11329 Star (*) expansion for StructTypes - Makes it easier to nest and unnest arbitrary numbers of columns
  • SPARK-4849  Advanced Layout of Cached Data - storing partitioning and ordering schemes in In-memory table scan, and adding distributeBy and localSort to DF API
  • SPARK-11778  - DataFrameReader.table supports specifying database name. For example, sqlContext.read.table(“dbName.tableName”) can be used to create a DataFrame from a table called “tableName” in the database “dbName”.
  • SPARK-10947  - With schema inference from JSON into a Dataframe, users can set primitivesAsString to true (in data source options) to infer all primitive value types as Strings. The default value of primitivesAsString is false.
• Performance
  • SPARK-10000 Unified Memory Management - Shared memory for execution and caching instead of exclusive division of the regions.
  • SPARK-11787 Parquet Performance - Improve Parquet scan performance when using flat schemas.
  • SPARK-9241  Improved query planner for queries having distinct aggregations - Query plans of distinct aggregations are more robust when distinct columns have high cardinality.
  • SPARK-9858  Adaptive query execution - Initial support for automatically selecting the number of reducers for joins and aggregations.
  • SPARK-10978 Avoiding double filters in Data Source API - When implementing a data source with filter pushdown, developers can now tell Spark SQL to avoid double evaluating a pushed-down filter.
  • SPARK-11111 Fast null-safe joins - Joins using null-safe equality (<=>) will now execute using SortMergeJoin instead of computing a cartisian product.
  • SPARK-10917, SPARK-11149 In-memory Columnar Cache Performance - Significant (up to 14x) speed up when caching data that contains complex types in DataFrames or SQL.
  • SPARK-11389 SQL Execution Using Off-Heap Memory - Support for configuring query execution to occur using off-heap memory to avoid GC overhead

Spark Streaming

• API Updates
  • SPARK-2629  New improved state management - mapWithState - a DStream transformation for stateful stream processing, supercedes updateStateByKey in functionality and performance.
  • SPARK-11198 Kinesis record deaggregation - Kinesis streams have been upgraded to use KCL 1.4.0 and supports transparent deaggregation of KPL-aggregated records.
  • SPARK-10891 Kinesis message handler function - Allows arbitrary function to be applied to a Kinesis record in the Kinesis receiver before to customize what data is to be stored in memory.
  • SPARK-6328  Python Streaming Listener API - Get streaming statistics (scheduling delays, batch processing times, etc.) in streaming.
• UI Improvements
  • Made failures visible in the streaming tab, in the timelines, batch list, and batch details page.
  • Made output operations visible in the streaming tab as progress bars.

MLlib

• New algorithms/models
  • SPARK-8518  Survival analysis - Log-linear model for survival analysis
  • SPARK-9834  Normal equation for least squares - Normal equation solver, providing R-like model summary statistics
  • SPARK-3147  Online hypothesis testing - A/B testing in the Spark Streaming framework
  • SPARK-9930  New feature transformers - ChiSqSelector, QuantileDiscretizer, SQL transformer
  • SPARK-6517  Bisecting K-Means clustering - Fast top-down clustering variant of K-Means
• API improvements
  • ML Pipelines
    • SPARK-6725  Pipeline persistence - Save/load for ML Pipelines, with partial coverage of spark.ml algorithms
    • SPARK-5565  LDA in ML Pipelines - API for Latent Dirichlet Allocation in ML Pipelines
  • R API
    • SPARK-9836  R-like statistics for GLMs - (Partial) R-like stats for ordinary least squares via summary(model)
    • SPARK-9681  Feature interactions in R formula - Interaction operator “:” in R formula
  • Python API - Many improvements to Python API to approach feature parity
• Misc improvements
  • SPARK-7685 , SPARK-9642  Instance weights for GLMs - Logistic and Linear Regression can take instance weights
  • SPARK-10384, SPARK-10385 Univariate and bivariate statistics in DataFrames - Variance, stddev, correlations, etc.
  • SPARK-10117 LIBSVM data source - LIBSVM as a SQL data source
• Documentation improvements
  • SPARK-7751  @since versions - Documentation includes initial version when classes and methods were added
  • SPARK-11337 Testable example code - Automated testing for code in user guide examples