spark 3 0 sql功能更新ansi sql合规性存储分配策略已升级查询

Spark has added a lot of notable features with Spark SQL. Some will have a huge impact on checks like data quality and data validations. Though there is a lot of upgraded features, I’m listing out a few of those as these would be used in most common cases.

Spark在Spark SQL中添加了许多显着的功能。 有些将对检查产生巨大影响，例如数据质量和数据验证。 尽管有很多升级的功能，但我列出了其中的一些功能，因为这些功能将在最常见的情况下使用。

ANSI SQL投诉功能 (ANSI SQL Complaint Features)

For Spark developers, who are at the initial stage of learning SQL commands, query identifier validation might be helpful. They might use keywords as identifiers that are not meant to be used. Even it will work completely fine with spark, This will confuse others who work with the code in the future.

对于处于学习SQL命令初期阶段的Spark开发人员，查询标识符验证可能会有所帮助。 他们可能使用关键字作为不打算使用的标识符。 即使使用spark也会完全正常工作，这会使将来使用该代码的其他人感到困惑。

Spark will allow certain unusual cases like using some reserved keywords as identifiers. Something like this:

Spark将允许某些特殊情况，例如使用一些保留的关键字作为标识符。 像这样：

select * from table_1 create where create.column_1= 1

This query will run without any issues in spark. create is the most common reserved keyword, used for creating tables with SQL, but it can be used as an identifier in spark without any issues.

此查询将运行，不会在spark中出现任何问题。 create是最常用的保留关键字，用于通过SQL创建表，但可以用作spark中的标识符，不会出现任何问题。

To overcome this, and involve query validation in runtime, Spark is now compliant with ANSI SQL standard. Validation has been added to the catalyst parser level. To enable ANSI mode query validation, switch property spark.sql.ansi.enabled to true. For the same query shown above, Spark will now throw the exception.

为了克服这个问题，并在运行时进行查询验证，Spark现在符合ANSI SQL标准。 验证已添加到催化剂解析器级别。 要启用ANSI模式查询验证，请将属性spark.sql.ansi.enabled切换为true。 对于上面显示的相同查询，Spark现在将引发异常。

Error in SQL statement: ParseException:  no viable alternative at input 'create'(line 1, pos 38)== SQL == 
select * from table_1 create  where create.column_1= 1 
----------------------^^^com.databricks.backend.common.rpc.DatabricksExceptions$SQLExecutionException: org.apache.spark.sql.catalyst.parser.ParseException:

Following are the keywords standardized in Spark 3.0:

以下是Spark 3.0中标准化的关键字：

演示地址

To disable the ANSI standard validation, Falsify spark.sql.ansi.enabled.

要禁用ANSI标准验证，请伪造spark.sql.ansi.enabled。

商店分配政策 (Store Assignment policy)

This feature is introduced for stern data quality checks during migration from SQL kind of environment. Below Insert table statement(Ingesting string into an integer column) would execute without any runtime exception in Spark 2.4 and below.

引入此功能是为了在从SQL类环境迁移期间进行严厉的数据质量检查。 在Spark 2.4及更低版本中，Insert table语句(将字符串吸收到整数列中)将在没有任何运行时异常的情况下执行。