qq_33246702的博客

Load and Use Secrets列出当前的secretsdatabricks secrets list-scopes创建一个scopedatabricks secrets create-scope —scope + scope名称databricks secrets create-scope --scope demo-scope删除scopedatabricks secrets delete-scope --scope demo-scope创建或更新secretdatab

DBFS CLI查看DBFS上都有哪些文件databricks fs lsdatabricks fs lsdatabricks fs ls dbfs:/mnt查看一个文件的具体内容 databricks fs catdatabricks fs cat dbfs:/tmp/my-file.txtApache Spark is awesome!文件复制databricks fs cpdatabricks fs cp dbfs:/tmp/your_file.txt dbfs:/parent

Setting up the CLI安装pip install databricks-cli设置连接到databricks从portal上获取host的url ,一般类似于这样https://adb-79479539573402579589.1.databricks.azure.cn/创建一个tokenSettings > User Settings > Access Tokens > Generate New Token通过如下命令，然后把host和toke

说明首先调整配置信息spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")在写入分区表的时候，一定要注意字段顺序，需要把分区字段放到最后，且如果有多个字段分区的话，顺序也要对应。def re_arrange_partition_columns(df,partition_columns): ''' df : 输入的df,spark.DataFrame类型 partition_columns :

常用语法SET TIME ZONE LOCALSET TIME ZONE 'timezone_value'SET TIME ZONE INTERVAL interval_literal参数解释LOCALSet the time zone to the one specified in the java user.timezone property, or to the environment variable TZ if user.timezone is undefined, or to

常用语法SET TIME ZONE LOCALSET TIME ZONE 'timezone_value'SET TIME ZONE INTERVAL interval_literal参数解释LOCALSet the time zone to the one specified in the java user.timezone property, or to the environment variable TZ if user.timezone is undefined, or to

Create/Write a Delta TableeventsDF = spark.read.parquet(eventsPath)Convert data to a Delta table using the schema provided by the DataFrame使用save到路径保存delta tabledeltaPath = f"{delta_path}/delta-events"eventsDF.write.format("delta").mode("overwrite

Reading Datadisplay(dbutils.fs.ls('/mnt/training/ecommerce/events/events-2020-07-03.json'))schema = "device STRING, ecommerce STRUCT<purchase_revenue_in_usd: DOUBLE, total_item_quantity: BIGINT, unique_items: BIGINT>, event_name STRING, event_pre

readStreamschema = "device STRING, ecommerce STRUCT<purchase_revenue_in_usd: DOUBLE, total_item_quantity: BIGINT, unique_items: BIGINT>, event_name STRING, event_previous_timestamp BIGINT, event_timestamp BIGINT, geo STRUCT<city: STRING, state: S

Get partitions and coresUse an rdd method to get the number of DataFrame partitionsdf = spark.read.parquet(eventsPath)df.rdd.getNumPartitions()Access SparkContext through SparkSession to get the number of cores or slotsSparkContext is also provided

User-Defined FunctionsDefine a functionCreate and apply UDFRegister UDF to use in SQLUse Decorator Syntax (Python Only)Use Vectorized UDF (Python Only)MethodsUDF Registration (spark.udf): registerBuilt-In Functions : udfPython UD

Extract item detailsdf = spark.read.parquet(salesPath).select('email','items')display(df)这里的字段items是一个列表，里面装了一个或多个字典explore: 将一个列表展开，相当于把数据一行转多行了。split:将一个文件按分隔符拆开。from pyspark.sql.functions import *detailsDF = (df.withColumn("items", explode

Datetime FunctionsCurrent Date/TimestampCast to timestampFormat datetimesExtract from timestampConvert to dateManipulate datetimesMethodsColumn : castBuilt-In Functions : date_format, to_date, date_add, year, month, dayofweek, mi

Grouping dataUse the DataFrame groupBy method to create a grouped data objectThis grouped data object is called RelationalGroupedDataset in Scala and GroupedData in Pythondf.groupBy("geo.state", "geo.city")Grouped data methodsVarious aggregate metho

join两个DataFrame根据某个条件进行关联。类似的还有crossJoin返回一个笛卡尔积表ParametersotherDataFrameRight side of the joinonstr, list or Column, optionala string for the join column name, a list of column names, a join expression (Column), or a list of Columns. If on is a str

Construct columnsA column is a logical construction that will be computed based on the data in a DataFrame using an expressionConstruct a new column based on the input columns existing in a DataFramefrom pyspark.sql.functions import colcol("device")d

常用方法spark.sql执行sql语句sqlstr = """select store_code,store_name,locationfrom storewhere country = 'CN'order by id"""spark.sql(sqlstr)2. show & display查看数据showdf.show()display(推荐)display(df)3. 从现有表创建一个DataFrame。spark.table & spar

ReaderRead from CSV filesspark.read.csv也可以读取csv文件，而且更常用。Read from CSV with DataFrameReader’s csv method and the following options:Tab separator, use first line as header, infer schemafile_csv = "/mnt/training/ecommerce/users/users-500k.csv"df = (sp

Apache-DBUtils简介commons-dbutils 是 Apache 组织提供的一个开源 JDBC工具类库，它是对JDBC的简单封装，学习成本极低，并且使用dbutils能极大简化jdbc编码的工作量，同时也不会影响程序的性能。API介绍：org.apache.commons.dbutils.QueryRunnerorg.apache.commons.dbutils.ResultSetHandler工具类：org.apache.commons.dbutils.DbUtils

数据库连接池的必要性在使用开发基于数据库的web程序时，传统的模式基本是按以下步骤：在主程序（如servlet、beans）中建立数据库连接进行sql操作断开数据库连接这种模式开发，存在的问题:普通的JDBC数据库连接使用 DriverManager 来获取，每次向数据库建立连接的时候都要将 Connection 加载到内存中，再验证用户名和密码(得花费0.05s～1s的时间)。需要数据库连接的时候，就向数据库要求一个，执行完成后再断开连接。这样的方式将会消耗大量的资源和时间。**