Spark+Python lab1

目标：Python review + Intro to PySpark

Part 1: Math review

略

Part 2: Numpy

求逆矩阵方法：

# Invert AAt with np.linalg.inv()
AAtInv = np.linalg.inv(AAt)

可参见资料：Numpy

Part 3: Additional NumPy and Spark linear algebra

• 切片和重组
  重组方法：hstack(), vstack()

    zeros = np.zeros(8)
    ones = np.ones(8)

zerosThenOnes = np.hstack((zeros,ones))   # A 1 by 16 array
zerosAboveOnes = np.vstack((zeros,ones))  # A 2 by 8 array

• PySpark’s DenseVector
  PySpark用来存放vector的对象

Part 4: Python lambda expressions

NB. lambda 表达式的参数意义一定记清楚
例子，lambda拿来给tuples进行排序：

#two ways, same result
swap1 = lambda x: (x[1],x[0])
swap2 = lambda (x0, x1): (x1, x0)
print 'swap1((1, 2)) = {0}'.format(swap1((1, 2)))
print 'swap2((1, 2)) = {0}'.format(swap2((1, 2)))

相关资料：
lambda tutorial
lambda function

Part 5: CTR data download

略