swimhigh的博客

目录 第一章 Perl概述............................................................................................................................10 1.1 从头开始..........................................................................................................................10 1.2 自然语言与人工语言......................................................................................................11 1.2.1 变量语法...............................................................................................................12 1.2.2 单数变量...............................................................................................................13 1.2.3 复数变量...............................................................................................................14 1.2.4 复杂数据结构.......................................................................................................17 1.2.5 简单数据结构.......................................................................................................19 1.2.6 动词.......................................................................................................................21 1.3 一个平均值例子..............................................................................................................22 1.3.1 如何运行...............................................................................................................24 1.4 文件句柄..........................................................................................................................26 1.5 操作符..............................................................................................................................28 1.5.1 双目算术操作符...................................................................................................28 1.5.2 字符串操作符.......................................................................................................28 1.5.3 赋值操作符...........................................................................................................29

非常好的一本SPARK入门书 强烈推荐！ 第1 章　Spark 数据分析导论 ..........................................................................................................1 1.1　Spark 是什么...............................................................................................................................1 1.2　一个大一统的软件栈 .................................................................................................................2 1.2.1　Spark Core ......................................................................................................................2 1.2.2　Spark SQL.......................................................................................................................3 1.2.3　Spark Streaming ..............................................................................................................3 1.2.4　MLlib ..............................................................................................................................3 1.2.5　GraphX............................................................................................................................3 1.2.6　集群管理器 ....................................................................................................................4 1.3　Spark 的用户和用途...................................................................................................................4 1.3.1　数据科学任务 ................................................................................................................4 1.3.2　数据处理应用 ................................................................................................................5 1.4　Spark 简史...................................................................................................................................5 1.5　Spark 的版本和发布...................................................................................................................6 1.6　Spark 的存储层次.......................................................................................................................6 第2 章　Spark 下载与入门 ...............................................................................................................7 2.1　下载Spark...................................................................................................................................7 vi ｜ 目录 2.2　Spark 中Python 和Scala 的shell ..............................................................................................9 2.3　Spark 核心概念简介.................................................................................................................12 2.4　独立应用 ...................................................................................................................................14 2.4.1　初始化SparkContext ...................................................................................................15 2.4.2　构建独立应用 ..............................................................................................................16 2.5　总结 ...........................................................................................................................................19 第3 章　RDD 编程 ............................................................................................................................21 3.1　RDD 基础 .................................................................................................................................21 3.2　创建RDD .................................................................................................................................23 3.3　RDD 操作 .................................................................................................................................24 3.3.1　转化操作 ......................................................................................................................24 3.3.2　行动操作 ......................................................................................................................26 3.3.3　惰性求值 ......................................................................................................................27 3.4　向Spark 传递函数....................................................................................................................27 3.4.1　Python ...........................................................................................................................27 3.4.2　Scala ..............................................................................................................................28 3.4.3　Java ...............................................................................................................................29 3.5　常见的转化操作和行动操作 ...................................................................................................30 3.5.1　基本RDD .....................................................................................................................30 3.5.2　在不同RDD 类型间转换 ............................................................................................37 3.6　持久化( 缓存) .........................................................................................................................39 3.7　总结 ...........................................................................................................................................40 第4 章　键值对操作 .........................................................................................................................41 4.1　动机 ...........................................................................................................................................41 4.2　创建Pair RDD ..........................................................................................................................42 4.3　Pair RDD 的转化操作 ..............................................................................................................42 4.3.1　聚合操作 ......................................................................................................................45 4.3.2　数据分组 ......................................................................................................................49 4.3.3　连接 ..............................................................................................................................50 4.3.4　数据排序 ......................................................................................................................51 4.4　Pair RDD 的行动操作 ..............................................................................................................52 4.5　数据分区（进阶） .....................................................................................................................52 4.5.1　获取RDD 的分区方式 ................................................................................................55 4.5.2　从分区中获益的操作 ..................................................................................................56 4.5.3　影响分区方式的操作 ..................................................................................................57 4.5.4　示例：PageRank ..........................................................................................................57 4.5.5　自定义分区方式 ..........................................................................................................59 4.6　总结 ...........................................................................................................................................61 目录 ｜ vii 第5 章　数据读取与保存 ................................................................................................................63 5.1　动机 ...........................................................................................................................................63 5.2　文件格式 ...................................................................................................................................64 5.2.1　文本文件 ......................................................................................................................64 5.2.2　JSON .............................................................................................................................66 5.2.3　逗号分隔值与制表符分隔值 ......................................................................................68 5.2.4　SequenceFile .................................................................................................................71 5.2.5　对象文件 ......................................................................................................................73 5.2.6　Hadoop 输入输出格式 .................................................................................................73 5.2.7　文件压缩 ......................................................................................................................77 5.3　文件系统 ...................................................................................................................................78 5.3.1　本地/“常规”文件系统 ............................................................................................78 5.3.2　Amazon S3 ....................................................................................................................78 5.3.3　HDFS ............................................................................................................................79 5.4　Spark SQL 中的结构化数据 ....................................................................................................79 5.4.1　Apache Hive .................................................................................................................80 5.4.2　JSON .............................................................................................................................80 5.5　数据库 .......................................................................................................................................81 5.5.1　Java 数据库连接 ..........................................................................................................81 5.5.2　Cassandra ......................................................................................................................82 5.5.3　HBase ............................................................................................................................84 5.5.4　Elasticsearch .................................................................................................................85 5.6　总结 ...........................................................................................................................................86 第6 章　Spark 编程进阶 .................................................................................................................87 6.1　简介 ...........................................................................................................................................87 6.2　累加器 .......................................................................................................................................88 6.2.1　累加器与容错性 ..........................................................................................................90 6.2.2　自定义累加器 ..............................................................................................................91 6.3　广播变量 ...................................................................................................................................91 6.4　基于分区进行操作 ...................................................................................................................94 6.5　与外部程序间的管道 ...............................................................................................................96 6.6　数值RDD 的操作 ....................................................................................................................99 6.7　总结 .........................................................................................................................................100 第7 章　在集群上运行Spark ......................................................................................................101 7.1　简介 .........................................................................................................................................101 7.2　Spark 运行时架构...................................................................................................................101 7.2.1　驱动器节点 ................................................................................................................102 viii ｜ 目录 7.2.2　执行器节点 ................................................................................................................103 7.2.3　集群管理器 ................................................................................................................103 7.2.4　启动一个程序 ............................................................................................................104 7.2.5　小结 ............................................................................................................................104 7.3　使用spark-submit 部署应用 ................................................................................................105 7.4　打包代码与依赖 .....................................................................................................................107 7.4.1　使用Maven 构建的用Java 编写的Spark 应用 .......................................................108 7.4.2　使用sbt 构建的用Scala 编写的Spark 应用 ............................................................109 7.4.3　依赖冲突 .................................................................................................................... 111 7.5　Spark 应用内与应用间调度...................................................................................................111 7.6　集群管理器 .............................................................................................................................112 7.6.1　独立集群管理器 ........................................................................................................112 7.6.2　Hadoop YARN ............................................................................................................115 7.6.3　Apache Mesos .............................................................................................................116 7.6.4　Amazon EC2 ...............................................................................................................117 7.7　选择合适的集群管理器 .........................................................................................................120 7.8　总结 .........................................................................................................................................121 第8 章　Spark 调优与调试 ...........................................................................................................123 8.1　使用SparkConf 配置Spark ...................................................................................................123 8.2　Spark 执行的组成部分：作业、任务和步骤.......................................................................127 8.3　查找信息 .................................................................................................................................131 8.3.1　Spark 网页用户界面 ..................................................................................................131 8.3.2　驱动器进程和执行器进程的日志 ............................................................................134 8.4　关键性能考量 .........................................................................................................................135 8.4.1　并行度 ........................................................................................................................135 8.4.2　序列化格式 ................................................................................................................136 8.4.3　内存管理 ....................................................................................................................137 8.4.4　硬件供给 ....................................................................................................................138 8.5　总结 .........................................................................................................................................139 第9 章　Spark SQL ........................................................................................................................141 9.1　连接Spark SQL ......................................................................................................................142 9.2　在应用中使用Spark SQL ......................................................................................................144 9.2.1　初始化Spark SQL......................................................................................................144 9.2.2　基本查询示例 ............................................................................................................145 9.2.3　SchemaRDD ...............................................................................................................146 9.2.4　缓存 ............................................................................................................................148 9.3　读取和存储数据 .....................................................................................................................149 9.3.1　Apache Hive ...............................................................................................................149 目录 ｜ ix 9.3.2　Parquet ........................................................................................................................150 9.3.3　JSON ...........................................................................................................................150 9.3.4　基于RDD ...................................................................................................................152 9.4　JDBC/ODBC 服务器 ..............................................................................................................153 9.4.1　使用Beeline ...............................................................................................................155 9.4.2　长生命周期的表与查询 ............................................................................................156 9.5　用户自定义函数 .....................................................................................................................156 9.5.1　Spark SQL UDF ..........................................................................................................156 9.5.2　Hive UDF ....................................................................................................................157 9.6　Spark SQL 性能 ......................................................................................................................158 9.7　总结 .........................................................................................................................................159 第10 章　Spark Streaming ..........................................................................................................161 10.1　一个简单的例子 ...................................................................................................................162 10.2　架构与抽象 ...........................................................................................................................164 10.3　转化操作 ...............................................................................................................................167 10.3.1　无状态转化操作 .....................................................................................................167 10.3.2　有状态转化操作 .....................................................................................................169 10.4　输出操作 ...............................................................................................................................173 10.5　输入源 ...................................................................................................................................175 10.5.1　核心数据源 .............................................................................................................175 10.5.2　附加数据源 .............................................................................................................176 10.5.3　多数据源与集群规模 .............................................................................................179 10.6　24/7 不间断运行 ...................................................................................................................180 10.6.1　检查点机制 .............................................................................................................180 10.6.2　驱动器程序容错 .....................................................................................................181 10.6.3　工作节点容错 .........................................................................................................182 10.6.4　接收器容错 .............................................................................................................182 10.6.5　处理保证 .................................................................................................................183 10.7　Streaming 用户界面 .............................................................................................................183 10.8　性能考量 ...............................................................................................................................184 10.8.1　批次和窗口大小 .....................................................................................................184 10.8.2　并行度 .....................................................................................................................184 10.8.3　垃圾回收和内存使用 .............................................................................................185 10.9　总结 .......................................................................................................................................185 第11 章　基于MLlib 的机器学习 ...............................................................................................187 11.1　概述 .......................................................................................................................................187 11.2　系统要求 ...............................................................................................................................188 11.3　机器学习基础 .......................................................................................................................189

portmap工具 ，非常好用的端口映射工具

DOS内在驻留技术，DOS时代的经典，WINDOWS操作系统的多任务由此诞生