SSIS----改进数据流的性能

可以配置数据流任务的下列属性,这些属性都会对性能产生影响:

为缓冲区数据(BufferTempStoragePath 属性)和包含二进制大型对象 (BLOB) 数据的列(BLOBTempStoragePath 属性)指定临时存储位置。默认情况下,这些属性包含 TEMP 和 TMP 环境变量的值。您可能希望指定不同或更快的硬盘驱动器上的其他文件夹来存放临时文件,或将它们分布在多个驱动器上。可以指定多个目录,并用分号来分隔这些目录名。

通过设置 DefaultBufferSize 属性定义任务使用缓冲区的默认大小,并通过设置 DefaultBufferMaxRows 属性来定义每个缓冲区中最大的行数。默认缓冲区大小为 10 MB,最大缓冲区大小为 100 MB。默认最大行数为 10,000。

通过设置 EngineThreads 属性来设置任务在执行过程中可使用的线程数。此属性为数据流引擎提供有关使用线程数的建议。默认值为 5,最小值为 3。但是,不论此属性值为多少,引擎都不会使用超过其所需的线程数。如果需要避免并发问题,引擎所使用的线程数也可能会超过此属性指定的线程数。

指示数据流任务是否以优化模式运行(RunInOptimizedMode 属性)。优化模式会从数据流删除未使用的列、输出和组件,从而提高性能。


注意 注意

可以在 Business Intelligence Development Studio 中的项目级设置同名属性 RunInOptimizedMode,以指示调试过程中数据流任务以优化模式运行。此项目属性在设计时将覆盖数据流任务的 RunInOptimizedMode 属性。


调整缓冲区大小

数据流引擎通过计算一行数据的估计大小来开始调整其缓冲区大小的任务。然后引擎将估计的单行大小与 DefaultBufferMaxRows 值相乘以获得缓冲区大小的初步工作值。


如果该结果大于 DefaultBufferSize 值,引擎将减少行数。


如果该结果小于内部计算的最小缓冲区大小,引擎将增加行数。


如果结果在最小缓冲区大小和 DefaultBufferSize 值之间,引擎将调整缓冲区大小,以尽可能接近估计行大小乘以 DefaultBufferMaxRows 值得出的结果。


当您开始测试数据流任务的性能时,请使用 DefaultBufferSize 和 DefaultBufferMaxRows 的默认值。对数据流任务启用日志记录,并选择 BufferSizeTuning 事件以查看每个缓冲区中包含多少行。

在开始调整缓冲区大小之前,您可以采取的重要改进措施是通过删除不需要的列并配置相应的数据类型来减少每一个数据行的大小。

在有足够的可用内存时,请使用少量的大缓冲区,而不是大量的小缓冲区。换而言之,可以通过减少存放数据所需的缓冲区总数并在一个缓冲区中放置尽可能多的数据行来改善性能。若要确定缓冲区的最佳数目及其大小,请在试验 DefaultBufferSize 值和 DefaultBufferMaxRows 值的同时监视性能以及由 BufferSizeTuning 事件报告的信息。

不要将缓冲区大小增加到开始对磁盘进行分页的点。与未经过优化的缓冲区大小相比,对磁盘进行分页对性能的阻碍作用更大。若要确定是否在进行分页,可监视 Microsoft 管理控制台 (MMC) 的性能管理单元中的“Buffers spooled”性能计数器。


将包配置为支持并行执行

并行执行能改善具有多个物理或逻辑处理器的计算机的性能。为了支持在包中并行执行不同的任务,Integration Services 使用两个属性:MaxConcurrentExecutables 和 EngineThreads。


MaxConcurrentExcecutables 属性

MaxConcurrentExecutables 属性是包本身的一个属性。此属性定义可同时运行的任务的数量。默认值为 -1,表示物理或逻辑处理器的个数加 2。

若要了解此属性的工作原理,可参考一个包含三个数据流任务的示例包。如果将 MaxConcurrentExecutables 设置为 3,则可以同时运行所有三个数据流任务。但是,假定每个数据流任务都具有 10 个源到目标执行树。将 MaxConcurrentExecutables 设置为 3 不能确保每个数据流任务内的执行树都能并行运行。


EngineThreads 属性

EngineThreads 属性是每个数据流任务的属性。此属性定义数据流引擎可以创建和并行运行的线程数。EngineThreads 属性同样适用于数据流引擎为源创建的源线程和该引擎为转换和目标创建的工作线程。因此,将 EngineThreads 设置为 10 表示该引擎可以创建多达 10 个源线程和多达 10 个工作线程。

若要理解此属性的工作原理,可参考包含三个数据流任务的示例包。每个数据流任务都包含 10 个源到目标执行树。如果将每个数据流任务的 EngineThreads 设置为 10,则可以同时运行所有 30 个执行树。


注意 注意

线程不在本主题的讨论范围之内。但是,通用规则是并行运行的线程数不要多于可用的处理器个数。运行的线程数多于可用的处理器个数可能会降低性能,因为此时需要在线程间频繁进行上下文切换。


改进数据流的性能 - Adam - Adams 博客 配置单个数据流组件 


若要配置单个数据流组件以优化性能,可以按照某些通用指导原则进行操作。同时还存在针对各种类型的数据流组件的特定指导原则:源数据流组件、转换数据流组件和目标数据流组件等。


通用指导原则

无论采用何种数据流组件,为了改善性能您应该遵循下面两个通用指导原则:优化查询和避免不必要的字符串。


优化查询

大量数据流组件都将在从源中提取数据时,或在查询操作中创建引用表时使用查询。默认查询使用 SELECT * FROM <表名> 语法。这种类型的查询返回源表中的所有列。在设计时使所有列可用,这意味着可以选择任意列作为查找列、传递列或源列。但是,在选择了要使用的列后,您应该修改查询使其只包括那些所选择的列。删除多余的列可以使包中的数据流更高效,因为列越少则创建的行越小。因为行越小,可以置入一个缓冲区的行就越多,对数据集中所有行进行处理的工作量也就越少。

您可以键入查询或使用查询生成器来构造查询。


注意 注意

在 Business Intelligence Development Studio 中运行包时,SSIS 设计器的“进度”选项卡将列出警告信息。其中包括当源向数据流提供了某个数据列但下游数据流组件却没有使用它时出现的警告信息。您可以使用 RunInOptimizedMode 属性来自动删除这些列。


避免不必要的排序

排序本身是非常缓慢的操作,因此避免不必要的排序可以提高包数据流的性能。

某些情况下,源数据在下游组件使用其之前已经进行了排序。当 SELECT 查询使用 ORDER BY 子句或者数据按排序顺序插入源中时,即出现这种预排序。对于这种预排序的源数据,您可以提供一个提示说明数据已排序,从而避免使用排序转换来满足特定下游转换的排序要求。(例如,合并和合并联接转换要求使用已排序的输入。)若要提供一个提示说明数据已排序,必须执行下面的任务:


将上游数据流组件输出上的 IsSorted 属性设置为 True。


然后指定数据排序所依据的排序键列。


有关详细信息,请参阅如何为合并转换和合并联接转换排序数据

如果必须在数据流中对数据排序,则可以将数据流设计为使用尽可能少的排序操作来提高性能。例如,数据流使用多播转换复制数据集。可以在多播转换运行前对数据集进行一次排序,而不是在转换后再对多个输出进行排序。

有关详细信息,请参阅排序转换合并转换合并联接转换多播转换



OLE DB 源

使用 OLE DB 源从视图中检索数据时,选择“SQL 命令”作为数据访问模式并输入 SELECT 语句。访问数据时,使用 SELECT 语句要比选择“表或视图”作为数据访问模式的执行效果更佳。


转换

使用本节中的建议可以改善聚合、模糊查找、模糊分组、查找、合并联接和渐变维度转换的性能。


聚合转换

聚合转换包括 Keys、KeysScale、CountDistinctKeys 和 CountDistinctScale 属性。通过使用这些属性,使转换能够为转换缓存的数据预先分配转换所需的内存量,从而提高了性能。如果知道要从“分组依据”操作产生的准确或近似组数,则可分别设置 Keys 和 KeysScale 属性。如果知道要从“非重复计数”操作产生的非重复值的准确或近似数量,则可分别设置 CountDistinctKeys 和 CountDistinctScale 属性。

如果需要在数据流中创建多个聚合,应考虑使用一个聚合转换而不是创建多个转换来创建多个聚合。如果聚合是其他聚合的子集,这种方法能够提高性能,因为转换可以优化内部存储,并且只需扫描传入的数据一次。例如,如果聚合使用 GROUP BY 子句和 AVG 聚合,将它们组合成一个转换可以提高性能。但是,在一个聚合转换内执行多个聚合会序列化聚合操作,因此,当必须独立计算多个聚合时,这种方法可能不会改善性能。

有关详细信息,请参阅聚合转换


模糊查找和模糊分组转换

有关如何优化模糊查找和模糊分组转换的性能的信息,请参阅白皮书:Fuzzy Lookup and Fuzzy Grouping in SQL Server Integration Services 2005(SQL Server Integration Services 2005 中的模糊查找和模糊分组转换)。


查找转换

通过输入仅查找所需列的 SELECT 语句,最小化内存中引用数据的大小。这种方法优于选择整个表或视图,因为后者将返回大量不必要的数据。


合并联接转换

合并联接转换包括 MaxBuffersPerInput 属性,该属性指定可以同时为每个输入处于活动状态的最大缓冲区数。可以使用此属性来优化缓冲区所使用的内存量,并由此优化转换的性能。缓冲区数越大,转换所使用的内存越多,性能越好。MaxBuffersPerInput 的默认值是 5,这是适合大多数工作情况的缓冲区数。若要优化性能,可能需要尝试使用稍有不同的缓冲区数,例如,4 或 6。如果可能,应当避免使用非常小的缓冲区数。例如,将 MaxBuffersPerInput 设置为 1 而不是 5,则可能对性能造成很大影响。另外,不应将 MaxBuffersPerInput 设置为 0 或更小的值。此值范围表示没有中止发生,并且由于数据负载和可用内存数量,包可能无法完成。

若要避免死锁,合并联接转换可能临时增加它所使用的缓冲区数,使其超过 MaxBuffersPerInput 的值。死锁条件消除之后,MaxBuffersPerInput 将返回它的配置值。

有关详细信息,请参阅合并联接转换


渐变维度转换

渐变维度向导和渐变维度转换是能满足大多数用户需要的通用工具。但是,该向导生成的数据流未针对性能进行优化。

通常,渐变维度转换中最慢的组件是一次对单行执行 UPDATE 的 OLE DB 命令转换。因此,改善渐变维度转换性能最有效的方法是替换 OLE DB 命令转换。可以用目标组件来替换这些转换,目标组件将要更新的所有行保存到一个临时表中。然后,可以添加执行 SQL 任务,该任务同时对所有行执行基于单集的 Transact-SQL UPDATE。

高级用户可以为渐变维度处理设计自定义数据流,此数据流将针对大型维度进行优化。有关此方法的讨论和示例,请参阅白皮书 Project REAL: Business Intelligence ETL Design Practices(Project REAL:Business Intelligence ETL 设计实践)中的章节 "Unique dimension scenario"(唯一维度方案)。


目标

若要改善目标的性能,请考虑使用 SQL Server 目标并测试目标的性能。


SQL Server 目标

当包将数据加载到同一计算机上的 SQL Server 实例时,请使用 SQL Server 目标。此目标针对高速大容量加载进行优化。


测试目标的性能

您可能会发现将数据保存到目标时所花的时间比预期的要长。为了确定速度缓慢是否是由于目标处理数据的能力不足造成的,可以暂时将目标替换为行计数转换。如果吞吐量显著提高,很可能是加载数据的目标导致速度减缓。


改进数据流的性能 - Adam - Adams 博客 监视包的性能 


Integration Services 包括可以用于监视包性能的工具和功能。例如,日志记录可以捕获有关包的运行时信息,性能计数器则可以监视数据流引擎。请使用下列建议确定包中对性能影响最大的部分。


查看“进度”选项卡上的信息

SSIS 设计器提供有关在 Business Intelligence Development Studio 中运行包时控制流和数据流的信息。“进度”选项卡按执行顺序列出任务和容器,而且还包括每个任务和容器及包自身的开始时间和结束时间、警告以及错误消息。它还按执行顺序列出数据流组件并包括进度信息(显示为完成百分比)和处理的行数。

若要允许或禁止在“进度”选项卡上显示消息,请在 SSIS 菜单上切换“调试进度报告”选项。禁用进度报告有助于在 BI Development Studio 中运行复杂包时改进性能。


配置包中的日志记录

Integration Services 包括各种日志提供程序,这些提供程序允许包在运行时将信息记录到不同类型的文件中或记录到 SQL Server 中。您可以为包和各个包对象(例如任务和容器)启用日志项。Integration Services 包括各种任务和容器,每个任务和容器都具有其自己的一组说明性日志项。例如,包括执行 SQL 任务的包可以写入一个日志项,列出该任务执行的 SQL 语句(包括该语句的参数值)。

这些日志项包括诸如包和包对象的开始时间和完成时间这样的信息,从而可以确定运行缓慢的任务和容器。有关详细信息,请参阅记录包执行的日志在包中实现日志记录日志记录的自定义消息


配置数据流任务的日志记录

数据流任务提供了许多可用于监视和调整性能的自定义日志项。例如,您可以监视可能会导致内存泄漏的组件,或者跟踪特定组件运行所用的时间。有关这些自定义日志项的列表和日志记录输出示例,请参阅数据流任务


使用 PipelineComponentTime 事件

最有用的自定义日志项可能是 PipelineComponentTime 事件。该日志项报告数据流中的每个组件执行五个主要处理步骤中的每个步骤所用的毫秒数。下表对这些处理步骤进行了说明。Integration Services 开发人员应将这些步骤视为 PipelineComponent 的主要方法。



步骤

说明

Validate

该组件查看有效的属性值和配置设置。

PreExecute

该组件在开始处理数据行之前执行一次性处理。

PostExecute

该组件在处理所有数据行之后执行一次性处理。

ProcessInput

转换或目标组件处理由上游源或转换传递的传入数据行。

PrimeOutput

源或转换组件填充数据缓冲区,以传递给下游转换或目标组件。


启用 PipelineComponentTime 事件时,Integration Services 将针对每个组件执行的各处理步骤记录一则消息。以下日志项显示 Integration Services CalculatedColumns 包示例记录的消息的子集。

“Calculate LineItemTotalCost”组件 (3522) 在 ProcessInput 中耗时 356 毫秒。

“Sum Quantity and LineItemTotalCost”组件 (3619) 在 ProcessInput 中耗时 79 毫秒。

“Calculate Average Cost”组件 (3662) 在 ProcessInput 中耗时 16 毫秒。

“Sort by ProductID”组件 (3717) 在 ProcessInput 中耗时 125 毫秒。

“Load Data”组件 (3773) 在 ProcessInput 中耗时 0 毫秒。

“Extract Data”组件 (3869) 在“OLE DB Source Output”输出 (3879) 上的 PrimeOutput 填充缓冲区中耗时 688 毫秒。

“Sum Quantity and LineItemTotalCost”组件 (3619) 在“Aggregate Output 1”输出 (3621) 上的 PrimeOutput 填充缓冲区中耗时 141 毫秒。

“Sort by ProductID”组件 (3717) 在“Sort Output”输出 (3719) 上的 PrimeOutput 填充缓冲区中耗时 16 毫秒。

下列日志项显示数据流任务在下列步骤中消耗了大多数时间,如下所示(按降序排序):


名为“Extract Data”的 OLE DB 源在加载数据期间耗时 688 毫秒。


名为“Calculate LineItemTotalCost”的派生列转换在对传入行执行计算期间耗时 356 毫秒。


名为“Sum Quantity and LineItemTotalCost”的聚合转换在执行计算和将数据传递到下一转换期间共耗时 220 毫秒,其中有 141 毫秒用于 PrimeOutput,有 79 毫秒用于 ProcessInput。


监视数据流引擎的性能

Integration Services 包括一组性能计数器,用于监视数据流引擎的性能。例如,您可以跟踪所有缓冲区使用的内存总量(以字节为单位),并检查组件是否内存不足。缓冲区是组件用于存储数据的内存块。有关详细信息,请参阅监视数据流引擎的性能

  • 1
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Contents About the Author...............................................................................................xix About the Technical Reviewer and Contributing Author.................xxi Chapter1 Apache and the Internet..............................................1 Apache: The Anatomy of a Web Server.........................................................1 The Apache Source .............................................................................................1 The Apache License............................................................................................1 Support for Apache.............................................................................................2 How Apache Works..............................................................................................3 The Hypertext Transfer Protocol..................................................................7 HTTP Requests and Responses..........................................................................7 HTTP Headers...................................................................................................12 Networking and TCP/IP......................................................................................13 Definitions.........................................................................................................13 Packets and Encapsulation...............................................................................14 ACKs, NAKs, and Other Messages....................................................................15 The TCP/IP Network Model.............................................................................16 Non-IP Protocols...............................................................................................19 IP Addresses and Network Classes...................................................................19 Special IP Addresses..........................................................................................20 Netmasks and Routing......................................................................................21 Web Services: Well-Known Ports......................................................................23 Internet Daemon: The Networking Super Server...........................................24 The Future: IPv6................................................................................................25 Networking Tools...............................................................................................26 Server Hardware...................................................................................................29 Supported Platforms.........................................................................................29 Basic Server Requirements...............................................................................30 Memory..............................................................................................................31 Network Interface..............................................................................................32 Internet Connection.........................................................................................32 Hard Disk and Controller.................................................................................33 Operating System Checklist.............................................................................33 Redundancy and Backup..................................................................................34 Specific Hardware Solutions............................................................................35 Get Someone Else to Do It.............................................................................36 Summary....................................................................................................................36 v 3006_Ch00_CMP2 12/14/03 8:56 AM Page v Chapter 2 Getting Started with Apache.................................37 Installing Apache..............................................................................................38 Getting Apache..................................................................................................38 Installing Apache from Binary Distribution....................................................39 Installing Apache from Source.........................................................................41 Installing Apache from Prebuilt Packages.......................................................41 Installing Apache by Hand...............................................................................45 Upgrading Apache.............................................................................................47 Other Issues.......................................................................................................49 Basic Configuration..........................................................................................50 Decisions............................................................................................................50 Introducing the Master Configuration File.....................................................55 Other Basic Configuration Directives..............................................................56 Starting, Stopping, and Restarting the Server.................................57 Starting Apache on Unix...................................................................................58 Starting Apache on Windows...........................................................................59 Invocation Options...........................................................................................60 Restarting the Server.........................................................................................73 Stopping the Server...........................................................................................75 Starting the Server Automatically....................................................................76 Testing the Server............................................................................................81 Testing with a Browser......................................................................................82 Testing from the Command Line or a Terminal Program..............................82 Testing the Server Configuration Without Starting It.....................................85 Getting the Server Status from the Command Line.......................................86 Using Graphical Configuration Tools.......................................................86 Comanche..........................................................................................................87 TkApache...........................................................................................................91 LinuxConf..........................................................................................................91 Webmin..............................................................................................................91 ApacheConf.......................................................................................................97 Other Configuration Tools................................................................................99 Summary..................................................................................................................100 Chapter 3 Building Apache the Way You Want It...........101 Why Build Apache Yourself?.........................................................................101 Verifying the Apache Source Archive.............................................................103 Building Apache from Source......................................................................105 Configuring and Building Apache.................................................................106 Determining Which Modules to Include.......................................................111 Building Apache As a Dynamic Server..........................................................116 Contents vi 3006_Ch00_CMP2 12/14/03 8:56 AM Page vi Changing the Module Order (Apache 1.3)....................................................118 Checking the Generated Configuration........................................................120 Building Apache from Source As an RPM (Apache 2)..................................122 Advanced Configuration.................................................................................124 Configuring Apache’s Layout..........................................................................124 Choosing a Layout Scheme............................................................................124 Choosing a Multiprocessing Module (Apache 2)..........................................132 Rules (Apache 1.3)...........................................................................................135 Building Apache with suExec support...........................................................137 Configuring Apache’s Supporting Files and Scripts.....................................139 Configuring Apache 2 for Cross-Platform Builds.........................................140 Configuring Apache for Production or Debug Builds..................................142 Configuring Apache for Binary Distribution.................................................143 Configuring Apache’s Library and Include Paths..........................................143 Configuring the Build Environment.........................................................144 Building Modules with configure and apxs..........................................146 Adding Third-Party Modules with configure................................................146 Building Modules with apxs...........................................................................148 Installing Modules with apxs..........................................................................150 Generating Module Templates with apxs......................................................151 Overriding apxs Defaults and Using apxs in makefiles................................152 Summary..................................................................................................................153 Chapter 4 Configuring Apache the Way You Want It...155 Where Apache Looks for Its Configuration..........................................155 Configuration File Syntax...............................................................................156 Configuration for Virtual Hosts......................................................................156 Including Multiple Configuration Files.........................................................157 Per-Directory Configuration..........................................................................159 Conditional Configuration.............................................................................160 How Apache Structures Its Configuration............................................163 Apache’s Container Directives........................................................................164 Directive Types and Locations.......................................................................168 Where Directives Can Go................................................................................171 Container Scope and Nesting.........................................................................172 How Apache Combines Containers and Their Contents.............................174 Legality of Directives in Containers...............................................................175 Options and Overrides....................................................................................176 Enabling and Disabling Features with Options............................................176 Overriding Directives with Per-Directory Configuration.............................179 Contents vii 3006_Ch00_CMP2 12/14/03 8:56 AM Page vii Restricting Access with allow and deny..............................................182 Controlling Access by Name...........................................................................183 Controlling Access by IP Address...................................................................184 Controlling Subnet Access by Network and Netmask..................................185 Controlling Access by HTTP Header.............................................................186 Combining Host-Based Access with User Authentication...........................187 Overriding Host-Based Access.......................................................................188 Directory Listings..........................................................................................188 Enabling and Disabling Directory Indices....................................................189 How mod_autoindex Generates the HTML Page.........................................190 Controlling Which Files Are Seen with IndexIgnore.....................................196 Controlling the Sort Order..............................................................................197 Assigning Icons................................................................................................199 Assigning Descriptions...................................................................................202 Apache’s Environment......................................................................................203 Setting, Unsetting, and Passing Variables from the Shell.............................204 Setting Variables Conditionally......................................................................205 Special Browser Variables...............................................................................207 Detecting Robots with BrowserMatch...........................................................209 Passing Variables to CGI.................................................................................209 Conditional Access Control............................................................................210 Caveats with SetEnvIf vs. SetEnv....................................................................210 Setting Variables with mod_rewrite...............................................................211 Controlling Request and Response Headers..........................................211 Setting Custom Response Headers................................................................213 Setting Custom Request Headers...................................................................215 Inserting Dynamic Values into Headers........................................................216 Setting Custom Headers Conditionally.........................................................217 Retrieving Response Headers from Metadata Files......................................217 Setting Expiry Times.......................................................................................219 Sending Content As-Is....................................................................................222 Controlling the Server Identification Header.................................223 Sending a Content Digest.............................................................................224 Handling the Neighbors.................................................................................225 Controlling Robots with robots.txt................................................................226 Controlling Robots in HTML..........................................................................227 Controlling Robots with Access Control........................................................227 Attracting Robots.............................................................................................228 Making Sure Robots Index the Right Information........................................228 Known Robots, Bad Robots, and Further Reading.......................................229 Summary..................................................................................................................229 Contents viii 3006_Ch00_CMP2 12/14/03 8:56 AM Page viii Chapter 5 Deciding What the Client Needs........................231 Content Handling and Negotiation...........................................................231 File Types.........................................................................................................232 File Encoding...................................................................................................236 File Languages.................................................................................................243 File Character Sets...........................................................................................245 Handling URLs with Extra Path Information................................................247 Content Negotiation.......................................................................................248 Content Negotiation with MultiViews...........................................................250 File Permutations and Valid URLs with MultiViews.....................................256 Magic MIME Types..........................................................................................260 Error and Response Handling......................................................................264 How Apache Handles Errors...........................................................................265 Error and Response Codes.............................................................................265 The ErrorDocument Directive.......................................................................266 Limitations of ErrorDocument......................................................................270 Aliases and Redirection...............................................................................271 Aliases and Script Aliases................................................................................271 Redirections.....................................................................................................273 Rewriting URLs with mod_rewrite.................................................................277 Server-Side Image Maps.................................................................................300 Matching Misspelled URLS............................................................................305 Summary..................................................................................................................306 Chapter 6 Delivering Dynamic Content..................................307 Server-Side Includes......................................................................................308 Enabling SSI.....................................................................................................309 Format of SSI Commands...............................................................................311 The SSI Command Set....................................................................................312 SSI Variables.....................................................................................................312 Passing Trailing Path Information to SSIs (and Other Dynamic Documents).................................................................315 Setting the Date and Error Format.................................................................316 Templating with SSIs.......................................................................................317 Caching Server-Parsed Documents...............................................................319 Identifying Server-Parsed Documents by Execute Permission...................320 CGI: The Common Gateway Interface.........................................................321 CGI and the Environment..............................................................................321 Configuring Apache to Recognize CGI Scripts.............................................323 Setting Up a CGI Directory with ExecCGI: A Simple Way............................327 Triggering CGI Scripts on Events...................................................................330 Contents ix 3006_Ch00_CMP2 12/14/03 8:56 AM Page ix ISINDEX-Style CGI Scripts and Command Line Arguments................332 Writing and Debugging CGI Scripts.........................................................333 A Minimal CGI Script......................................................................................333 Interactive Scripts: A Simple Form................................................................337 Adding Headers...............................................................................................338 Debugging CGI Scripts....................................................................................339 Setting the CGI Daemon Socket.....................................................................345 Limiting CGI Resource Usage.........................................................................346 Actions, Handlers, and Filters................................................................347 Handlers...........................................................................................................348 Filters................................................................................................................354 Dynamic Content and Security....................................................................363 CGI Security Issues..........................................................................................363 Security Advice on the Web............................................................................364 Security Issues with Apache CGI Configuration...........................................364 An Example of an Insecure CGI Script..........................................................365 Known Insecure CGI Scripts...........................................................................370 CGI Wrappers...................................................................................................370 Security Checklist............................................................................................380 Inventing a Better CGI Script with FastCGI......................................381 Summary..................................................................................................................403 Chapter 7 Hosting More Than One Web Site........................405 Implementing User Directories with UserDir......................................406 Enabling and Disabling Specific Users..........................................................407 Redirecting Users to Other Servers................................................................408 Alternative Ways to Implement User Directories.........................................409 Separate Servers..............................................................................................410 Restricting Apache’s Field of View..................................................................411 Specifying Different Configurations and Server Roots................................412 Starting Separate Servers from the Same Configuration.............................412 Sharing External Configuration Files.............................................................413 IP-Based Virtual Hosting.............................................................................414 Multiple IPs, Separate Networks, and Virtual Interfaces..............................415 Configuring What Apache Listens To.............................................................416 Defining IP-Based Virtual Hosts.....................................................................418 Virtual Hosts and the Server-Level Configuration........................................421 Specifying Virtual Host User Privileges..........................................................422 Excluded Directives.........................................................................................426 Default Virtual Hosts....................................................................................427 Contents x 3006_Ch00_CMP2 12/14/03 8:56 AM Page x Name-Based Virtual Hosting.........................................................................428 Defining Named Virtual Hosts.......................................................................428 Server Names and Aliases...............................................................................430 Defining a Default Host for Name-Based Virtual Hosting...........................430 Mixing IP-Based and Name-Based Hosting..................................................431 Issues Affecting Virtual Hosting...........................................................434 Log Files and File Handles..............................................................................434 Virtual Hosts and Server Security..................................................................436 Secure HTTP and Virtual Hosts......................................................................437 Handling HTTP/1.0 Clients with Name-Based Virtual Hosts......................439 Dynamic Virtual Hosting...............................................................................441 Mass Hosting with Virtual-Host Aliases........................................................441 Mapping Hostnames Dynamically with mod_rewrite.................................448 Generating On the Fly and Included Configuration Files with mod_perl..449 Summary..................................................................................................................455 Chapter 8 Improving Apache’s Performance........................457 Apache’s Performance Directives..............................................................458 Configuring MPMs: Processes and Threads..................................................459 Network and IP-Related Performance Directives.........................................470 HTTP-Related Performance Directives.........................................................472 HTTP Limit Directives....................................................................................475 Configuring Apache for Better Performance........................................477 Directives That Affect Performance...............................................................477 Additional Directives for Tuning Performance.............................................482 Benchmarking Apache’s Performance.........................................................490 Benchmarking Apache with ab......................................................................490 Benchmarking Apache with gprof.................................................................495 External Benchmarking Tools........................................................................496 Benchmarking Strategy and Pitfalls...............................................................496 A Performance Checklist...............................................................................497 Proxying................................................................................................................498 Installing and Enabling Proxy Services..........................................................498 Normal Proxy Operation.................................................................................499 Configuring Apache As a Proxy......................................................................500 URL Matching with Directory Containers....................................................502 Blocking Sites via the Proxy............................................................................504 Localizing Remote URLs and Hiding Servers from View.............................504 Relaying Requests to Remote Proxies............................................................508 Proxy Chains and the Via Header...................................................................509 Proxies and Intranets......................................................................................512 Handling Errors...............................................................................................512 Contents xi 3006_Ch00_CMP2 12/14/03 8:56 AM Page xi Timing Out Proxy Requests............................................................................514 Tunneling Other Protocols.............................................................................514 Tuning Proxy Operations................................................................................515 Squid: A High-Performance Proxy Alternative.............................................516 Caching..................................................................................................................516 Enabling Caching............................................................................................516 File-Based Caching.........................................................................................517 In-Memory Caching (Apache 2 Only)............................................................520 Coordinating Memory-Based and Disk-Based Caches................................522 General Cache Configuration.........................................................................522 Maintaining Good Relations with External Caches......................................527 Fault Tolerance and Clustering................................................................529 Backup Server via Redirected Secondary DNS.............................................530 Load Sharing with Round-Robin DNS...........................................................531 Backup Server via Floating IP Address..........................................................531 Hardware Load Balancing..............................................................................532 Clustering with Apache...................................................................................533 Other Clustering Solutions.............................................................................536 Summary..................................................................................................................537 Chapter 9 Monitoring Apache.........................................................539 Logs and Logging..............................................................................................539 Log Files and Security.....................................................................................540 The Error Log...................................................................................................540 Setting the Log Level.......................................................................................541 Logging Errors to the System Log..................................................................542 Transfer Logs...................................................................................................544 Driving Applications Through Logs...............................................................554 Log Rotation....................................................................................................556 Lies, Logs, and Statistics.........................................................................560 What You Can’t Find Out from Logs...............................................................560 Analog: A Log Analyzer...................................................................................561 Server Information..........................................................................................577 Server Status....................................................................................................578 Server Info........................................................................................................581 Securing Access to Server Information.........................................................582 User Tracking.....................................................................................................583 Alternatives to User Tracking.........................................................................584 Cookie Tracking with mod_usertrack............................................................584 URL Tracking with mod_session....................................................................589 Other Session Tracking Options.....................................................................594 Summary..................................................................................................................595 Contents xii 3006_Ch00_CMP2 12/14/03 8:56 AM Page xii Chapter 10Securing Apache..............................................................597 User Authentication........................................................................................597 Apache Authentication Modules...................................................................598 Authentication Configuration Requirements...............................................599 Using Authentication Directives in .htaccess...............................................601 Basic Authentication.......................................................................................601 Digest Authentication.....................................................................................603 Anonymous Authentication...........................................................................606 Setting Up User Information..........................................................................606 Specifying User Requirements.......................................................................614 LDAP Authentication......................................................................................617 Using Multiple Authentication Schemes.......................................................624 Combining User- and Host-Based Authentication......................................626 Securing Basic Authentication with SSL.......................................................627 SSL and Apache...................................................................................................627 Downloading OpenSSL and ModSSL............................................................628 Building and Installing the OpenSSL Library...............................................629 Building and Installing mod_ssl for Apache 2..............................................633 Building and Installing mod_ssl for Apache 1.3...........................................633 Basic SSL Configuration.................................................................................637 Installing a Private Key....................................................................................639 Creating a Certificate Signing Request and Temporary Certificate.............640 Getting a Signed Certificate............................................................................642 Advanced SSL Configuration.........................................................................644 Server-Level Configuration............................................................................644 Client Certification..........................................................................................657 Using Client Certification with User Authentication..................659 SSL and Logging..............................................................................................660 SSL Environment Variables and CGI.............................................................662 SSL and Virtual Hosts......................................................................................666 Advanced Features..........................................................................................668 Summary..................................................................................................................671 Chapter 11Improving Web Server Security..........................673 Apache Features.................................................................................................673 Unwanted Files................................................................................................674 Automatic Directory Indices..........................................................................674 Symbolic Links................................................................................................675 Server-Side Includes.......................................................................................676 ISINDEX-Style CGI Scripts.............................................................................677 Server Tokens...................................................................................................677 Contents xiii 3006_Ch00_CMP2 12/14/03 8:56 AM Page xiii File Permissions..............................................................................................678 Viewing Server Information with mod_info..........................................679 Restricting Server Privileges..................................................................679 Restricting Access by Hostname and IP Address...............................680 Other Server Security Measures................................................................682 Dedicated Server..............................................................................................682 File Integrity...................................................................................................683 md5sum...........................................................................................................684 Tripwire............................................................................................................685 Hardening the Server......................................................................................686 Minimizing Services........................................................................................686 Port Scanning with nmap ..............................................................................688 Probing with Nessus.......................................................................................689 Hardening Windows 2000 and XP..................................................................689 Disabling Network Services.........................................................................690 File Transfer Protocol (FTP)............................................................................690 telnet................................................................................................................690 rlogin, rsh, rexec, rcp.......................................................................................690 Network Filesystem (NFS)..............................................................................690 sendmail/Other Mail Transport Agents (MTAs)...........................................691 Restricting Services with TCP Wrappers........................................................691 Security Fixes, Alerts, and Online Resources.................................693 The WWW Security FAQ..................................................................................693 The BugTraQ Mailing List and Archive..........................................................693 Operating System Newsletters.......................................................................693 Package and Module Notification..................................................................694 Removing Important Data from the Server............................................694 Enabling Secure Logins with SSH..............................................................694 Building and Installing OpenSSH..................................................................695 Authentication Strategies...............................................................................698 Configuring SSH..............................................................................................699 Testing SSH......................................................................................................702 Expanding SSH to Authenticate Users..........................................................703 Secure Server Backups with Rsync and SSH.................................................704 Forwarding Client Connections to Server Applications...............................705 Firewalls and Multifacing Servers.........................................................706 Types of Firewall..............................................................................................706 Designing the Network Topology...................................................................707 Running Apache Under a Virtual chroot Root Directory................709 What chroot Is.................................................................................................709 What chroot Isn’t.............................................................................................710 Setting Up Apache for chroot Operation.......................................................711 Contents xiv 3006_Ch00_CMP2 12/14/03 8:56 AM Page xiv Server Security Checklist...........................................................................723 Avoid Root Services.........................................................................................723 Maintain Logs Properly..................................................................................723 Keep It Simple..................................................................................................724 Block Abusive Clients......................................................................................724 Have an Effective Backup and Restore Process............................................725 Plan for High Availability, Capacity, and Disaster Recovery........................725 Monitor the Server..........................................................................................725 Take Care with Information Flow...................................................................726 Choose an Effective robots.txt Policy............................................................726 Summary..................................................................................................................726 Chapter 12Extending Apache............................................................727 WebDAV....................................................................................................................727 Adding WebDAV to Apache.............................................................................728 The WebDAV Protocol.....................................................................................729 Configuring Apache for WebDAV...................................................................731 Restricting Options and Disabling Overrides...............................................734 WebDAV and Virtual Hosts.............................................................................735 Configuring the DAV Lock Time.....................................................................735 Limitations of File-Based Repositories..........................................................736 Protecting WebDAV Servers............................................................................737 More Advanced Configurations.....................................................................737 Cooperating with CGI and Other Content Handlers....................................740 ISAPI......................................................................................................................741 Supported ISAPI Support Functions.............................................................742 Configuring ISAPI Extensions........................................................................743 Setting the Maximum Initial Request Data Size...........................................744 Logging ISAPI Extensions...............................................................................745 Preloading and Caching ISAPI Extensions....................................................746 Handling Asynchronous ISAPI Extensions...................................................746 Perl.........................................................................................................................746 Building and Installing mod_perl..................................................................748 Migrating mod_perl from Apache 1.3 to Apache 2.......................................755 Configuring and Implementing Perl Handlers.............................................758 Configuring and Implementing Perl Filters..................................................771 Warnings, Taint Mode, and Debugging.........................................................772 Managing Perl Threads in mod_perl 2...........................................................774 Initializing Modules at Startup.......................................................................779 Restarting mod_perl and Auto-Reloading Modules.....................................780 Creating a mod_perl Status Page...................................................................782 Running CGI Scripts Under mod_perl..........................................................782 Contents xv 3006_Ch00_CMP2 12/14/03 8:56 AM Page xv CGI Caveats......................................................................................................785 Passing Variables to Perl Handlers.................................................................787 Using mod_perl with Server-Side Includes...................................................788 Embedding Perl in HTML...............................................................................789 Embedding Perl in Apache’s Configuration..................................................794 PHP...........................................................................................................................795 Installing PHP..................................................................................................796 Getting the PHP source...................................................................................796 Configuring Apache to Work with PHP..........................................................802 Configuring PHP.............................................................................................803 Testing PHP with Apache................................................................................807 Tomcat/Java.........................................................................................................807 So What Is Tomcat?..........................................................................................807 Installation.......................................................................................................808 Tomcat Configuration.....................................................................................813 mod_jk.............................................................................................................818 Mod_python....................................................................................................829 mod_ruby.........................................................................................................835 Summary..................................................................................................................839 Index....................................................................................................................843
数据来源:中经数据库 主要指标110多个(全部都是纯粹的 市辖区 指标),大致是: GDP GDP增速 第一产业增加值占GDP比重 第二产业增加值占GDP比重 第三产业增加值占GDP比重 人均GDP 社会消费品零售总额 固定资产投资(不含农户) 新设外商投资企业数_外商直接投资 实际利用外资金额(美元) 一般公共预算收入 一般公共预算支出 一般公共预算支出_教育 一般公共预算支出_科学技术 金融机构人民币各项存款余额_个人储蓄存款 金融机构人民币各项存款余额 金融机构人民币各项贷款余额 规模以上工业企业单位数 规模以上工业企业单位数_内资企业 规模以上工业企业单位数_港澳台商投资企业 规模以上工业企业单位数_外商投资企业 规模以上工业总产值 规模以上工业总产值_内资企业 规模以上工业总产值_港澳台商投资企业 规模以上工业总产值_外商投资企业 规模以上工业企业流动资产合计 规模以上工业企业固定资产合计 规模以上工业企业利润总额 规模以上工业企业应交增值税 规模以上工业企业主营业务税金及附加 户籍人口数 年均户籍人口数 户籍人口自然增长率 第一产业就业人员占全部城镇单位就业人员比重 第二产业就业人员占全部城镇单位就业人员比重 第三产业就业人员占全部城镇单位就业人员比重 城镇非私营单位就业人员数 城镇非私营单位就业人员数_第一产业 城镇非私营单位就业人员数_第二产业 城镇非私营单位就业人员数_第三产业 城镇非私营单位就业人员数_农、林、牧、渔业 城镇非私营单位就业人员数_采矿业 城镇非私营单位就业人员数_制造业 城镇非私营单位就业人员数_电力、热力、燃气及水生产和供应业 城镇非私营单位就业人员数_建筑业 城镇非私营单位就业人员数_批发和零售业 城镇非私营单位就业人员数_交通运输、仓储和邮政业 城镇非私营单位就业人员数_住宿和餐饮业 城镇非私营单位就业人员数_信息传输、软件和信息技术服务业 城镇非私营单位就业人员数_金融业 城镇非私营单位就业人员数_房地产业 城镇非私营单位就业人员数_租赁和商务服务业 城镇非私营单位就业人员数_科学研究和技术服务业 城镇非私营单位就业人员数_水利、环境和公共设施管理业 城镇非私营单位就业人员数_居民服务、修理和其他服务业 城镇非私营单位就业人员数_教育 城镇非私营单位就业人员数_卫生和社会工作 城镇非私营单位就业人员数_文化、体育和娱乐业 城镇非私营单位就业人员数_公共管理、社会保障和社会组织 城镇非私营单位在岗职工平均人数 城镇就业人员数_私营企业和个体 城镇非私营单位在岗职工工资总额 城镇非私营单位在岗职工平均工资 城镇登记失业人员数 建成区面积 建设用地面积 建设用地面积_居住用地 液化石油气供气总量 液化石油气供气总量_居民家庭 人工煤气、天然气供气总量 人工煤气、天然气供气总量_居民家庭 液化石油气用气人口 人工煤气、天然气用气人口 城市公共汽电车运营车辆数 城市出租汽车运营车辆数 城市公共汽电车客运总量 道路面积 排水管道长度 建成区绿化覆盖面积 建成区绿化覆盖率 绿地面积 公园绿地面积 维护建设资金支出 土地面积 生活用水供水量 供水总量 全社会用电量 城乡居民生活用电量 工业生产用电量 房地产开发投资 房地产开发投资_住宅 限额以上批发和零售业法人单位数 限额以上批发和零售业商品销售总额 普通中学学校数 中等职业教育学校数 普通小学学校数 普通高等学校专任教师数 普通中学专任教师数 中等职业教育专任教师数 普通小学专任教师数 普通高等学校在校生数 普通中学在校生数 中等职业教育在校生数 普通小学在校生数 电视节目综合人口覆盖率 公共图书馆总藏量_图书 医疗卫生机构数_医院和卫生院 卫生人员数_执业(助理)医师 医疗卫生机构床位数_医院和卫生院 城镇职工基本养老保险参保人数 职工基本医疗保险参保人数 失业保险参保人数

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值