Why use Store procedure

  第一:提高性能

        当存储过程被创建,它要经过以下几步,第一步 ,它所包含的T-SQL语句将被分析和解析,并被存储.当第一次被执行时,它将被调出并被优化.SQLServer会根据statistics自动选择相应的优化策略。
此后查询计划将被存储在高速缓存中,以利于将来使用 .由于不用被重新编译,所以可以大大提高效率。
 
第二: 减少网络流量
        你可能使用 T-SQL语句来对表执行插入操作。但是如果我们创建一个存储过程来进行这样操作的话,每次插入的时候只要传输存储过程名。参数和这些参数的数值。当这些操作非常频繁时我们将会发现使用存储过程可以减少额外的网络传输。这在我们使用Internet时进行传输时非常有用。
比较以下两个语句:
INSERT INTO EmployeeTerritories (EmployeeID, TerritoryID) VALUES ( 3 , 12345 )
Ins_EmployeeTerritories @empId = 3 ,@terrId = 12345
        如果一个图片类型数据被上传或者下载应该如何处理哪??对于二进制数据类型,例如图片、声音等等,将会以二进制值的形式被发送。当使用 T-SQL内联的时候,这些二进制数据类型的数据,将会被转换成字符串,这也会使我们发送的实时查询数据大小加倍。
        第一个语句有 74个字符,第二个有46个字符,相比而言网络传输量减少了37.84%,如果我们这个插入语句中包括有更多要插入数据的列,并且每天被执行10,000次,将会学杂费280K左右的带宽。
 
第三: 减少注入式攻击
        3.1易受注入式攻击的代码:
// DANGER! User input used to generate database query
string  sql  =  String.Format ( " select count (*) from users where username='{0}' and cast (password as varbinary)=cast ('{1}' as varbinary) " , username, password); 
SqlCommand command 
=   new  SqlCommand (sql, connection); 
int  count  =  ( int ) command.ExecuteScalar (); 
       
        3.2优化过的代码
下面的代码被注入式攻击的机率要少些
// BETTER: Input passed to parameterized command
SqlCommand command  =   new  SqlCommand 
(
" select count (*) from users where username=@username and cast (password as varbinary)=cast (@password as varbinary) " ,  connection); 
command.Parameters.Add (
" @username " , SqlDbType.VarChar).Value  =  username; 
command.Parameters.Add (
" @password " , SqlDbType.VarChar).Value  =  password; 
int  count  =  ( int ) command.ExecuteScalar (); 
       3.3使用存储过程来减少注入攻击
通过存储过程来执行效果会更好
// BEST: Input passed to stored procedure 
SqlCommand command  =   new  SqlCommand ( " proc_IsUserValid " , connection); 
command.CommandType 
=  CommandType.StoredProcedure; 
command.Parameters.Add (
" @username " , SqlDbType.VarChar).Value  =  username; 
command.Parameters.Add (
" @password " , SqlDbType.VarChar).Value  =  password; 
command.Parameters.Add (
" @return " , SqlDbType.Int).Direction  =  ParameterDirection.ReturnValue; 
int  count  =  ( int ) command.ExecuteScalar (); 

        所以我们应该尽量在我们的应用系统中使用存储过程
 
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 2
    评论
Usage: ora [-u user] [-i instance#] [] General -u user/pass use USER/PASS to log in -i instance# append # to ORACLE_SID -sid set ORACLE_SID to sid -top # limit some large queries to on # rows - repeat Repeat an coomand time. Sleep between two calls Command are: - execute: cursors currently being executed - longops: run progression monitor - sessions: currently open sessions - stack get process stack using oradebug - cursors [all] : [all] parsed cursors - sharing : print why cursors are not shared - events [px]: events that someone is waiting for - events [read_by_other_session] events that someone is read by other session - ash [duration] [-f ] active session history for specified period e.g. 'ash 30' to display from [now - 30min] to [now] e.g. 'ash 30 10 -f foo.txt' to display a 10 minutes period from [now - 30min] and store the result in file foo.txt - ash_wait_graph [duration] [-f ] PQ event wait graph using ASH data Arguments are the same as for ash except that the output must be shown with the mxgraph tool - ash_sql Show all ash rows group by sampli_time and event for the specified sql_id - [-u ] degree degree of objects for a given user - [-u ] colstats stats for each table, column - [-u ] tabstats stats for each table - params []: view all parameters, even hidden ones - snap: view all snapshots status - bc: view contents of buffer cache - temp: view used space in temp tbs - asm: Show asm space/free space - space []: view used/free space in a given tbs - binds : display bind capture information for specified cursor - fulltext : display the entire SQL text of the specified statement - last_sql_hash []: hash value of the last styatement executed by the specified sid. If no sid speficied, return the last hash_value of user sessions - openv []: display optimizer env parameters for specified cursor - plan []: get explain plan of a particular cursor - pxplan : get explain plan of a particular cursor and all connected cursor slave SQL - wplan []: get explain plan with work area information - pxwplan : get explain plan with work area information of a particular cursor and all connected cursor slave SQL - eplan []: get explain plan with execution statistics - pxeplan : get explain plan with execution statistics of a particular cursor and all connected cursor slave SQL - gplan : get graphical explain plan of a particular cursor using dot specification - webplan get graphical explain plan of a particular [/] cursor using gdl specification []: optional: child_number, default is zero. optional: decorate to print further node information. default is 0, 1 => print further node information such as cost, filter_predicates etc. 2 => in addition to the above, print row vector information sample usage: # ora webplan 4019453623 print more information (decorate 1) # ora webplan 4019453623/1 1 more information, overload! (decorate 2) # ora webplan 4019453623/1 2 using sql_id along with child number instead of hash value # ora webplan aca4xvmz0rzup/3 1 - hash_to_sqlid : get the sql_id of the cursor given its hash value - sqlid_to_hash : get the hash value of the cursor given its (unquoted) sql_id - exptbs: generate export tablespace script - imptbs: generate import tablespace script - smm [limited]: SQL memory manager stats for active workareas - onepass: Run an ora wplan on all one-pass cursors - mpass: Run an ora wplan on all multi-pass cursors - pga: tell how much pga memory is used - pga_detail | -mem : Gives details on how PGA memory is consumed by a process (given its os PID) or by the set of precesses consuming more than MB of PGA memory (-mem option) - pgasnap [] Snapshot the pga advice stats - pgaadv [-s []] [-o graphfile] [-m min_size]: generate a graph from v and display it or store it in a file if the -o option is used. -s [] to diff with a previous snapshot (see pgasnap cmd) -o [graphfile] to store the result in a file instead of displaying it -m [min_size] only consider workareas with a minimum size - pgaadvhist [-f []] display the advice history for all factors or for factor between f_min and f_max - sga: tell how much sga memory is used - sga_stats: tell how sga is dynamically used - sort_usage: tell how temp tablespace is used in detail - sgasnap [] Snapshot the sga advice stats - sgaadv [-s []] [-o graphfile] generate a graph from v and v and store it in a file if the -o option is used. -s [] to diff with a previous snapshot (see sgasnap cmd) -o [graphfile] to store the result in a file instead of displaying it - process []: display process info with pga memory - version: display Oracle version number - cur_mem [ ] display the memory used for a given or all cursors - shared_mem [ ] detailed dump of cursor shared mem allocations - runtime_mem [ ] detailed dump of cursor runtime memory allocations - all_mem [ ] do all of the memory dumps - pstack |all [] run pstack on specified process (or all if 'all' specified) and store files in specified dir ( when not specified) - idxdesc [username] list all indexes for a given user or for a given user and table - segsize [username] list size of all objects(segments) for given user for a given user and object - tempu list temporary ts usage of all users or for a given user - sqlstats [ ] list sql execution stats (like buffer_gets, phy. reads etc) for a given sql_id/hash_value of statement - optstats [username] list optimizer stats for all tables stored in dictionary for a given user or for a given user and table - userVs list all user Views (user_tables, user_indexes etc) - fixedVs list all V$ Views - fixedXs list all X$ Views - px_processes list all px processes (QC and slaves) - cursor_summary summarize stats about (un)pinned cursors - rowcache summarizes row cache statistics - monitor_list lists all the statements that have been monitored - monitor [xml]: wraps dbms_sqltune.report_sql_monitor(). Directly passe the arguments to the PL/SQL procedure. Args are: sql_id, session_id, session_serial, sql_exec_start, sql_exec_id, inst_id, instance_id_filter, parallel_filter, report_level, type. Examples: - monitor xml shows XML report - monitor show last monitored stmt - monitor sql_id=>'8vz99cy9bydv8', session_id=>105 will show monitor info for sql_id 8vz99cy9bydv8 and session_id 105 Use simply ora monitor 8vz99cy9bydv8 to display monitoring information for sql_id 8vz99cy9bydv8. Syntax for parallel filters is: [qc][servers([,] [,] )] Use /*+ monitor */ to force monitoring. - monitor_old [ash_all] [] [qc| [ []]] Old version of SQL monitoring, use a SQL query versus the report_sql_monitor() package. Display monitoring info for the LAST execution of the specified cursor. Cursor response time needs to be at least 5s for monitoring to start (use the monitor hint to force monitoring). Without any parameter, will display monitoring info for the last cursor that was monitored - ash_all will aggregate ash data over all executions of the cursor (useful for short queries that are executed many times). If parallel: - qc to see only data for qc - slave_grp# to see only data for one parallelizer - slave_grp# + slave_set# to see only data for one slave set of one parallelizer, - slave_grp# + slave_set# + slave# to see data only for the specified slave - sql_task [progress | interrupt | history | report ] progress: progress monitoring for executing sql tasks interrupt: interrupt an executing sql task history: print a history of last n executions report: get a sql tune report - sql_use_temp_segment Find Who And What SQL Is Using Temp Segments. - sh Run a shell command. E.g. ora repeat 5 10 sh 'ps -edf | grep DESC' - awr_dbid Show AWR dbid - awr_dbtime [dbid] Show AWR dbtime - awr_dbtime [dbid] [inst] Show AWR dbtime - awr_dbtime_order [dbid] Show AWR dbtime order by desc - awr_sql_elaps_time [dbid] Show AWR SQL elapsed time - awr_sql_elaps_time [dbid] [inst] Show AWR SQL elapsed time - awr_sql_elaps_time_order [dbid] Show AWR SQL elapsed time order by desc - awr_logical_reads_order [dbid] - awr_logical_reads [dbid] Show AWR logical reads M Show AWR logical reads M order by desc - awr_physical_reads [dbid] Show AWR physical reads M - awr_physical_reads_order [dbid] Show AWR physical reads M order by desc - awr_db_cpu_per [dbid] [inst] Show AWR db_cpu_time cpu percent - awr_user_cpu_per [dbid] [inst] Show AWR oracle user_time cpu percent including backgroud process - awr_sql sql_id [dbid] Show AWR sql_id executions, per elapsed time. - awr_fulltext sql_id [dbid] Show AWR sql fulltext - awr_plan sql_id plan_hash [dbid] Show AWR sql plan, if plan_hash is null, show all plans. - awr_binds sql_id end_snap_id [dbid] Show AWR bind values in end_snap_id. - tab_frag owner [frag_percent] Show table fragment. - index_frag owner [frag_percent] Show index fragment. - rman_fullrestore_scripts dest_dbfile_dir Generate rman full database restore scripts - top_buffers_gets Top 10 by buffer gets > 10000 - top_physical_reads Top 10 by Physical Reads (disk_reads > 1000) - top_executions Top 10 by Executions > 100 - top_parse_calls Top 10 by Parse Calls > 1000 - top_sharable_memory Top 10 by Sharable Memory > 1M - top_version_count Top 10 by Version Count > 20 - top_cpu_usage Top 10 by CPU usage (cpu_time) - top_running_time Top 10 by Running Time (first_load_time desc) - create_tbs path size Create test database's tablespace script - create_tbs path size [dbid]Create dbid's test database's tablespace script - hold_txlock Show sessions holding a TX lock - wait_txlock Show sessions waiting a TX lock - rowid Display rowid's file_id, file_name, block info, object info, extent_id Memory: The detailed memory dumps need to have events set to work. The events bellow can be added to the init.ora file event="10277 trace name context forever, level 10" # mutable mem event="10235 trace name context forever, level 4" # shared mem NOTE ==== - Set environment variable ORA_USE_HASH to 1 to get SQL hash values instead of SQL ids - Set environment variable DBUSER to change default connect string which is "/ as sysdba" - Set environment variable ORA_TMP to the default temp directory (default if /tmp when not set)
SakEmail components Copyright ?1997 - 2003 Sergio A. Kessler web: http://groups.yahoo.com/group/sakemail/To subscribe to the mailing list of sakemail, just go tohttp://groups.yahoo.com/group/sakemail/History:0.9 - First released version0.9.1b -Fixed when a mail server reply on the connection with more than one line0.9.2b - I forget to return a value in functions retrieveHeader/Message =) and fixed it. Some minor bugs that I don‘t remember fixed.- Added MIME-compliant base64 support (not for use by now). Added examples.0.9.2.1b- Fixed a bug when send a mail and the first line disappear (thanks to Arun)- Now, you could do MySMTP.MsgTo := ‘[email protected]; [email protected];[email protected]‘; the spaces before/after semicolon doesn‘t matter (I hope ;)).0.9.3b- Many changes, I added a SakMsg component that make send binary attachments a snap. But have one problem, if you send as attach a file > 20 Kb, it doesn‘t work (I don‘t know why, maybe a problem of sockets). Developed with a version 2.0b of WSockets and D3.0.9.3.1b- Changed the POP.login to a function that return the number of new msgs.- Added the event OnRetrieveProgress on the SakPOP, and fixed the example, sorry =)- Minor changes to the code.1.0- Developed with WSockets 1.2 POP.Login now return a boolean depending id the user is authorized, and POP.Init return the number of new msgs.1.01- Fixed a bug with a bounced mail.1.02- Minor bugs fixed (some variants of boundary)14/10/971.1.0- Warning: WSockets1.2 have some bugs that result in bad attachments. So I decided to use the sockets of Delphi 3 founded in D3 c/s D3.01 pro and D3.01 c/s. Now all seems to work fine and much more smooth. And of course the interface of SakEmail hasn‘t changed.26/10/971.2.0- Added the Reply-To field to TSakMsg comp. Now you must use ‘,‘ when you want to send the msg. to multiple recipients, i.e.: ‘[email protected], [email protected],[email protected]‘ This change is done for better compatibility with other emails clients.- Better formatting of the field Date of TSakMsg. Some changes to the code.17/11/971.2.1- Now, all searches are made in case-insensitive, it could prevent some unexpected responses (no one reported, but...). Some changes to the code (again).20/11/971.2.2- Some bugs fixed. (Thanks to Serge Wagener from .lu)24/11/971.2.3- Added the field ‘MIME-Version: 1.0‘. It seems that is necessary :)25/11/971.3.0- Added compatibility with SCO and VAX servers. Fixed a minor bug with the boundary.- Change the generator of the message id.- Added the field MessageId and InReplyTo to the TSakMsg component.- Added the field In-Reply-To that is added to the message generated when it is <> ‘‘.30/11/971.3.1- Almost rewrote the parsing code. Now is more easy for you if you want hack/modify the code.- Better treatment of emails with html inside.15/12/971.4- Added support for UUCoded attachments.- Added a small delay when sending the email, seems that some servers can‘t deglut the info too fast, causing problems with sockets buffers and leading to crash the client machine, I don‘t know if is a Borland bug or Microsoft bug. (thanks to Don Higgins).19/12/971.4.1- Fixed a bug that send double ‘<‘ and ‘>‘ (ie. <<[email protected]>>) when the full user name is used. Check the new SMTP demo. Thanks to Serge Wagener for locate this bug, track it down and send me the fix.2/2/981.5.0- Added the Canceled property to TSakPOP and to TSakSMTP. Due to this addition now RetrieveAllMessages is a function that return the number of msgs. retrieved and SendMessage is a boolean function (maybe someone has pressed the cancel btn).- Fixed a bug when the subject field is too large.9/2/981.5.1- Fixed a bug with a message within a message (recursive msgs).18/2/981.5.2- Fixed a bug what happens when after the field ‘To:‘ appear a blank line(Thanks to Osvaldo Fillia). Fixed a bug when sending email to more than two address (the separator is still ‘,‘).9/3/981.6.0- Sometimes the filenames of an attachment contain invalid chars making very dificult to open a TSaveDialog (you have noted this ?), now SakEmail deletes the invalid chars.- Applied a patch from Matjaz Bravc, that resolve the problem of localized dates, letting you choose (in design time) if you want localized dates (NOT recommended) or standards dates (english) via the LocalizedDates boolean property in the TSakSMTP comp. Thanks also to Serge Dosyukov for sending me a fix.- Also I applied another patch of Gregor Duchalski that cure a bug with PChar when this unit is used under NT. - It seems that some machines need more delay when sendig a msg (see previous posting 19/12/97), thanks to Matjaz Bravc.- I discover a bug in the transparency code, it is fixed now. Did you see the benefits of Open Source Software ? :)26/3/981.6.1- Added a FUNCFileName private variable to manage the complete path of the attached file. I receive problems reports with this, it work now ?.- Reduced the line sleep to 30 (tell me if this value doesn‘t work for you).27/4/981.7.0- Fixed a memory leak, thanks to Don Higgins.- Moved the string esErrorInFormatOfMsg to a property of SakPOP.- Because some people need to use IP addresses instead of Host names, I‘ve added a new property IPAddress to SakPOP and SakSMTP. If both are filled, then the Host name will be used, thanks to Roger F. Reghin for reporting this. The side effect for this is that YOUR app must check if the host is a host name or a IP address, in my app I remove the periods and try to convert the result to a float (long integers don‘t work, but float accept chars ‘e‘) if it doesn‘t work I assume that is a host name (someone has a better and simple idea ?).- Added the property FileStream to the class TAtachedFile and the procedure SaveToStream, this was done by Brian Sheperd- The address separator (in the TO: field) is ‘,‘ and ‘;‘ now (before it was ‘,‘ only).1.7.1- Roger F. Reghin has sended me a pair of nice patches that resolve in a good behavior when the destination address is something like "Roger Reghin" <[email protected]> and some servers says that they couldn‘t relay that mail, etc. Also Roger has made the IPAddress property obsolete (do not use it, use Host instead), SakEmail will resolve the host properly no matter if it is a host name or a IP address. So in the next version I will remove the IPAddress property. Thank you, Roger.1.8.0- Well, it seems that I made a mistake, I investigated the previous behavior and it is a fault of the SMTP (RFC 821), so I fixed it.- The IPAddress property has been removed, use Host. Goeran Strehl (asem) has sended me a patch that fix a memory leak and one problem with the object inspector and the Text property of a SakMsg. Dmitry Bondarenko say that some servers do not send the msg size after the RETR command, so he fix that issuing a LIST n command first (work nicely).- Added the property CC (Carbon Copy) to the SakMsg object.1.8.1- Added the property ReturnPath to the SakMsg comp. Minor changes to the scanning code for the filename of attachments.1.8.2- Fixed a bug with the filename of attachments (thanks to Taufer Pavel Ing.).- Added the function IsIPAddress from hou yg (the actual code don‘t work if the server is 265.net :) Fixed a minor bug with html pages like attachments. Some fucking email server return a bounded message declaring the boundary like ‘boundary = ‘ and not ‘boundary=‘ wich is clear in the RFC, fixed.1.8.3- A obscure bug was found by HuangYeJun from china, in the RetrieveHeaders function if the retrieved text was larger than 1024 bytes and the crlf.crlf fall in the middle of two chunks, the function is blocked. I don‘t use this function, btw.1.8.3.1- Just cleaned up a bit the FindUUAtachs function. Not bug or enhancements release. Serge Wagener put me to work >:|1.8.4- Dmitry Bondarenko (again) has found a bug in wich I do not respect the RFC, wich say that replys from the SMTP server could be multi-line, and the previous version just manage as far as two lines. He also send me a nice patch, so the bug is fixed.- Craig Manley added a ExtraHeaders property, please, use with care, it‘s just not valid to put inside it whatever thing.- The CC header was not being added to the headers that were being sent, so Craig fixed it.- Warning: I‘ve put try/except in the TSakPOP.Connect and TSMTP.Connect function around the line FSocket.Open, so you will need to write something like: myPOP.Connect; if POPError then ... in your code, the old way was: try myPOP.Connect; except ..... end; If you are strongly opossed to this change, drop me a line and tell me why (I‘m in doubts).1.8.5- Greg Nixon added the priority property. The default priority for each msg created will be prNormal, so you don‘t need to change your code any bit.1.8.6- Ulf Sturegren has added D4 compatibility, not many changes to the source (one letter), but he found the error.- Hou yg has sent to me a revisited IsIPAddress function, so I put the newer function in, infortunely my reply to him doesn‘t want to go.1.8.7- Ok, I discovered a weird bug, some old emailers (navigator 2) does not format the message in multipart mode if people send an attach, without writing any text and with no MIME settings. Fixed. This could be serious, I recommend upgrading.1.8.8- A small fix with the CC field. Some stupid mail servers put tabs in some fields (CC:, TO:) when they want to make a new line, the correct is to put at least a space in the beginning of the line, added a little code to "fix" that.1.8.9- Some ‘moderns‘ pop3 servers doesn‘t support the LAST command, so I‘ve added a little code to cope with this and added a boolean property ServerSupportLastCmd. See TSakPOP.Init for more details. Reported by Jan Najvarek.1.9.0- Kaufman Alex has added two properties to the SakMsg object, the ContentType and the Headers property, that should be self explaining (I modified a little the code he sended me, btw).1.9.1- I rewrote and greatly simplified the code that deal with the multiple address in the TO: field and remove some possible bugs in it.1.9.2- Alex discovered and fix a bug when a file attached is not enclosed between quotes, resulting in the filename without the first and last character.1.9.3- Better detection of the boundary in multipart messages. Fixed a bug when the attached file is empty.1.9.4- Chris G黱ther send me *lots* of memory leaks fixes, very good job, Chris. - Some weird PGP messages are now processed well.- Yang Qiandong from china fixed a compiler hint and a warning.- Modified TSakSMTP.FReceiveTextFromSocket as suggested by Greg Nixon.- Dmitry Bondarenko send me a patch that fixes some issues with the LAST command (that some servers don‘t implement) and other patch that fixes a problem when servers add spare words in the tail of the answer.- Some minor changes suggested by Matthew Vincent.- Support for _big_ attachments files (me).- Make the code more modular and simple (still is not very modular).1.10.0- Move some stuff to a sak_util unit.- Support for quoted-printable msgs, thanks to Chris G黱ther.- Fix the BCC field.- New property sakMsg.ContentTransferEncoding.2.0.0- Major reestructure of the files and the source code.- Simplifyied sakPOP3.pas a _lot_- Support encapsulated messages (message/rfc822).- Nested multipart messages are processed fine.- Attachs with quoted-printable are processed fine.- Many bugs fixes.2.0.1- A fiasco, sorry.2.0.2- Fixed a bug in the sak_CleanUpAddress.- Do the rigth job if the ContentType is ‘plain/text‘ and the encoding is base64.- Redone sak_ExtractAddress and sak_ExtractAlias.- New ‘Sender‘ property in SakMsg (normally not used, so do not use it, unless you know what you are doing) ‘Thanks‘ to Alex Kaufman for this.2.0.3- A *severe* bug with multiple addresses was fixed.2.0.4- Fixed bogus Message-number (Message-id is the correct) Thanks to Peter Honan- Added SizeInBytes property to the SakMsg component. (petition of Alex Kaufman)- Fixed a minor bug in TSakPOP.RetrieveHeaders. Fix from Alex.- Added RetrieveMessageOnlyHeaders and - RetrieveAllMessagesOnlyHeaders.2.0.5- Fix when the mail server reply is like <cr><cr><lf> (two cr).- Fix function IsIpAddress.- Both fixes by Alessandro Rossi.2.0.6- Fix a bug in the sak_Base64Decode function when the data to decode is null (I found it in the hard way).- Andy Charalambous make it sure you can send more than one email without disconnecting and connecting again.- And Chris ‘Memory Hunter‘ G黱ther killed some memory leaks (again).2.2.0- the f* sleep line that was bothering us for years is gone, gone, gone. Thanks to Syed Ahmed.- a getUIDL method of SakPOP. Thanks to Alex Kaufman.- a UIDL property on SakMsg. (me)- a SakPOP.GetUIDLsOnRetrieve boolean property (default false) (me)- change some ‘Exception.Create()‘ to ‘raise Exception.Create()‘ Thanks to Anton Saburov.- change SakPOP.Init from function to procedure (me)- new SakPOP.NewMsgsCount property (me)- changed SakPOP.Password to SakPOP.UserPassword (me)- changed SakPOP.ErrorInFormatOfMsg to SakPOP.StrErrorInFormatOfMsg- OnLookup event on SakPOP and SakSMTP. Thanks to Syed Ahmed.- OnConnecting event on SakPOP and SakSMTP (me).- OnReceiveTextFromSocket event on SakPOP and SakSMTP (me). (mostly for debug)- OnSendTextToSocket event on SakPOP and SakSMTP (me). (mostly for debug)- Headers are retrieved without the mail body (ugly bug, fix from Alex Kaufman)2.4.0- I‘ve revamped TSakMsg, many funcionality from SakPOP was moved to SakMsg, where it belongs.- Now SakMsg has a RawMail property wich you may find useful, now you can do: SakMsg1.RawMail.LoadFromFile(‘(uidl).mail‘); SakMsg1.ParseMsg; or SakMsg1.RawMail.LoadFromStream( myStream); SakMsg1.ParseMsg; or SakMsg1.RawMail.SaveToFile( ‘(uidl).mail‘); etc, etc...- Added a property TSakMsg.ClearRawMailAfterParse for memory saving.- the return of the f* sleep line (it causes freezes on winsock 1.1 systems like win95, win98 has winsock 2 so there is no problem if you remove the line)- lost of the DecodeProgess events :( (sorry, I don‘t know how to fit this events on the new SakMsg)2.6.0- the sleep() line is dead, it will never come back. Sending an email is a pleasure now.- SakMsg has a TextEncoding (8Bit, Base64) property, I think this will be useful to people with others charset than iso-8859-1- the base64 routines have been rewritten, they are more OO and faster (they are now in SakMIME.pas).- cosmetic changes all over the place.2.6.1- simplifyed ParseMsg2 a lot, it work better now.- speed up the search for uucoded attachs (the previous search was very dumb)- fixed bug Msg.SizeInBytes always 0- added a couple of Application.ProcessMessages to make the app more responsive.2.6.2- moved some functions from sak_utils to SakMIME.- make const parameters all over the place.- fix the bug that introduces a final crlf in quoted-printable attachs.- fix a division by zero if attached file is 0 bytes long, fixed by Peter Kollanyi.2.6.3- fix a rare bug when the header of a email (more probably a encapsulated one) has first line/s in blank. Easy and innocuous bug.2.6.4- fix the bug that insert the attachs of type text/* on the body of the email.- change the Smtp.SendMessage for Smtp.SendTheMessage to avoid a BCBuilder problem. Both problems reported by Andreas Franzen. SendMessage is still there, but it‘s now deprecated, I will remove it in the future.2.6.5- moved the ParseMsg activation from SakPOP to SakMsg (where it belong), this means that after setting the RawMail property of SakMsg, this does a ParseMsg automatically. before: SakMsg1.RawMail := ... SakMsg1.ParseMsg; now: SakMsg1.RawMail := ... hope I‘m not breaking too much code out there ... :)- some changes in the way attachments are processed (now the html part is separated correctly and images within the html are recognized)- RetrieveMessage() and RetrieveMessageOnlyHeaders() now take an additional parameter, a TSakMsg var, so people can change some parameters before parsing, see the source in SakPOP3.pas (the old way is still supported, but they will be removed in the future)- bug fixes that I do not remember.3.0.0- moved code around.- removed deprecated functions (I told you about this)- new SakAttFile unit.- Base64Encode( AttFile), Base64Decode( AttFile), UUDecode( AttFile) has been moved to the TAtachedFile object, so you can do AttFile.Base64Encode, AttFile.Base64Decode, etc- SakSMTP have lost EncodeStart, EncodeProgess and EncodeEnd events as a consequence of the previous change.- SakPOP.Canceled and SakSMTP.Canceled properties have been made read-only and SakPOP.Cancel and SakSMTP.Cancel procedures (or methods) have been added.- add a SakMsg.FillRawMail method that will fill the RawMail property with a rfc822 message based on the properties of SakMsg.- changed SakSMTP.Quit & SakPOP.Quit to Disconnect- deleted TAttachedFile.FileStream (redundant), use BodyBin- removed the function sak_getTempFileName (as it should no be trusted) use function sak_GetTempPath- the new SakIMAP component !, this make a pleasure to work with incoming emails (as you can have folders, etc). Note: the IMAP component has only been tested with the Uni. of Washington server, but it should work with any *STANDARD COMPLIANT* server. Anyways, the code of this component is very simple, so if you have problems, a look in the source code can enligthen you.3.0.1- fixed a brown paper type of bug.3.0.2- support the case where attachs do not come from files (Lars Karlslund)- minor bugfix in UUDecode function (Lars Karlslund)- if the SakMsg.Username is empty, do a VRFY command at the smtp server to try to get the full user name (sergio)- function TSakIMAP.GetFolderList (Peter Nagel)- function TSakIMAP.GetHierarchyDelim (Peter Nagel)- frustrated intent (ie. commented out) to remove memory leaks in POP, SMTP & IMAP destroy functions (Ronald Moesbergen)3.0.3- actually create (and free) the FolderList in sakIMAP (Neculau Andrei)- try to send the FQDM to the HELO command in SMTP (sergio)- commented out the VRFY command in SakSMTP, and cut the from address in the From field (in SakMsg), so if the username is empty, the SMTP server rewrite the from address in a complete way, with username & full address (sergio)- fix a minor bug in TBase64DecodingStream.Write function (Lars Karlslund)3.4.0- many, many improvements to the IMAP component by Peter Honan (I applied the patch with minor modifications, mainly to respect delphi coding standard, taking out the overloading, the selectFolder function was overcomplicated, etc)- FAQ updated (me)3.4.1- minimize the chance for two temporal messages stored on disk to collide (can be hit in previous versions if you run multiple instances of retrieveMessage at the same time)- FAQ updated.3.4.2- a new sak_CleanUpAddresses() implementation, by Knut Baardsen- better handling for temporal messages, suggested by Andrew- many improvements (including ACL -Access Control List) to the IMAP component by James Chaplin3.4.3- reverted to the old sak_CleanUpAddresses() implementation Knut‘s one is almost rigth, but don‘t let us use addresses without domains- add Headers.Clear before filling headers, by "Antonio Carlos Ribeiro Faria" <[email protected]>3.5.0- add TSakMsg.LoadFromTextFile from Oak Chantosa- big jumbo mambo patch from James Chaplin first patch: 1) Operation timeout - OperationTimeout timeout for non-responding receive operations. 2) Forced abend - ForceAbend method that will disconnect and reset state. 3) Optional folder lists - AvFolderList and AvSUBFolderList provide alternatives to FolderList and SUBFolderList that ensure the lists do not contain inacessible folders ( flagged by the server ). 4) Folder name fix - Provided a function to "fix" folder names before submission. Currently it fixes names containing spaces. second patch: 1) Capability - Ask for server capabilities/extensions. 2) Noop - Basic noop command - updates message counts as well - preferred alternative to status. 3) Status - Explicit status command - generally useful for status of a non-selected mailbox. 4) Fetch - Retrieve message data. 5) FetchBody - Retrieve the body of the message. 6) ExamineFolder - A read-only select command. 7) CloseSelectedFolder - Close the currently selected folder. 8) Idle - RFC2177 extension - not implemented on very many servers. 9) Search - Search based on RFC2066 criteria. 10) UIDSearch - Search based on RFC2066 criteria - results are in UID form. 11) UIDStoreFlags - Store message flags based on UID. 12) UIDFetch - Fetch message data by UID. 13) UIDCopyMessageToFolder - Copy a message by UID. 14) Authenticate - Basic framework. Only plain authentication extension implemented. 15) CloseOnError - A new property that allows the user to turn off the default behaviour of disconnecting from the server when an IMAP error is received 16) Namespace - RFC2342 Namespace query command. 17) ListFullHierarchy - Property which allows a switch between "*" ( default ) or "%" as the wilcard for default folder/list methods. 18) List - Explicit list command in case it is needed. third patch: 1) fix problem with imapd 2001a, reported by Holger Mauermann. 2) remove all warnings.3.5.1- revert change to the base64 encoding routine.3.5.2- changes from James Chaplin: 1) TSakIMAP will now properly process non-numeric UIDs for messages ( there was a sak_StrWord2Int transform being used before - which always produced a 0 value for non-numeric UIDs ). 2) TSakIMAP.RetrieveMessageExt ( private method ) was modified to provide a retrieval by either MsgID or UID. 3) TSakIMAP.RetrieveMessageByUID was modified to use the slightly more efficient TSakIMAP.RetrieveMessageExt(UID) method specified in 2) above. I also made an update to the SakMIME.pas unit. The changes that were implemented are: 1) sak_Base64Encode - a basic Base64 encoder. String input and string output with the option for CRLF splitting. 2) sak_Base64Decode - a basic Base64 decoder. String input and string output with a control for CRLF interpretation. 3) sak_Base64Verify - a very basic Base64 string verifier.3.5.3- robustify and code cleanups by Paul Vernon.3.5.4- access violation fix by Paul Vernon.3.5.5- go back to good old trusty 3.5.23.5.6- this time, all the cleanup & fixes from Paul Vernon seems to work well.3.6.0- Paul Vernon latest minor fixes- added basic SMTP authentication, by Delfi and Antonio Carlos Ribeiro Faria3.6.1- fix a mayor bug when sending to many addresses (by sergio)3.7.0- add full support for html mails, by Paul Vernon. (The TAttachedFile now has an extra boolean property called embedded. This property lets you use the syntax <img src=DownloadFiles2"[email protected]"> in your HTML mails)- fix a weird typo for BCC fields- add Content-ID, by alejandro Castro- fix "_" characters in subject, regression fix.- cleanups all around, by Paul Vernon.- SMTP example updated to cope with html emails.**warning** from this version, the html part of mails will not be stored as attachments by default, if you want this behavior, you just do something like: aSakMsg := TSakMsg.Create( self); aSakMsg.HTMLAsAttachment := true; ...3.7.1- fix TSakMsg.PopulateList (Jalin)3.7.2 (codenamed "melissa")- fixed a bug when the Populatelist procedure got re-written in sakMsg. It wasn‘t populating the SendTo field if there was only one e-mail address... (Paul Vernon)3.7.3 - Congratulations to Sergio on the addition to his family. This release was made by Paul Vernon who has temporarily taken over the release functions for the SakMail components whilst Sergio spends time AFK!- The 3.7.2 bug fix added blank entries to the address lists. The PopulateList procedure has been re-written again to hopefully cope with any type of e-mail address formatting.- The SMTP example noted in 3.7.0 actually shipped with this release!3.7.4- Bugfix for detecting UUEncoded mails correctly. Previous versions processed MIME mails with the value ‘begin xyz‘ if it appeared at the beginning of a line as a UUEncoded mail when they should not have.- POP and SMTP connect procedures are now functions. Existing code is unaffected. However, you can now use the following code if (sakPOP.Connect) then begin end;- POP gracefully quits if it receives an error now by calling Disconnect correctly.3.7.5- Further code to improve identification of UUEncoded mails. Essentially looking for the end as well as the beginning to ensure that it is correct.- Code optimisation of certain UUEncoded mail id functions.- Fix to ensure that the body of a mail that is UUEncoded is not lost.- MIME-Version string introduced into TsakMsg component to help with UUEncoded mail identification.- SizeInBytes property altered to read private variable using a function. If the private variable is 0, the function reads the length of the FRawMail.Text property.- Fix to make sure that the filename is not overwritten by a blank value when parsing mail-headers.3.7.6- Fixed list index out of bounds error.- Added POP3 RSET call TSakPOP.Reset.3.7.7- Altered SizeInBytes and Octets values to return server-side size when d/l headers only and use actual size once the entire message is downloaded.- Fixed a bug in GetBasicHeaders where To and CC fields could be mishandled if the mail headers were formed in a particular way.4.0.0 beta- All methods are now wrapped in classes. sak_util is now included for backwards compatibility only.- Several changes to make sakMail thread safe including the introduction of Mutexes which are cross process safe. Critical sections were an option however, although mutexes are a little slower, they are much more effective when you aren‘t sure how the code is going to be deployed...- Made several changes to the way connections are tracked, now making better use of the underlying Delphi components own properties and functions.- Several bug fixes included from solutions posted on mailing lists. Including change to datetime function to respect local time separator. There are more including one that Adem re-raised.- Removed almost all pointers as per Adems suggestion. Makes for neater code.- Hopefully backwards compatibility is kept. This is one of the objectives of the excersice although, internally, the components no longer use any of the non-object based methods. Also some of the non-object based methods actually have been re-written to create an object use the instance of the original method and then destroy the object again. This introduces a minor overhead however, because the objects are discreet, the trade is for much better memory usage and greater thread safety.- Introduced an include file to define compiler directives. Currently there are two directives. One defines whether to use the VCL or not, the other defines whether or not to use the FastStrings components. - With the intoduction of the Include file, this allows the development of code that is optional for users. One of these such changed is the use of the FastStrings base64 decoder. If you install the FastStrings components and turn on the compiler directive, you should have no functional changes however, the base64 decoder routines should have a much higher performance rating. Tests clock in at over 2000% faster attachment decoding on a P4 1.8GHz machine. (1.2Mb file 1686mS native sak Base64 Decoder, 79mS using FastStrings!)- This version is being released as a beta as the changes are pretty drastic. If the code is deemed to be stable and backwards compatible then it will be re-released as v4.0.1 with no changes.4.0.1 beta- Fixed an issue where Range Checking highlighted that the Attachment b64 decode routine raise a Range Error if the line that was to be decoded was empty. i.e. ‘‘.- Introduced a compiler directive to turn off range checking in the sakMIME procedure TBase64DecodingStream.Write to make sure that it runs correctly as Range Checking causes issues in this function.4.0.2 beta- Changed MailDateToDateTime function to the one provided by DengZhaoHui with a few modifications as even though it has better date processing than the original it caused EConvertErrors with some non-rfc dates.- Added the compiler directive to allow the inclusion of MD5 components from the DCPCrypt suite of encryption components. This allows the components to do APOP and SMTP AUTH functions as specified in RFCs 2095, 2104, 2449 and 2554. {UseDCP} ***** NOTE: These functions are experimental as although they are RFC compliant, they have not been tested against a secure mail server yet... *****- Using EurekaLog during load testing of the POP mail component, found and fixed several AV‘s in sakMSG, sakMIME and sakPOP. Mainly simple mistakes that required re-ordering of code or more checks before trying to manipulate data.- Altered the sockets code to be more stable with some servers. The previous implementation was totally incompatible with SendMail NT v3.0.2.- Fix added to compensate for incorrect operation of Connected property in some versions of Delphi.- Altered GetMultiLineFieldBody as per Adems suggestion. Also took some of Adems code and added it to GetFieldValueFromLine as the escape characters can appear in single line headers as well as multi-line ones.- TClientSocket is deprecated in Delphi 7. This may be the next large change in the sakEmail components. - Updated distribution to include more RFC‘s regarding the message format, POP and IMAP and hashing functions for CRAM mechanisms.- Fixed the handling of redirected mails as created by Eudora.- Force PopulateList to clear the list before populating it again.- Created a Delphi 6 package file.4.0.3 - Fixed AUTHSMTP buffer initialisation error. (Dmitry G. Kozhinov and Gabi Slonto)- Improved identification of servers that do not support the UIDL command. A small overhead is intorduced on servers that do support the command and have several mails to download but the feature allows better interaction with those servers that do not support UIDL.- Priority is now reported correctly when an e-mail is being decoded rather than only being used when sending an e-mail.4.0.4- Fixed an issue with a malformed header in a mail sent from MS Word through an Exchange server- Added a couple of try...finally blocks to the sakIMAP component.- Altered the sakIMAP components connected function to mirror the more accurate sakPOP method.- Consolidated all compiler directives into sakDef.inc- Added versioning compiler directives to allow the compilation of sakemail under Delphi 4.- General tidying of code. 4.0.5- Created a Delphi 7 package- Added properties to the IMAP component to allow read access to the LocalAddr and LocalHost socket properties.- Bugfix to sakMsg PopulateList function where a comma separated list did not contain any spaces- Access violation in sakPOP component due to incorrect use of free,freeandnil and compiler directives4.0.6- Added several features to the IMAP components.- Tidied up SMTP authentication routines (Improved use of MD5 for authentication using DCP components)- Included capability to send messages without an SMTP server (using Indy DNS components for MX lookups)- Bugfix in message parsing to stop a recursion loop due to a malformed mail.4.0.7- Memory leaks found by Amos and Paul regarding the sakMsg and sakPOP units respectively.- Bug fixes to attachment save code including stripping out invalid .. sequences from filenames- Improved the GetConnectedState method to check against the RemoteHost value on the Socket.- Updated POP example to be more responsive when downloading mail. Fixed a memory leak.Don‘t forget to subscribe to the mailing list (see the web pages at http://groups.yahoo.com/group/sakemail/)
Contents Module Overview 1 Lesson 1: Memory 3 Lesson 2: I/O 73 Lesson 3: CPU 111 Module 3: Troubleshooting Server Performance Module Overview Troubleshooting server performance-based support calls requires product knowledge, good communication skills, and a proven troubleshooting methodology. In this module we will discuss Microsoft® SQL Server™ interaction with the operating system and methodology of troubleshooting server-based problems. At the end of this module, you will be able to:  Define the common terms associated the memory, I/O, and CPU subsystems.  Describe how SQL Server leverages the Microsoft Windows® operating system facilities including memory, I/O, and threading.  Define common SQL Server memory, I/O, and processor terms.  Generate a hypothesis based on performance counters captured by System Monitor.  For each hypothesis generated, identify at least two other non-System Monitor pieces of information that would help to confirm or reject your hypothesis.  Identify at least five counters for each subsystem that are key to understanding the performance of that subsystem.  Identify three common myths associated with the memory, I/O, or CPU subsystems. Lesson 1: Memory What You Will Learn After completing this lesson, you will be able to:  Define common terms used when describing memory.  Give examples of each memory concept and how it applies to SQL Server.  Describe how SQL Server user and manages its memory.  List the primary configuration options that affect memory.  Describe how configuration options affect memory usage.  Describe the effect on the I/O subsystem when memory runs low.  List at least two memory myths and why they are not true. Recommended Reading  SQL Server 7.0 Performance Tuning Technical Reference, Microsoft Press  Windows 2000 Resource Kit companion CD-ROM documentation. Chapter 15: Overview of Performance Monitoring  Inside Microsoft Windows 2000, Third Edition, David A. Solomon and Mark E. Russinovich  Windows 2000 Server Operations Guide, Storage, File Systems, and Printing; Chapters: Evaluating Memory and Cache Usage  Advanced Windows, 4th Edition, Jeffrey Richter, Microsoft Press Related Web Sites  http://ntperformance/ Memory Definitions Memory Definitions Before we look at how SQL Server uses and manages its memory, we need to ensure a full understanding of the more common memory related terms. The following definitions will help you understand how SQL Server interacts with the operating system when allocating and using memory. Virtual Address Space A set of memory addresses that are mapped to physical memory addresses by the system. In a 32-bit operation system, there is normally a linear array of 2^32 addresses representing 4,294,967,269 byte addresses. Physical Memory A series of physical locations, with unique addresses, that can be used to store instructions or data. AWE – Address Windowing Extensions A 32-bit process is normally limited to addressing 2 gigabytes (GB) of memory, or 3 GB if the system was booted using the /3G boot switch even if there is more physical memory available. By leveraging the Address Windowing Extensions API, an application can create a fixed-size window into the additional physical memory. This allows a process to access any portion of the physical memory by mapping it into the applications window. When used in combination with Intel’s Physical Addressing Extensions (PAE) on Windows 2000, an AWE enabled application can support up to 64 GB of memory Reserved Memory Pages in a processes address space are free, reserved or committed. Reserving memory address space is a way to reserve a range of virtual addresses for later use. If you attempt to access a reserved address that has not yet been committed (backed by memory or disk) you will cause an access violation. Committed Memory Committed pages are those pages that when accessed in the end translate to pages in memory. Those pages may however have to be faulted in from a page file or memory mapped file. Backing Store Backing store is the physical representation of a memory address. Page Fault (Soft/Hard) A reference to an invalid page (a page that is not in your working set) is referred to as a page fault. Assuming the page reference does not result in an access violation, a page fault can be either hard or soft. A hard page fault results in a read from disk, either a page file or memory-mapped file. A soft page fault is resolved from one of the modified, standby, free or zero page transition lists. Paging is represented by a number of counters including page faults/sec, page input/sec and page output/sec. Page faults/sec include soft and hard page faults where as the page input/output counters represent hard page faults. Unfortunately, all of these counters include file system cache activity. For more information, see also…Inside Windows 2000,Third Edition, pp. 443-451. Private Bytes Private non-shared committed address space Working Set The subset of processes virtual pages that is resident in physical memory. For more information, see also… Inside Windows 2000,Third Edition, p. 455. System Working Set Like a process, the system has a working set. Five different types of pages represent the system’s working set: system cache; paged pool; pageable code and data in the kernel; page-able code and data in device drivers; and system mapped views. The system working set is represented by the counter Memory: cache bytes. System working set paging activity can be viewed by monitoring the Memory: Cache Faults/sec counter. For more information, see also… Inside Windows 2000,Third Edition, p. 463. System Cache The Windows 2000 cache manager provides data caching for both local and network file system drivers. By caching virtual blocks, the cache manager can reduce disk I/O and provide intelligent read ahead. Represented by Memory:Cache Resident bytes. For more information, see also… Inside Windows 2000,Third Edition, pp. 654-659. Non Paged Pool Range of addresses guaranteed to be resident in physical memory. As such, non-paged pool can be accessed at any time without incurring a page fault. Because device drivers operate at DPC/dispatch level (covered in lesson 2), and page faults are not allowed at this level or above, most device drivers use non-paged pool to assure that they do not incur a page fault. Represented by Memory: Pool Nonpaged Bytes, typically between 3-30 megabytes (MB) in size. Note The pool is, in effect, a common area of memory shared by all processes. One of the most common uses of non-paged pool is the storage of object handles. For more information regarding “maximums,” see also… Inside Windows 2000,Third Edition, pp. 403-404 Paged Pool Range of address that can be paged in and out of physical memory. Typically used by drivers who need memory but do not need to access that memory from DPC/dispatch of above interrupt level. Represented by Memory: Pool Paged Bytes and Memory:Pool Paged Resident Bytes. Typically between 10-30MB + size of Registry. For more information regarding “limits,” see also… Inside Windows 2000,Third Edition, pp. 403-404. Stack Each thread has two stacks, one for kernel mode and one for user mode. A stack is an area of memory in which program procedure or function call addresses and parameters are temporarily stored. In Process To run in the same address space. In-process servers are loaded in the client’s address space because they are implemented as DLLs. The main advantage of running in-process is that the system usually does not need to perform a context switch. The disadvantage to running in-process is that DLL has access to the process address space and can potentially cause problems. Out of Process To run outside the calling processes address space. OLEDB providers can run in-process or out of process. When running out of process, they run under the context of DLLHOST.EXE. Memory Leak To reserve or commit memory and unintentionally not release it when it is no longer being used. A process can leak resources such as process memory, pool memory, user and GDI objects, handles, threads, and so on. Memory Concepts (X86 Address Space) Per Process Address Space Every process has its own private virtual address space. For 32-bit processes, that address space is 4 GB, based on a 32-bit pointer. Each process’s virtual address space is split into user and system partitions based on the underlying operating system. The diagram included at the top represents the address partitioning for the 32-bit version of Windows 2000. Typically, the process address space is evenly divided into two 2-GB regions. Each process has access to 2 GB of the 4 GB address space. The upper 2 GB of address space is reserved for the system. The user address space is where application code, global variables, per-thread stacks, and DLL code would reside. The system address space is where the kernel, executive, HAL, boot drivers, page tables, pool, and system cache reside. For specific information regarding address space layout, refer to Inside Microsoft Windows 2000 Third Edition pages 417-428 by Microsoft Press. Access Modes Each virtual memory address is tagged as to what access mode the processor must be running in. System space can only be accessed while in kernel mode, while user space is accessible in user mode. This protects system space from being tampered with by user mode code. Shared System Space Although every process has its own private memory space, kernel mode code and drivers share system space. Windows 2000 does not provide any protection to private memory being use by components running in kernel mode. As such, it is very important to ensure components running in kernel mode are thoroughly tested. 3-GB Address Space 3-GB Address Space Although 2 GB of address space may seem like a large amount of memory, application such as SQL Server could leverage more memory if it were available. The boot.ini option /3GB was created for those cases where systems actually support greater than 2 GB of physical memory and an application can make use of it This capability allows memory intensive applications running on Windows 2000 Advanced Server to use up to 50 percent more virtual memory on Intel-based computers. Application memory tuning provides more of the computer's virtual memory to applications by providing less virtual memory to the operating system. Although a system having less than 2 GB of physical memory can be booted using the /3G switch, in most cases this is ill-advised. If you restart with the 3 GB switch, also known as 4-Gig Tuning, the amount of non-paged pool is reduced to 128 MB from 256 MB. For a process to access 3 GB of address space, the executable image must have been linked with the /LARGEADDRESSAWARE flag or modified using Imagecfg.exe. It should be pointed out that SQL Server was linked using the /LAREGEADDRESSAWARE flag and can leverage 3 GB when enabled. Note Even though you can boot Windows 2000 Professional or Windows 2000 Server with the /3GB boot option, users processes are still limited to 2 GB of address space even if the IMAGE_FILE_LARGE_ADDRESS_AWARE flag is set in the image. The only thing accomplished by using the /3G option on these system is the reduction in the amount of address space available to the system (ISW2K Pg. 418). Important If you use /3GB in conjunction with AWE/PAE you are limited to 16 GB of memory. For more information, see the following Knowledge Base articles: Q171793 Information on Application Use of 4GT RAM Tuning Q126402 PagedPoolSize and NonPagedPoolSize Values in Windows NT Q247904 How to Configure Paged Pool and System PTE Memory Areas Q274598 W2K Does Not Enable Complete Memory Dumps Between 2 & 4 GB AWE Memory Layout AWE Memory Usually, the operation system is limited to 4 GB of physical memory. However, by leveraging PAE, Windows 2000 Advanced Server can support up to 8 GB of memory, and Data Center 64 GB of memory. However, as stated previously, each 32-bit process normally has access to only 2 GB of address space, or 3 GB if the system was booted with the /3-GB option. To allow processes to allocate more physical memory than can be represented in the 2GB of address space, Microsoft created the Address Windows Extensions (AWE). These extensions allow for the allocation and use of up to the amount of physical memory supported by the operating system. By leveraging the Address Windowing Extensions API, an application can create a fixed-size window into the physical memory. This allows a process to access any portion of the physical memory by mapping regions of physical memory in and out of the applications window. The allocation and use of AWE memory is accomplished by  Creating a window via VirtualAlloc using the MEM_PHYSICAL option  Allocating the physical pages through AllocateUserPhysicalPages  Mapping the RAM pages to the window using MapUserPhysicalPages Note SQL Server 7.0 supports a feature called extended memory in Windows NT® 4 Enterprise Edition by using a PSE36 driver. Currently there are no PSE drivers for Windows 2000. The preferred method of accessing extended memory is via the Physical Addressing Extensions using AWE. The AWE mapping feature is much more efficient than the older process of coping buffers from extended memory into the process address space. Unfortunately, SQL Server 7.0 cannot leverage PAE/AWE. Because there are currently no PSE36 drivers for Windows 2000 this means SQL Server 7.0 cannot support more than 3GB of memory on Windows 2000. Refer to KB article Q278466. AWE restrictions  The process must have Lock Pages In Memory user rights to use AWE Important It is important that you use Enterprise Manager or DMO to change the service account. Enterprise Manager and DMO will grant all of the privileges and Registry and file permissions needed for SQL Server. The Service Control Panel does NOT grant all the rights or permissions needed to run SQL Server.  Pages are not shareable or page-able  Page protection is limited to read/write  The same physical page cannot be mapped into two separate AWE regions, even within the same process.  The use of AWE/PAE in conjunction with /3GB will limit the maximum amount of supported memory to between 12-16 GB of memory.  Task manager does not show the correct amount of memory allocated to AWE-enabled applications. You must use Memory Manager: Total Server Memory. It should, however, be noted that this only shows memory in use by the buffer pool.  Machines that have PAE enabled will not dump user mode memory. If an event occurs in User Mode Memory that causes a blue screen and root cause determination is absolutely necessary, the machine must be booted with the /NOPAE switch, and with /MAXMEM set to a number appropriate for transferring dump files.  With AWE enabled, SQL Server will, by default, allocate almost all memory during startup, leaving 256 MB or less free. This memory is locked and cannot be paged out. Consuming all available memory may prevent other applications or SQL Server instances from starting. Note PAE is not required to leverage AWE. However, if you have more than 4GB of physical memory you will not be able to access it unless you enable PAE. Caution It is highly recommended that you use the “max server memory” option in combination with “awe enabled” to ensure some memory headroom exists for other applications or instances of SQL Server, because AWE memory cannot be shared or paged. For more information, see the following Knowledge Base articles: Q268363 Intel Physical Addressing Extensions (PAE) in Windows 2000 Q241046 Cannot Create a dump File on Computers with over 4 GB RAM Q255600 Windows 2000 utilities do not display physical memory above 4GB Q274750 How to configure SQL Server memory more than 2 GB (Idea) Q266251 Memory dump stalls when PAE option is enabled (Idea) Tip The KB will return more hits if you query on PAE rather than AWE. Virtual Address Space Mapping Virtual Address Space Mapping By default Windows 2000 (on an X86 platform) uses a two-level (three-level when PAE is enabled) page table structure to translate virtual addresses to physical addresses. Each 32-bit address has three components, as shown below. When a process accesses a virtual address the system must first locate the Page Directory for the current process via register CR3 (X86). The first 10 bits of the virtual address act as an index into the Page Directory. The Page Directory Entry then points to the Page Frame Number (PFN) of the appropriate Page Table. The next 10 bits of the virtual address act as an index into the Page Table to locate the appropriate page. If the page is valid, the PTE contains the PFN of the actual page in memory. If the page is not valid, the memory management fault handler locates the page and attempts to make it valid. The final 12 bits act as a byte offset into the page. Note This multi-step process is expensive. This is why systems have translation look aside buffers (TLB) to speed up the process. One of the reasons context switching is so expensive is the translation buffers must be dumped. Thus, the first few lookups are very expensive. Refer to ISW2K pages 439-440. Core System Memory Related Counters Core System Memory Related Counters When evaluating memory performance you are looking at a wide variety of counters. The counters listed here are a few of the core counters that give you quick overall view of the state of memory. The two key counters are Available Bytes and Committed Bytes. If Committed Bytes exceeds the amount of physical memory in the system, you can be assured that there is some level of hard page fault activity happening. The goal of a well-tuned system is to have as little hard paging as possible. If Available Bytes is below 5 MB, you should investigate why. If Available Bytes is below 4 MB, the Working Set Manager will start to aggressively trim the working sets of process including the system cache.  Committed Bytes Total memory, including physical and page file currently committed  Commit Limit • Physical memory + page file size • Represents the total amount of memory that can be committed without expanding the page file. (Assuming page file is allowed to grow)  Available Bytes Total physical memory currently available Note Available Bytes is a key indicator of the amount of memory pressure. Windows 2000 will attempt to keep this above approximately 4 MB by aggressively trimming the working sets including system cache. If this value is constantly between 3-4 MB, it is cause for investigation. One counter you might expect would be for total physical memory. Unfortunately, there is no specific counter for total physical memory. There are however many other ways to determine total physical memory. One of the most common is by viewing the Performance tab of Task Manager. Page File Usage The only counters that show current page file space usage are Page File:% Usage and Page File:% Peak Usage. These two counters will give you an indication of the amount of space currently used in the page file. Memory Performance Memory Counters There are a number of counters that you need to investigate when evaluating memory performance. As stated previously, no single counter provides the entire picture. You will need to consider many different counters to begin to understand the true state of memory. Note The counters listed are a subset of the counters you should capture. *Available Bytes In general, it is desirable to see Available Bytes above 5 MB. SQL Servers goal on Intel platforms, running Windows NT, is to assure there is approximately 5+ MB of free memory. After Available Bytes reaches 4 MB, the Working Set Manager will start to aggressively trim the working sets of process and, finally, the system cache. This is not to say that working set trimming does not happen before 4 MB, but it does become more pronounced as the number of available bytes decreases below 4 MB. Page Faults/sec Page Faults/sec represents the total number of hard and soft page faults. This value includes the System Working Set as well. Keep this in mind when evaluating the amount of paging activity in the system. Because this counter includes paging associated with the System Cache, a server acting as a file server may have a much higher value than a dedicated SQL Server may have. The System Working Set is covered in depth on the next slide. Because Page Faults/sec includes soft faults, this counter is not as useful as Pages/sec, which represents hard page faults. Because of the associated I/O, hard page faults tend to be much more expensive. *Pages/sec Pages/sec represent the number of pages written/read from disk because of hard page faults. It is the sum of Memory: Pages Input/sec and Memory: Pages Output/sec. Because it is counted in numbers of pages, it can be compared to other counts of pages, such as Memory: Page Faults/sec, without conversion. On a well-tuned system, this value should be consistently low. In and of itself, a high value for this counter does not necessarily indicate a problem. You will need to isolate the paging activity to determine if it is associated with in-paging, out-paging, memory mapped file activity or system cache. Any one of these activities will contribute to this counter. Note Paging in and of itself is not necessarily a bad thing. Paging is only “bad” when a critical process must wait for it’s pages to be in-paged, or when the amount of read/write paging is causing excessive kernel time or disk I/O, thus interfering with normal user mode processing. Tip (Memory: Pages/sec) / (PhysicalDisk: Disk Bytes/sec * 4096) yields the approximate percentage of paging to total disk I/O. Note, this is only relevant on X86 platforms with a 4 KB page size. Page Reads/sec (Hard Page Fault) Page Reads/sec is the number of times the disk was accessed to resolve hard page faults. It includes reads to satisfy faults in the file system cache (usually requested by applications) and in non-cached memory mapped files. This counter counts numbers of read operations, without regard to the numbers of pages retrieved by each operation. This counter displays the difference between the values observed in the last two samples, divided by the duration of the sample interval. Page Writes/sec (Hard Page Fault) Page Writes/sec is the number of times pages were written to disk to free up space in physical memory. Pages are written to disk only if they are changed while in physical memory, so they are likely to hold data, not code. This counter counts write operations, without regard to the number of pages written in each operation. This counter displays the difference between the values observed in the last two samples, divided by the duration of the sample interval. *Pages Input/sec (Hard Page Fault) Pages Input/sec is the number of pages read from disk to resolve hard page faults. It includes pages retrieved to satisfy faults in the file system cache and in non-cached memory mapped files. This counter counts numbers of pages, and can be compared to other counts of pages, such as Memory:Page Faults/sec, without conversion. This counter displays the difference between the values observed in the last two samples, divided by the duration of the sample interval. This is one of the key counters to monitor for potential performance complaints. Because a process must wait for a read page fault this counter, read page faults have a direct impact on the perceived performance of a process. *Pages Output/sec (Hard Page Fault) Pages Output/sec is the number of pages written to disk to free up space in physical memory. Pages are written back to disk only if they are changed in physical memory, so they are likely to hold data, not code. A high rate of pages output might indicate a memory shortage. Windows NT writes more pages back to disk to free up space when physical memory is in short supply. This counter counts numbers of pages, and can be compared to other counts of pages, without conversion. This counter displays the difference between the values observed in the last two samples, divided by the duration of the sample interval. Like Pages Input/sec, this is one of the key counters to monitor. Processes will generally not notice write page faults unless the disk I/O begins to interfere with normal data operations. Demand Zero Faults/Sec (Soft Page Fault) Demand Zero Faults/sec is the number of page faults that require a zeroed page to satisfy the fault. Zeroed pages, pages emptied of previously stored data and filled with zeros, are a security feature of Windows NT. Windows NT maintains a list of zeroed pages to accelerate this process. This counter counts numbers of faults, without regard to the numbers of pages retrieved to satisfy the fault. This counter displays the difference between the values observed in the last two samples, divided by the duration of the sample interval. Transition Faults/Sec (Soft Page Fault) Transition Faults/sec is the number of page faults resolved by recovering pages that were on the modified page list, on the standby list, or being written to disk at the time of the page fault. The pages were recovered without additional disk activity. Transition faults are counted in numbers of faults, without regard for the number of pages faulted in each operation. This counter displays the difference between the values observed in the last two samples, divided by the duration of the sample interval. System Working Set System Working Set Like processes, the system page-able code and data are managed by a working set. For the purpose of this course, that working set is referred to as the System Working Set. This is done to differentiate the system cache portion of the working set from the entire working set. There are five different types of pages that make up the System Working Set. They are: system cache; paged pool; page-able code and data in ntoskrnl.exe; page-able code, and data in device drivers and system-mapped views. Unfortunately, some of the counters that appear to represent the system cache actually represent the entire system working set. Where noted system cache actually represents the entire system working set. Note The counters listed are a subset of the counters you should capture. *Memory: Cache Bytes (Represents Total System Working Set) Represents the total size of the System Working Set including: system cache; paged pool; pageable code and data in ntoskrnl.exe; pageable code and data in device drivers; and system-mapped views. Cache Bytes is the sum of the following counters: System Cache Resident Bytes, System Driver Resident Bytes, System Code Resident Bytes, and Pool Paged Resident Bytes. Memory: System Cache Resident Bytes (System Cache) System Cache Resident Bytes is the number of bytes from the file system cache that are resident in physical memory. Windows 2000 Cache Manager works with the memory manager to provide virtual block stream and file data caching. For more information, see also…Inside Windows 2000,Third Edition, pp. 645-650 and p. 656. Memory: Pool Paged Resident Bytes Represents the physical memory consumed by Paged Pool. This counter should NOT be monitored by itself. You must also monitor Memory: Paged Pool. A leak in the pool may not show up in Pool paged Resident Bytes. Memory: System Driver Resident Bytes Represents the physical memory consumed by driver code and data. System Driver Resident Bytes and System Driver Total Bytes do not include code that must remain in physical memory and cannot be written to disk. Memory: System Code Resident Bytes Represents the physical memory consumed by page-able system code. System Code Resident Bytes and System Code Total Bytes do not include code that must remain in physical memory and cannot be written to disk. Working Set Performance Counter You can measure the number of page faults in the System Working Set by monitoring the Memory: Cache Faults/sec counter. Contrary to the “Explain” shown in System Monitor, this counter measures the total amount of page faults/sec in the System Working Set, not only the System Cache. You cannot measure the performance of the System Cache using this counter alone. For more information, see also…Inside Windows 2000,Third Edition, p. 656. Note You will find that in general the working set manager will usually trim the working sets of normal processes prior to trimming the system working set. System Cache System Cache The Windows 2000 cache manager provides a write-back cache with lazy writing and intelligent read-ahead. Files are not written to disk immediately but differed until the cache manager calls the memory manager to flush the cache. This helps to reduce the total number of I/Os. Once per second, the lazy writer thread queues one-eighth of the dirty pages in the system cache to be written to disk. If this is not sufficient to meet the needs, the lazy writer will calculate a larger value. If the dirty page threshold is exceeded prior to lazy writer waking, the cache manager will wake the lazy writer. Important It should be pointed out that mapped files or files opened with FILE_FLAG_NO_BUFFERING, do not participate in the System Cache. For more information regarding mapped views, see also…Inside Windows 2000,Third Edition, p. 669. For those applications that would like to leverage system cache but cannot tolerate write delays, the cache manager supports write through operations via the FILE_FLAG_WRITE_THROUGH. On the other hand, an application can disable lazy writing by using the FILE_ATTRIBUTE_TEMPORARY. If this flag is enabled, the lazy writer will not write the pages to disk unless there is a shortage of memory or the file is closed. Important Microsoft SQL Server uses both FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH Tip The file system cache is not represented by a static amount of memory. The system cache can and will grow. It is not unusual to see the system cache consume a large amount of memory. Like other working sets, it is trimmed under pressure but is generally the last thing to be trimmed. System Cache Performance Counters The counters listed are a subset of the counters you should capture. Cache: Data Flushes/sec Data Flushes/sec is the rate at which the file system cache has flushed its contents to disk as the result of a request to flush or to satisfy a write-through file write request. More than one page can be transferred on each flush operation. Cache: Data Flush Pages/sec Data Flush Pages/sec is the number of pages the file system cache has flushed to disk as a result of a request to flush or to satisfy a write-through file write request. Cache: Lazy Write Flushes/sec Represents the rate of lazy writes to flush the system cache per second. More than one page can be transferred per second. Cache: Lazy Write Pages/sec Lazy Write Pages/sec is the rate at which the Lazy Writer thread has written to disk. Note When looking at Memory:Cache Faults/sec, you can remove cache write activity by subtracting (Cache: Data Flush Pages/sec + Cache: Lazy Write Pages/sec). This will give you a better idea of how much other page faulting activity is associated with the other components of the System Working Set. However, you should note that there is no easy way to remove the page faults associated with file cache read activity. For more information, see the following Knowledge Base articles: Q145952 (NT4) Event ID 26 Appears If Large File Transfer Fails Q163401 (NT4) How to Disable Network Redirector File Caching Q181073 (SQL 6.5) DUMP May Cause Access Violation on Win2000 System Pool System Pool As documented earlier, there are two types of shared pool memory: non-paged pool and paged pool. Like private memory, pool memory is susceptible to a leak. Nonpaged Pool Miscellaneous kernel code and structures, and drivers that need working memory while at or above DPC/dispatch level use non-paged pool. The primary counter for non-paged pool is Memory: Pool Nonpaged Bytes. This counter will usually between 3 and 30 MB. Paged Pool Drivers that do not need to access memory above DPC/Dispatch level are one of the primary users of paged pool, however any process can use paged pool by leveraging the ExAllocatePool calls. Paged pool also contains the Registry and file and printing structures. The primary counters for monitoring paged pool is Memory: Pool Paged Bytes. This counter will usually be between 10-30MB plus the size of the Registry. To determine how much of paged pool is currently resident in physical memory, monitor Memory: Pool Paged Resident Bytes. Note The paged and non-paged pools are two of the components of the System Working Set. If a suspected leak is clearly visible in the overview and not associated with a process, then it is most likely a pool leak. If the leak is not associated with SQL Server handles, OLDEB providers, XPROCS or SP_OA calls then most likely this call should be pushed to the Windows NT group. For more information, see the following Knowledge Base articles: Q265028 (MS) Pool Tags Q258793 (MS) How to Find Memory Leaks by Using Pool Bitmap Analysis Q115280 (MS) Finding Windows NT Kernel Mode Memory Leaks Q177415 (MS) How to Use Poolmon to Troubleshoot Kernel Mode Memory Leaks Q126402 PagedPoolSize and NonPagedPoolSize Values in Windows NT Q247904 How to Configure Paged Pool and System PTE Memory Areas Tip To isolate pool leaks you will need to isolate all drivers and third-party processes. This should be done by disabling each service or driver one at a time and monitoring the effect. You can also monitor paged and non-paged pool through poolmon. If pool tagging has been enabled via GFLAGS, you may be able to associate the leak to a particular tag. If you suspect a particular tag, you should involve the platform support group. Process Memory Counters Process _Total Limitations Although the rollup of _Total for Process: Private Bytes, Virtual Bytes, Handles and Threads, represent the key resources being used across all processes, they can be misleading when evaluating a memory leak. This is because a leak in one process may be masked by a decrease in another process. Note The counters listed are a subset of the counters you should capture. Tip When analyzing memory leaks, it is often easier to a build either a separate chart or report showing only one or two key counters for all process. The primary counter used for leak analysis is private bytes, but processes can leak handles and threads just as easily. After a suspect process is located, build a separate chart that includes all the counters for that process. Individual Process Counters When analyzing individual process for memory leaks you should include the counters listed.  Process: % Processor Time  Process: Working Set (includes shared pages)  Process: Virtual Bytes  Process: Private Bytes  Process: Page Faults/sec  Process: Handle Count  Process: Thread Count  Process: Pool Paged Bytes  Process: Pool Nonpaged Bytes Tip WINLOGON, SVCHOST, services, or SPOOLSV are referred to as HELPER processes. They provide core functionality for many operations and as such are often extended by the addition of third-party DLLs. Tlist –s may help identify what services are running under a particular helper. Helper Processes Helper Processes Winlogon, Services, and Spoolsv and Svchost are examples of what are referred to as HELPER processes. They provide core functionality for many operations and, as such, are often extended by the addition of third-party DLLs. Running every service in its own process can waste system resources. Consequently, some services run in their own processes while others share a process with other services. One problem with sharing a process is that a bug in one service may cause the entire process to fail. The resource kit tool, Tlist when used with the –s qualifier can help you identify what services are running in what processes. WINLOGON Used to support GINAs. SPOOLSV SPOOLSV is responsible for printing. You will need to investigate all added printing functionality. Services Service is responsible for system services. Svchost.exe Svchost.exe is a generic host process name for services that are run from dynamic-link libraries (DLLs). There can be multiple instances of Svchost.exe running at the same time. Each Svchost.exe session can contain a grouping of services, so that separate services can be run depending on how and where Svchost.exe is started. This allows for better control and debugging. The Effect of Memory on Other Components Memory Drives Overall Performance Processor, cache, bus speeds, I/O, all of these resources play a roll in overall perceived performance. Without minimizing the impact of these components, it is important to point out that a shortage of memory can often have a larger perceived impact on performance than a shortage of some other resource. On the other hand, an abundance of memory can often be leveraged to mask bottlenecks. For instance, in certain environments, file system cache can significantly reduce the amount of disk I/O, potentially masking a slow I/O subsystem. Effect on I/O I/O can be driven by a number of memory considerations. Page read/faults will cause a read I/O when a page is not in memory. If the modified page list becomes too long the Modified Page Writer and Mapped Page Writer will need to start flushing pages causing disk writes. However, the one event that can have the greatest impact is running low on available memory. In this case, all of the above events will become more pronounced and have a larger impact on disk activity. Effect on CPU The most effective use of a processor from a process perspective is to spend as much time possible executing user mode code. Kernel mode represents processor time associated with doing work, directly or indirectly, on behalf of a thread. This includes items such as synchronization, scheduling, I/O, memory management, and so on. Although this work is essential, it takes processor cycles and the cost, in cycles, to transition between user and kernel mode is expensive. Because all memory management and I/O functions must be done in kernel mode, it follows that the fewer the memory resources the more cycles are going to be spent managing those resources. A direct result of low memory is that the Working Set Manager, Modified Page Writer and Mapped Page Writer will have to use more cycles attempting to free memory. Analyzing Memory Look for Trends and Trend Relationships Troubleshooting performance is about analyzing trends and trend relationships. Establishing that some event happened is not enough. You must establish the effect of the event. For example, you note that paging activity is high at the same time that SQL Server becomes slow. These two individual facts may or may not be related. If the paging is not associated with SQL Servers working set, or the disks SQL is using there may be little or no cause/affect relationship. Look at Physical Memory First The first item to look at is physical memory. You need to know how much physical and page file space the system has to work with. You should then evaluate how much available memory there is. Just because the system has free memory does not mean that there is not any memory pressure. Available Bytes in combination with Pages Input/sec and Pages Output/sec can be a good indicator as to the amount of pressure. The goal in a perfect world is to have as little hard paging activity as possible with available memory greater than 5 MB. This is not to say that paging is bad. On the contrary, paging is a very effective way to manage a limited resource. Again, we are looking for trends that we can use to establish relationships. After evaluating physical memory, you should be able to answer the following questions:  How much physical memory do I have?  What is the commit limit?  Of that physical memory, how much has the operating system committed?  Is the operating system over committing physical memory?  What was the peak commit charge?  How much available physical memory is there?  What is the trend associated with committed and available? Review System Cache and Pool Contribution After you understand the individual process memory usage, you need to evaluate the System Cache and Pool usage. These can and often represent a significant portion of physical memory. Be aware that System Cache can grow significantly on a file server. This is usually normal. One thing to consider is that the file system cache tends to be the last thing trimmed when memory becomes low. If you see abrupt decreases in System Cache Resident Bytes when Available Bytes is below 5 MB you can be assured that the system is experiencing excessive memory pressure. Paged and non-paged pool size is also important to consider. An ever-increasing pool should be an indicator for further research. Non-paged pool growth is usually a driver issue, while paged pool could be driver-related or process-related. If paged pool is steadily growing, you should investigate each process to see if there is a specific process relationship. If not you will have to use tools such as poolmon to investigate further. Review Process Memory Usage After you understand the physical memory limitations and cache and pool contribution you need to determine what components or processes are creating the pressure on memory, if any. Be careful if you opt to chart the _Total Private Byte’s rollup for all processes. This value can be misleading in that it includes shared pages and can therefore exceed the actual amount of memory being used by the processes. The _Total rollup can also mask processes that are leaking memory because other processes may be freeing memory thus creating a balance between leaked and freed memory. Identify processes that expand their working set over time for further analysis. Also, review handles and threads because both use resources and potentially can be mismanaged. After evaluating the process resource usage, you should be able to answer the following:  Are any of the processes increasing their private bytes over time?  Are any processes growing their working set over time?  Are any processes increasing the number of threads or handles over time?  Are any processes increasing their use of pool over time?  Is there a direct relationship between the above named resources and total committed memory or available memory?  If there is a relationship, is this normal behavior for the process in question? For example, SQL does not commit ‘min memory’ on startup; these pages are faulted in into the working set as needed. This is not necessarily an indication of a memory leak.  If there is clearly a leak in the overview and is not identifiable in the process counters it is most likely in the pool.  If the leak in pool is not associated with SQL Server handles, then more often than not, it is not a SQL Server issue. There is however the possibility that the leak could be associated with third party XPROCS, SP_OA* calls or OLDB providers. Review Paging Activity and Its Impact on CPU and I/O As stated earlier, paging is not in and of itself a bad thing. When starting a process the system faults in the pages of an executable, as they are needed. This is preferable to loading the entire image at startup. The same can be said for memory mapped files and file system cache. All of these features leverage the ability of the system to fault in pages as needed The greatest impact of paging on a process is when the process must wait for an in-page fault or when page file activity represents a significant portion of the disk activity on the disk the application is actively using. After evaluating page fault activity, you should be able to answer the following questions:  What is the relationship between PageFaults/sec and Page Input/sec + Page Output/Sec?  What is the relationship if any between hard page faults and available memory?  Does paging activity represent a significant portion of processor or I/O resource usage? Don’t Prematurely Jump to Any Conclusions Analyzing memory pressure takes time and patience. An individual counter in and of it self means little. It is only when you start to explore relationships between cause and effect that you can begin to understand the impact of a particular counter. The key thoughts to remember are:  With the exception of a swap (when the entire process’s working set has been swapped out/in), hard page faults to resolve reads, are the most expensive in terms its effect on a processes perceived performance.  In general, page writes associated with page faults do not directly affect a process’s perceived performance, unless that process is waiting on a free page to be made available. Page file activity can become a problem if that activity competes for a significant percentage of the disk throughput in a heavy I/O orientated environment. That assumes of course that the page file resides on the same disk the application is using. Lab 3.1 System Memory Lab 3.1 Analyzing System Memory Using System Monitor Exercise 1 – Troubleshooting the Cardinal1.log File Students will evaluate an existing System Monitor log and determine if there is a problem and what the problem is. Students should be able to isolate the issue as a memory problem, locate the offending process, and determine whether or not this is a pool issue. Exercise 2 – Leakyapp Behavior Students will start leaky app and monitor memory, page file and cache counters to better understand the dynamics of these counters. Exercise 3 – Process Swap Due To Minimizing of the Cmd Window Students will start SQL from command line while viewing SQL process performance counters. Students will then minimize the window and note the effect on the working set. Overview What You Will Learn After completing this lab, you will be able to:  Use some of the basic functions within System Monitor.  Troubleshoot one or more common performance scenarios. Before You Begin Prerequisites To complete this lab, you need the following:  Windows 2000  SQL Server 2000  Lab Files Provided  LeakyApp.exe (Resource Kit) Estimated time to complete this lab: 45 minutes Exercise 1 Troubleshooting the Cardinal1.log File In this exercise, you will analyze a log file from an actual system that was having performance problems. Like an actual support engineer, you will not have much information from which to draw conclusions. The customer has sent you this log file and it is up to you to find the cause of the problem. However, unlike the real world, you have an instructor available to give you hints should you become stuck. Goal Review the Cardinal1.log file (this file is from Windows NT 4.0 Performance Monitor, which Windows 2000 can read). Chart the log file and begin to investigate the counters to determine what is causing the performance problems. Your goal should be to isolate the problem to a major area such as pool, virtual address space etc, and begin to isolate the problem to a specific process or thread. This lab requires access to the log file Cardinal1.log located in C:\LABS\M3\LAB1\EX1  To analyze the log file 1. Using the Performance MMC, select the System Monitor snap-in, and click the View Log File Data button (icon looks like a disk). 2. Under Files of type, choose PERFMON Log Files (*.log) 3. Navigate to the folder containing Cardinal1.log file and open it. 4. Begin examining counters to find what might be causing the performance problems. When examining some of these counters, you may notice that some of them go off the top of the chart. It may be necessary to adjust the scale on these. This can be done by right-clicking the rightmost pane and selecting Properties. Select the Data tab. Select the counter that you wish to modify. Under the Scale option, change the scale value, which makes the counter data visible on the chart. You may need to experiment with different scale values before finding the ideal value. Also, it may sometimes be beneficial to adjust the vertical scale for the entire chart. Selecting the Graph tab on the Properties page can do this. In the Vertical scale area, adjust the Maximum and Minimum values to best fit the data on the chart. Lab 3.1, Exercise 1: Results Exercise 2 LeakyApp Behavior In this lab, you will have an opportunity to work with a partner to monitor a live system, which is suffering from a simulated memory leak. Goal During this lab, your goal is to observe the system behavior when memory starts to become a limited resource. Specifically you will want to monitor committed memory, available memory, the system working set including the file system cache and each processes working set. At the end of the lab, you should be able to provide an answer to the listed questions.  To monitor a live system with a memory leak 1. Choose one of the two systems as a victim on which to run the leakyapp.exe program. It is recommended that you boot using the \MAXMEM=128 option so that this lab goes a little faster. You and your partner should decide which server will play the role of the problematic server and which server is to be used for monitoring purposes. 2. On the problematic server, start the leakyapp program. 3. On the monitoring system, create a counter that logs all necessary counters need to troubleshoot a memory problem. This should include physicaldisk counters if you think paging is a problem. Because it is likely that you will only need to capture less than five minutes of activity, the suggested interval for capturing is five seconds. 4. After the counters have been started, start the leaky application program 5. Click Start Leaking. The button will now change to Stop Leaking, which indicates that the system is now leaking memory. 6. After leakyapp shows the page file is 50 percent full, click Stop leaking. Note that the process has not given back its memory, yet. After approximately one minute, exit. Lab 3.1, Exercise 2: Questions After analyzing the counter logs you should be able to answer the following: 1. Under which system memory counter does the leak show up clearly? Memory:Committed Bytes 2. What process counter looked very similar to the overall system counter that showed the leak? Private Bytes 3. Is the leak in Paged Pool, Non-paged pool, or elsewhere? Elsewhere 4. At what point did Windows 2000 start to aggressively trim the working sets of all user processes? <5 MB Free 5. Was the System Working Set trimmed before or after the working sets of other processes? After 6. What counter showed this? Memory:Cache Bytes 7. At what point was the File System Cache trimmed? After the first pass through all other working sets 8. What was the effect on all the processes working set when the application quit leaking? None 9. What was the effect on all the working sets when the application exited? Nothing, initially; but all grew fairly quickly based on use 10. When the server was running low on memory, which was Windows spending more time doing, paging to disk or in-paging? Paging to disk, initially; however, as other applications began to run, in-paging increased Exercise 3 Minimizing a Command Window In this exercise, you will have an opportunity to observe the behavior of Windows 2000 when a command window is minimized. Goal During this lab, your goal is to observe the behavior of Windows 2000 when a command window becomes minimized. Specifically, you will want to monitor private bytes, virtual bytes, and working set of SQL Server when the command window is minimized. At the end of the lab, you should be able to provide an answer to the listed questions.  To monitor a command window’s working set as the window is minimized 1. Using System Monitor, create a counter list that logs all necessary counters needed to troubleshoot a memory problem. Because it is likely that you will only need to capture less than five minutes of activity, the suggested capturing interval is five seconds. 2. After the counters have been started, start a Command Prompt window on the target system. 3. In the command window, start SQL Server from the command line. Example: SQL Servr.exe –c –sINSTANCE1 4. After SQL Server has successfully started, Minimize the Command Prompt window. 5. Wait approximately two minutes, and then Restore the window. 6. Wait approximately two minutes, and then stop the counter log. Lab 3.1, Exercise 3: Questions After analyzing the counter logs you should be able to answer the following questions: 1. What was the effect on SQL Servers private bytes, virtual bytes, and working set when the window was minimized? Private Bytes and Virtual Bytes remained the same, while Working Set went to 0 2. What was the effect on SQL Servers private bytes, virtual bytes, and working set when the window was restored? None; the Working Set did not grow until SQL accessed the pages and faulted them back in on an as-needed basis SQL Server Memory Overview SQL Server Memory Overview Now that you have a better understanding of how Windows 2000 manages memory resources, you can take a closer look at how SQL Server 2000 manages its memory. During the course of the lecture and labs you will have the opportunity to monitor SQL Servers use of memory under varying conditions using both System Monitor counters and SQL Server tools. SQL Server Memory Management Goals Because SQL Server has in-depth knowledge about the relationships between data and the pages they reside on, it is in a better position to judge when and what pages should be brought into memory, how many pages should be brought in at a time, and how long they should be resident. SQL Servers primary goals for management of its memory are the following:  Be able to dynamically adjust for varying amounts of available memory.  Be able to respond to outside memory pressure from other applications.  Be able to adjust memory dynamically for internal components. Items Covered  SQL Server Memory Definitions  SQL Server Memory Layout  SQL Server Memory Counters  Memory Configurations Options  Buffer Pool Performance and Counters  Set Aside Memory and Counters  General Troubleshooting Process  Memory Myths and Tips SQL Server Memory Definitions SQL Server Memory Definitions Pool A group of resources, objects, or logical components that can service a resource allocation request Cache The management of a pool or resource, the primary goal of which is to increase performance. Bpool The Bpool (Buffer Pool) is a single static class instance. The Bpool is made up of 8-KB buffers and can be used to handle data pages or external memory requests. There are three basic types or categories of committed memory in the Bpool.  Hashed Data Pages  Committed Buffers on the Free List  Buffers known by their owners (Refer to definition of Stolen) Consumer A consumer is a subsystem that uses the Bpool. A consumer can also be a provider to other consumers. There are five consumers and two advanced consumers who are responsible for the different categories of memory. The following list represents the consumers and a partial list of their categories  Connection – Responsible for PSS and ODS memory allocations  General – Resource structures, parse headers, lock manager objects  Utilities – Recovery, Log Manager  Optimizer – Query Optimization  Query Plan – Query Plan Storage Advanced Consumer Along with the five consumers, there are two advanced consumers. They are  Ccache – Procedure cache. Accepts plans from the Optimizer and Query Plan consumers. Is responsible for managing that memory and determines when to release the memory back to the Bpool.  Log Cache – Managed by the LogMgr, which uses the Utility consumer to coordinate memory requests with the Bpool. Reservation Requesting the future use of a resource. A reservation is a reasonable guarantee that the resource will be available in the future. Committed Producing the physical resource Allocation The act of providing the resource to a consumer Stolen The act of getting a buffer from the Bpool is referred to as stealing a buffer. If the buffer is stolen and hashed for a data page, it is referred to as, and counted as, a Hashed buffer, not a stolen buffer. Stolen buffers on the other hand are buffers used for things such as procedure cache and SRV_PROC structures. Target Target memory is the amount of memory SQL Server would like to maintain as committed memory. Target memory is based on the min and max server configuration values and current available memory as reported by the operating system. Actual target calculation is operating system specific. Memory to Leave (Set Aside) The virtual address space set aside to ensure there is sufficient address space for thread stacks, XPROCS, COM objects etc. Hashed Page A page in pool that represents a database page. SQL Server Memory Layout Virtual Address Space When SQL Server is started the minimum of physical ram or virtual address space supported by the OS is evaluated. There are many possible combinations of OS versions and memory configurations. For example: you could be running Microsoft Windows 2000 Advanced Server with 2 GB or possibly 4 GB of memory. To avoid page file use, the appropriate memory level is evaluated for each configuration. Important Utilities can inject a DLL into the process address space by using HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Windows\AppInit_DLLs When the USER32.dll library is mapped into the process space, so, too, are the DLLs listed in the Registry key. To determine what DLL’s are running in SQL Server address space you can use tlist.exe. You can also use a tool such as Depends from Microsoft or HandelEx from http://ww.sysinternals.com. Memory to Leave As stated earlier there are many possible configurations of physical memory and address space. It is possible for physical memory to be greater than virtual address space. To ensure that some virtual address space is always available for things such as thread stacks and external needs such as XPROCS, SQL Server reserves a small portion of virtual address space prior to determining the size of the buffer pool. This address space is referred to as Memory To Leave. Its size is based on the number of anticipated tread stacks and a default value for external needs referred to as cmbAddressSave. After reserving the buffer pool space, the Memory To Leave reservation is released. Buffer Pool Space During Startup, SQL Server must determine the maximum size of the buffer pool so that the BUF, BUFHASH and COMMIT BITMAP structures that are used to manage the Bpool can be created. It is important to understand that SQL Server does not take ‘max memory’ or existing memory pressure into consideration. The reserved address space of the buffer pool remains static for the life of SQL Server process. However, the committed space varies as necessary to provide dynamic scaling. Remember only the committed memory effects the overall memory usage on the machine. This ensures that the max memory configuration setting can be dynamically changed with minimal changes needed to the Bpool. The reserved space does not need to be adjusted and is maximized for the current machine configuration. Only the committed buffers need to be limited to maintain a specified max server memory (MB) setting. SQL Server Startup Pseudo Code The following pseudo code represents the process SQL Server goes through on startup. Warning This example does not represent a completely accurate portrayal of the steps SQL Server takes when initializing the buffer pool. Several details have been left out or glossed over. The intent of this example is to help you understand the general process, not the specific details.  Determine the size of cmbAddressSave (-g)  Determine Total Physical Memory  Determine Available Physical Memory  Determine Total Virtual Memory  Calculate MemToLeave maxworkterthreads * (stacksize=512 KB) + (cmbAddressSave = 256 MB)  Reserve MemToLeave and set PAGE_NOACCESS  Check for AWE, test to see if it makes sense to use it and log the results • Min(Available Memory, Max Server Memory) > Virtual Memory • Supports Read Scatter • SQL Server not started with -f • AWE Enabled via sp_configure • Enterprise Edition • Lock Pages In Memory user right enabled  Calculate Virtual Address Limit VA Limit = Min(Physical Memory, Virtual Memory – MemtoLeave)  Calculate the number of physical and virtual buffers that can be supported AWE Present Physical Buffers = (RAM / (PAGESIZE + Physical Overhead)) Virtual Buffers = (VA Limit / (PAGESIZE + Virtual Overhead)) AWE Not Present Physical Buffers = Virtual Buffers = VA Limit / (PAGESIZE + Physical Overhead + Virtual Overhead)  Make sure we have the minimum number of buffers Physical Buffers = Max(Physical Buffers, MIN_BUFFERS)  Allocate and commit the buffer management structures  Reserve the address space required to support the Bpool buffers  Release the MemToLeave SQL Server Startup Pseudo Code Example The following is an example based on the pseudo code represented on the previous page. This example is based on a machine with 384 MB of physical memory, not using AWE or /3GB. Note CmbAddressSave was changed between SQL Server 7.0 and SQL Server 2000. For SQL Server 7.0, cmbAddressSave was 128. Warning This example does not represent a completely accurate portrayal of the steps SQL Server takes when initializing the buffer pool. Several details have been left out or glossed over. The intent of this example is to help you understand the general process, not the specific details.  Determine the size of cmbAddressSave (No –g so 256MB)  Determine Total Physical Memory (384)  Determine Available Physical Memory (384)  Determine Total Virtual Memory (2GB)  Calculate MemToLeave maxworkterthreads * (stacksize=512 KB) + (cmbAddressSave = 256 MB) (255 * .5MB + 256MB = 384MB)  Reserve MemToLeave and set PAGE_NOACCESS  Check for AWE, test to see if it makes sense to use it and log the results (AWE Not Enabled)  Calculate Virtual Address Limit VA Limit = Min(Physical Memory, Virtual Memory – MemtoLeave) 384MB = Min(384MB, 2GB – 384MB)  Calculate the number of physical and virtual buffers that can be supported AWE Not Present 48664 (approx) = 384 MB / (8 KB + Overhead)  Make sure we have the minimum number of buffers Physical Buffers = Max(Physical Buffers, MIN_BUFFERS) 48664 = Max(48664,1024)  Allocate and commit the buffer management structures  Reserve the address space required to support the Bpool buffers  Release the MemToLeave Tip Trace Flag 1604 can be used to view memory allocations on startup. The cmbAddressSave can be adjusted using the –g XXX startup parameter. SQL Server Memory Counters SQL Server Memory Counters The two primary tools for monitoring and analyzing SQL Server memory usage are System Monitor and DBCC MEMORYSTATUS. For detailed information on DBCC MEMORYSTATUS refer to Q271624 Interpreting the Output of the DBCC MEMORYSTAUS Command. Important Represents SQL Server 2000 Counters. The counters presented are not the same as the counters for SQL Server 7.0. The SQL Server 7.0 counters are listed in the appendix. Determining Memory Usage for OS and BPOOL Memory Manager: Total Server memory (KB) - Represents all of SQL usage Buffer Manager: Total Pages - Represents total bpool usage To determine how much of Total Server Memory (KB) represents MemToLeave space; subtract Buffer Manager: Total Pages. The result can be verified against DBCC MEMORYSTATUS, specifically Dynamic Memory Manager: OS In Use. It should however be noted that this value only represents requests that went thru the bpool. Memory reserved outside of the bpool by components such as COM objects will not show up here, although they will count against SQL Server private byte count. Buffer Counts: Target (Buffer Manager: Target Pages) The size the buffer pool would like to be. If this value is larger than committed, the buffer pool is growing. Buffer Counts: Committed (Buffer Manager: Total Pages) The total number of buffers committed in the OS. This is the current size of the buffer pool. Buffer Counts: Min Free This is the number of pages that the buffer pool tries to keep on the free list. If the free list falls below this value, the buffer pool will attempt to populate it by discarding old pages from the data or procedure cache. Buffer Distribution: Free (Buffer Manager / Buffer Partition: Free Pages) This value represents the buffers currently not in use. These are available for data or may be requested by other components and mar
MSL 即 Microsoft Library 是 DOS 版的 "WinHelp",也就是现代版 Help Viewer 的始祖。 安装目录下有个 ini 文件,用来指定图书的路径,它即是目录。 文件来源自 http://wdl2.winworldpc.com/Abandonware%20SDKs/Microsoft Programmer's Library 1.3.7z Microsoft Programmer's Library 1.3.iso 这就是 DOS 版的 MSDN!使用 DOSBOX 就可以运行此库。此库含一大古董级MS官方编程参考材料,主要针对 Windows 3.0 平台,真可谓之应用尽有: MS Windows 3.0 SDK Guide to Programming MS Windows 3.0 SDK Install. & Update Guide MS Windows 3.0 SDK Programmer's Reference Vol. 1 MS Windows 3.0 SDK Programmer's Reference Vol. 2 MS Windows 3.0 SDK Tools MS Windows 3.0 SDK Articles All MS Windows 3.0 SDK Manuals MS Windows 3.0 DDK Install. & Update Guide MS Windows 3.0 DDK Adaptation Guide MS Windows 3.0 DDK Virtual Device Adapt. Guide MS Windows 3.0 DDK Printer & Font Kit All MS Windows 3.0 DDK Manuals MS Online User's Guide Programming MS Windows MS Windows Sample Code MS KnowledgeBase - MS Windows 以及 Options => Library 菜单下提供的 9 个重要的参考资料,其中就有 C 和 MASM 这些重要的参考资料。这些是已安装的目录部分,鉴于 MASM 的重要性,特将其添加到压缩包内,免CD运行: Windows References OS/S References Network References MS-DOS References MS Systems Journal Hardware References C References MASM References BASIC References Pascal References FORTUAN References 其中 C References 和 MASM References 包含: Installing and Using MS MASM 6.0 MS MASM 6.0 Reference MS MASM 6.0 Programmer's Guide MS MASM 6.0 White Paper QuickAssembler 2.01 Programmer's Guide MS Mixed-Language Programming Guide CodeView & Utilities User's Guide MS Editor User's Guide MS OnLine User's Guide MASM Sample Code MS KnowledgeBase - MASM MS C 6.0 Advanced Programming Techniques MS C 6.0 Installing and Using the P.D.S. MS C 6.0 Reference MS C 6.0 Run-Time Library Reference MS C 6.0 Developer's Toolkit Reference QuickC 2.5 Tool Kit QuickC 2.5 C for Yourself QuickC 2.5 Up and Running QuickC 2.5 Update MS Professional

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值