Understanding the Search Service Architecture

Search is one of the most important services in SharePoint 2013, like it has been throughout theSharePoint product development. This is because it enables users to quickly andeasily find the content held in SharePoint, both from SharePoint  Content Databases and from external datasources, through the Business Connectivity Service. In SharePoint 2013, theunderlying architecture of Search redeveloped, to provide a richer enterprisesearch experience for users and streamlined administration.

 

The Search Architecture

 

TheSearch service has been rearchitected in SharePoint 2013. The architecture changes have been made to ensure that Search offers ahigher level of redundancy forsingle and multiple farm environments.One of the most obvious changes in the new architecture is the inclusion of the FAST technologies into the Search service. Thisenhances search functionalityand makes it easier for solutionarchitects to design and deploy a fullfeatured search solution in their organizations. Search combines a range of elements on application servers, which include:

 

• Crawl Component
• Content Processing Component
• Analytics Processing Component
• Indexing Component
• Query Processing Component
• Search Administration Component

 

Thesecomponents interact to complete search data ingestion to the search index andfor query results surfacing. Toingest data, the Crawl Component interrogates the content sources that you have configured, either on the SharePoint farmor on external sources, such as Microsoft Exchange or Lotus Notes. The crawled items are processed bythe Content Processing Component to format them appropriately to be stored on the index. Information from theAnalytics Processing Component is used in this process, to identify useful associated item information, suchas previous user interaction. The data, or artifacts, are written to the index, which is a series of files andfolders that are stored on disk and referred to, collectively, as the index file. The Query Processing Componentreceives queries from the Web Front End (WFE) server, processes and sends the query to the IndexingComponent, which returns result sets. The Query Processing Component performs additional processing toaggregate and clean the results and then sends the result sets back to the WFE to be rendered for theuser. There are temporary and permanentstorage databases used throughout the process.

 

The Crawl Component

 

TheCrawl Component is a part of the crawl and content processing architecture, which comprises the Crawl Component, the crawl database,and the Content ProcessingComponent. The crawl role is responsiblefor crawling content sources to build a search index. This means that the component reviews each of the documents or pages in a source location.

 

Crawl Connectors:

 

• HTTP
• File Shares
• SharePoint Sites
• User profiles
• Exchange
• Lotus Notes
• Custom

 

TheCrawl Component delivers content from files and pages, together with theirassociated metadata, to theContent Processing Component. The crawled items that are passed to the ContentProcessing Component haveassociated properties, such as title and author. Crawl uses connectors, such asthe Lotus Notes Connector, toaccess content sources and retrieve data. These were known as protocol handlersin previous versions ofSharePoint Products and Technologies. The properties are grouped, based on the connector or IFilter—a piece of code thatenables the specific file formats to be indexed and thus be searchable, now referred to as a formathandler—used to crawl the content source; so, Microsoft Office documents (for example, Word and Exceldocuments) would be grouped under Office, whereas properties from websites would be grouped under Web.You can include the contents and metadata of crawled properties in the search index file. To do this, you must map thecrawled properties to managed properties,because only these are included in the search index.

Thecrawler itself does not complete document parsing; that function is provisionedby the Content ProcessingComponent.

TheCrawl Component uses one or more crawl databases to temporarily storeinformation about crawled itemsand to maintain a crawl history. The database holds information such as thelast crawl time, the last crawlID and the last crawl update type. Information about content sources, such astheir schedules and locations,are synchronized to the registry on crawl role servers from the searchadministration database

 

Content Processing Component

Whenthe Crawl Component forwards content and metadata, the Content Processing Component performs tasks on the content to prepare it for the search index file. The search index workswith processed content, calledartifacts. The Content ProcessingComponent tasks include parsing documents, property mapping, and linguisticsprocessing. The latter detects language and extracts language-based entities. TheContent Processing Componentalso writes information aboutthe URLs into the link database, which holds information about links rather than content.Including and excluding file types

 

TheContent Processing Component can only process file data if the file type(extension) is included in the listof available file types on the Manage Files Types page and the crawl server hasthe appropriate format handlerinstalled. By default, some file types, such as email messages with a .emlextension, are supported. Thedefault format handlers focus on content, so while a Microsoft PowerPoint .pptx format appears on the Manage File Types list by default, theMicrosoft PowerPoint .pps presentation file does not.

 

Analytics Processing Component

 

Oneof the major changes to the search service is the inclusion of web analytics functionality. This was previously a separate serviceapplication in SharePointServer 2010. The new Search Analytics function analyzes both the crawled items and how users interact with search results; theseare entitled Search Analyticsand Usage Analytics.

 

Search Analytics

 

Thesearch analytics information is used to improve search relevance and to create search reports, recommendations, and data links.This information is thenreturned to the Content ProcessingComponent for storage in the search index. Information about search activity, such as the number of searchclicks from a search results page, helps to improve the relevance of the search results by analyzing previoususer activities. This information is stored  in the link database.This information is then further analyzed by a series of sub-analyses. Thefollowing table shows thesub-analyses that act on Search Analysis results.

 

Usage Analytics

 

Usageanalytics analyzes usage events, such as views from the event store. When auser completes an action, suchas viewing a page, the event is collected and stored in usage files on each WFEserver. This information ispushed to an event store where it is stored until it is processed by theAnalytics Processing Component.The results are then returned to the Content Processing Component to beincluded in the search index.The usage events that are analyzed include:

 

• Views
• Recommendations displayed
• Recommendation Clicked

 

Index Component

 

Thesearch index is a set of files that are stored in separate folders on a server. The Content Processing Component processes items

provisionedby the Crawl Components, maps crawledproperties to managed properties, and formats these as artifacts that can be stored on the search index. The indexes can include:

 

• Full-text indexes
• Indexes of the Managed properties
• An index for attribute vectors
• Numeric indexes
• Index Component
• Index Partition
• Index Replica

 

Query Processing Component

 

TheQuery Processing Component, which sits between the Index Component and the search front-end client, handles processing when a user executes a search query and processes theresults to be returned. Whenthe Indexing Component— oranother search provider—returns a result set, the Query Processing Component performs any additional processing that is required.

 

TheQuery Processing Component performs some linguistic processing to maximize query efficiency and effectiveness, such as:

• Word stemming. Stemming returns words closely related to or stemming from another word. For example, the stemmer relates words such as “jumping,” “jumped,” and “jumper” to the verb “to jump.”
• Word breaking. This refers to the breaking of words that are linked by some form of hyphenation; for example, the term “server-based“ is linked by a hyphen. In this case, word breaking returns both “server” and “based,” with higher relevance given to an item with both present.

 

The queryprocessing workflow is as follows:

• The Query Processing Component receives a query from the search front-end client and processes to maximize precision, recall, and relevancy. These actions include:
• Applying Web Part transformations.