Build a Search Engine SiteMap in C#

Sitemaps are XML files that web masters can create to let search engines know what what pages to index and how frequently to check for changes on each page. The XML format of the sitemap file is detailed on sitemaps.org. Here is a sample of a sitemap with a single url.

 
 

 
 
  
  
  
    
   
   
    
    http://example.com
   
   
    
   
   
    
    daily
   
   
  
  
  

 
 

My sample differs from the one on sitemaps.org. The more defined namespace in my example will validate on Google, Yahoo! and with Ask.com. At the time of this writing, theirs doesn't. The loc and changefreq are required, priority and lastmod are optional.
My advice with the search engines is treat them like a passport agent. Say only what is required and nothing else or you could find yourself sent to the end of the line. Once you've determined what URLs will be on the sitemap, the only decision is defining the changefreq of each page. One strategy might be to set the home page to daily, section pages to weekly and content pages to monthly. If your home page has stock tickers or sports scores, you could set the changefreq to always or hourly.
Google prefers the sitemap file to be in the root folder and every example on their site names the file sitemap.xml. What Google wants, Google gets.

Additional Namespaces

using System.IO;
using System.Xml;

Sample Code

You will need write access to the sitemap.xml file. Since you don't want to give your entire root folder write access, my advice is to create a dummy sitemap.xml, place it into the root folder and then set write access to that file. Adding a try...catch to the code below will alert you if that write access is not there.

string SiteMapFile = @"~/sitemap.xml";
string xmlFile = Server.MapPath(SiteMapFile);

XmlTextWriter writer = new XmlTextWriter(xmlFile, System.Text.Encoding.UTF8);
writer.Formatting = Formatting.Indented;
writer.WriteStartDocument();
writer.WriteStartElement("urlset");
writer.WriteAttributeString("xmlns:xsi", "http://www.w3.org/2001/XMLSchema-instance");
writer.WriteAttributeString("xsi:schemaLocation", "http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd");
writer.WriteAttributeString("xmlns", "http://www.sitemaps.org/schemas/sitemap/0.9");

// Add Home Page
writer.WriteStartElement("url");
writer.WriteElementString("loc", "http://example.com");
writer.WriteElementString("changefreq", "daily");
writer.WriteEndElement(); // url

// Add Sections and Articles
SqlConnection con = new SqlConnection(connectionString);
string sql = @"SELECT url, 'weekly' as changefreq FROM Section 
UNION SELECT url, 'monthly' as changefreq FROM Articles ";
SqlCommand cmd = new SqlCommand(sql, con);
cmd.CommandType = CommandType.Text;

try
{
    con.Open();
    SqlDataReader reader = cmd.ExecuteReader();
    while (reader.Read())
    {
        string loc = "http://example.com" + reader["URL"].ToString();
        string changefreq = reader["changefreq"].ToString();
        writer.WriteStartElement("url");
        writer.WriteElementString("loc", loc);
        writer.WriteElementString("changefreq", changefreq);
        writer.WriteEndElement(); // url
    }
    reader.Close();
}
catch (SqlException err)
{
    throw new ApplicationException("Data Error (Sections):" + err.Message);
}
finally
{
    con.Close();
}

writer.WriteEndElement();// urlset        
writer.Close();

Validate Your Sitemap

Once you've confirmed you have a good looking sitemap.xml file in the root folder of your web site and it contains all the pages you want indexed by the search engines, it is now time to validate it. XML-Sitemaps.com has a sitemap validator that you can test out your new sitemap. Once it's fine, move to the next step.

Update Your robots.txt File

Add the location of your sitemap in your robots.txt file.
Sitemap: http://example.com/sitemap.xml

Google Webmaster

Google has a suite of tools for managing your relationship between your web sites and them. They call this suite Google Webmaster Central. It is here that you will register your web sites with validation files. Once they've established you are the webmaster of your site, they will present you will a screen to submit your sitemap. You will need a Google Account for this process. If you don't have one, follow the link to Create a Google Account.

Yahoo! Site Explorer

Yahoo! has a similar setup which is called Yahoo! Site Explorer. Using your Yahoo! ID, you will go through the same process of registering your web sites. And once that process has been completed, you can then submit your sitemap.xml file. Don't have a Yahoo ID? Get one.

Ask.com

Ask.com doesn't require any accounts or site validation. Just ping their server with the location of your sitemap.xml file modeled after the URL example below. Of course replace example.com with your domain name.
http://submissions.ask.com/ping?sitemap=http%3A//example.com/sitemap.xml

Monitoring the Sitemap Crawl

Both the Yahoo! Site Explorer and the Google Webmaster Tools have reports that provide updated status on the success and failure of the sitemap crawl. Ask.com to my knowledge doesn't have any such tools. And I could locate a sitemap submission tool at all for MSN.com.

Using an HTTP Handler

This example uses the File System. Another option is to use an HTTP Handler to deliver the sitemap.xml.

Final Word

A friend of mine with a low traffic site saw his page views double after adding a sitemap. If one page of code can potentially double your page views, then it is worth pursuing.