Redirect Following

One very handy feature in HTTP is “redirect following”. Many web sites make use of the
HTTP redirect internally, so you will most likely encounter redirects when writing a bot. The
HTTP redirect allows the server to redirect the web browser to a new URL.
To see an HTTP redirect in action, enter the following URL into your web browser. You
would expect the browser to take you to the URL you entered.
http://www.httprecipes.com/1/4/redirect.php
However, you do not end up on the above URL. You actually end up at the root of the
“Recipe Site” at the following URL:
http://www.httprecipes.com/
This was due to an HTTP redirect. By default, the HttpWebRequest class will
follow all such redirects automatically and often. You do not need to even be concerned with
them. Web browsers will always follow redirects automatically. However, if you would like
to handle the redirects yourself, you can disable auto following. The following line of code
would do this:
request.AllowAutoRedirect = false;
If you disable redirection following, you may manually follow the redirects by looking at
the location response header.