最好使用的是HTMLAgilityPack。您还可以根据需要从检索的页面中选择元素,使用Fizzler或CSQuery。使用LINQ或Regukar Expressions易于出错,尤其是在HTML格式可能不正确,缺少结束标记,具有嵌套的子元素等时。
您需要将页面流式传输到HtmlDocument对象,然后选择所需的元素。
// Call the page and get the generated HTML
var doc = new HtmlAgilityPack.HtmlDocument();
HtmlAgilityPack.HtmlNode.ElementsFlags["br"] = HtmlAgilityPack.HtmlElementFlag.Empty;
doc.OptionWriteEmptyNodes = true;
try
{
var webRequest = HttpWebRequest.Create(pageUrl);
Stream stream = webRequest.GetResponse().GetResponseStream();
doc.Load(stream);
stream.Close();
}
catch (System.UriFormatException uex)
{
Log.Fatal("There was an error in the format of the url: " + itemUrl, uex);
throw;
}
catch (System.Net.WebException wex)
{
Log.Fatal("There was an error connecting to the url: " + itemUrl, wex);
throw;
}
//get the div by id and then get the inner text
string testDivSelector = "//div[@id='test']";
var divString = doc.DocumentNode.SelectSingleNode(testDivSelector).InnerHtml.ToString();
[编辑]实际上,将其废弃。最简单的方法是使用FizzlerEx,它是原始Fizzler项目的更新的jQuery / CSS3选择器实现。
直接从其站点获取代码示例:
using HtmlAgilityPack;
using Fizzler.Systems.HtmlAgilityPack;
//get the page
var web = new HtmlWeb();
var document = web.Load("http://example.com/page.html");
var page = document.DocumentNode;
//loop through all div tags with item css class
foreach(var item in page.QuerySelectorAll("div.item"))
{
var title = item.QuerySelector("h3:not(.share)").InnerText;
var date = DateTime.Parse(item.QuerySelector("span:eq(2)").InnerText);
var description = item.QuerySelector("span:has(b)").InnerHtml;
}
我认为没有比这更简单的了。