C#网页采集数据的几种方式(WebClient、WebBrowser和HttpWebRequest/HttpWebResponse)
|
admin
2017年2月16日 16:25
本文热度 6266
|
获取网页数据有很多种方式。在这里主要讲述通过WebClient、WebBrowser和HttpWebRequest/HttpWebResponse三种方式获取网页内容。
这里获取的是包括网页的所有信息。如果单纯需要某些数据内容。可以自己构造函数甄别抠除出来!一般的做法是根据源码的格式,用正则来过滤出你需要的内容部分。
一、通过WebClient获取网页内容
这是一种很简单的获取方式,当然,其它的获取方法也很简单。在这里首先要说明的是,如果为了实际项目的效率考虑,需要考虑在函数中分配一个内存区域。大概写法如下
-
- byte[] buffer = new byte[1024];
- using (MemoryStream memory = new MemoryStream())
- {
- int index = 1, sum = 0;
- while (index * sum < 100 * 1024)
- {
- index = reader.Read(buffer, 0, 1024);
- if (index > 0)
- {
- memory.Write(buffer, 0, index);
- sum += index;
- }
- }
-
- Encoding.GetEncoding("gb2312").GetString(memory.ToArray());
- if (string.IsNullOrEmpty(html))
- {
- return html;
- }
- else
- {
- Regex re = new Regex(@"charset=(? charset[/s/S]*?)[ |'']");
- Match m = re.Match(html.ToLower());
- encoding = m.Groups[charset].ToString();
- }
- if (string.IsNullOrEmpty(encoding) || string.Equals(encoding.ToLower(), "gb2312"))
- {
- return html;
- }
- }
好了,现在进入正题,WebClient获取网页数据的代码如下
-
- try
- {
- WebClient webClient = new WebClient();
- webClient.Credentials = CredentialCache.DefaultCredentials;
- Byte[] pageData = webClient.DownloadData("http://www.360doc.com/content/11/0427/03/1947337_112596569.shtml");
-
- string pageHtml = Encoding.UTF8.GetString(pageData);
- using (StreamWriter sw = new StreamWriter("e:\\ouput.txt"))
- {
- htm = sw.ToString();
- sw.Write(pageHtml);
- }
- }
- catch (WebException webEx)
- {
- Console.W
- }
二、通过WebBrowser控件获取网页内容
相对来说,这是一种最简单的获取方式。拖WebBrowser控件进去,然后匹配下面这段代码
- WebBrowser web = new WebBrowser();
- web.Navigate("http://www.163.com");
- web.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(web_DocumentCompleted);
- void web_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
- {
- WebBrowser web = (WebBrowser)sender;
- HtmlElementCollection ElementCollection = web.Document.GetElementsByTagName("Table");
- foreach (HtmlElement item in ElementCollection)
- {
- File.AppendAllText("Kaijiang_xj.txt", item.InnerText);
- }
- }
三、使用HttpWebRequest/HttpWebResponse获取网页内容
这是一种比较通用的获取方式。
- public void GetHtml()
- {
- var url = "http://www.360doc.com/content/11/0427/03/1947337_112596569.shtml";
- string strBuff = "";
- int byteRead = 0;
-
- HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
- HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse();
-
- Stream reader = webResponse.GetResponseStream();
-
- StreamReader respStreamReader = new StreamReader(reader,Encoding.UTF8);
-
-
- char[] cbuffer = new char[1024];
- byteRead = respStreamReader.Read(cbuffer,0,256);
-
- while (byteRead != 0)
- {
- string strResp = new string(char,0,byteRead);
- strBuff = strBuff + strResp;
- byteRead = respStreamReader.Read(cbuffer,0,256);
- }
- using (StreamWriter sw = new StreamWriter("e:\\ouput.txt"))
- {
- htm = sw.ToString();
- sw.Write(strBuff);
- }
- }
该文章在 2017/2/16 16:25:49 编辑过