F# System.Net.WebRequest GetResponse

本文介绍了一个使用F#语言实现的简单网页抓取程序。该程序通过System.Net命名空间来发送HTTP请求并获取响应,然后读取响应的内容。示例中抓取了Google首页的内容并将其打印出来。
 
  1. #light
  2. open System.Net 
  3. open System 
  4. open System.IO 
  5. open Printf 
  6. /// Fetch the contents of a web page 
  7. let http(url: string) = 
  8. let req = System.Net.WebRequest.Create(url) 
  9. let resp = req.GetResponse() 
  10. let stream = resp.GetResponseStream() 
  11. let reader = new IO.StreamReader(stream) 
  12. let html = reader.ReadToEnd() 
  13. resp.Close() 
  14. html 
  15. let live = http("http://www.google.com"
  16. printf "%s"live 
  17. System.Console.ReadKey(true);
public static string GetHtml(string url) { string htmlCode; HttpWebRequest webRequest = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(url); webRequest.Timeout = 30000; webRequest.Method = "GET"; webRequest.UserAgent = "Mozilla/4.0"; webRequest.Headers.Add("Accept-Encoding", "gzip, deflate"); HttpWebResponse webResponse = (System.Net.HttpWebResponse)webRequest.GetResponse(); //获取目标网站的编码格式 string contentype = webResponse.Headers["Content-Type"]; Regex regex = new Regex("charset\\s*=\\s*[\\W]?\\s*([\\w-]+)", RegexOptions.IgnoreCase); using (System.IO.Stream streamReceive = webResponse.GetResponseStream()) { //匹配编码格式 if (regex.IsMatch(contentype)) { Encoding ending = Encoding.GetEncoding(regex.Match(contentype).Groups[1].Value.Trim()); using (System.IO.StreamReader sr = new System.IO.StreamReader(streamReceive, ending)) { htmlCode = sr.ReadToEnd(); } } else { using (StreamReader sr = new System.IO.StreamReader(streamReceive, Encoding.UTF8)) { htmlCode = sr.ReadToEnd(); } } } return htmlCode; } 上面是C#语言写的函数,调用GetHtml方法,返回的htmlCode的值为乱码,下面是部分的乱码值: Խ��<v�w+F�� ����"r�����h8 � ��%�gi��lN�"i\�G���jmj��3������?��o����������������?���/�˿��o9���������/��o�Ͽ������������ÿ�����G���?��1�-ǿ��������_������������ο������������_�z<�!
03-11
System.Net.WebException 未處理 HResult=-2146233079 Message=作業逾時 Source=System StackTrace: 於 System.Net.HttpWebRequest.GetResponse() 於 WindowsApplication1.Form1.CallWebAPIHttpClient(String urls, String send_methods) 於 E:\U232609\泰國廠\ashx文件應用客戶端\ashx文件應用客戶端\Form1.vb: 行 196 於 WindowsApplication1.Form1.Button4_Click(Object sender, EventArgs e) 於 E:\U232609\泰國廠\ashx文件應用客戶端\ashx文件應用客戶端\Form1.vb: 行 130 於 System.Windows.Forms.Control.OnClick(EventArgs e) 於 System.Windows.Forms.Button.OnClick(EventArgs e) 於 System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent) 於 System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks) 於 System.Windows.Forms.Control.WndProc(Message& m) 於 System.Windows.Forms.ButtonBase.WndProc(Message& m) 於 System.Windows.Forms.Button.WndProc(Message& m) 於 System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m) 於 System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m) 於 System.Windows.Forms.NativeWindow.DebuggableCallback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam) 於 System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg) 於 System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr dwComponentID, Int32 reason, Int32 pvLoopData) 於 System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context) 於 System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context) 於 Microsoft.VisualBasic.ApplicationServices.WindowsFormsApplicationBase.OnRun() 於 Microsoft.VisualBasic.ApplicationServices.WindowsFormsApplicationBase.DoApplicationModel() 於 Microsoft.VisualBasic.ApplicationServices.WindowsFormsApplicationBase.Run(String[] commandLine) 於 WindowsApplication1.My.MyApplication.Main(String[] Args) 於 17d14f5c-a337-4978-8281-53493378c1071.vb: 行 81 於 System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args) 於 System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args) 於 Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly() 於 System.Threading.ThreadHelper.ThreadStart_Context(Object state) 於 System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) 於 System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) 於 System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) 於 System.Threading.ThreadHelper.ThreadStart() InnerException:
08-22
小蓝窗“PS C:\Users\Administrator> function Invoke-EnhancedCurlRequest { >> <# >> .SYNOPSIS >> 增强版 HTTP 请求函数,提供更详细的响应信息 >> #> >> param( >> [Parameter(Mandatory=$true)] >> [string]$Uri, >> >> [ValidateSet('GET','POST','PUT','DELETE','PATCH','HEAD','OPTIONS')] >> [string]$Method = 'GET', >> >> [hashtable]$Headers = @{}, >> >> [object]$Body, >> >> [int]$Timeout = 30, >> >> [switch]$SkipCertificateCheck >> ) >> >> # 证书验证设置 >> if ($SkipCertificateCheck) { >> [System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true } >> } >> >> # 尝试使用 HttpClient(.NET 4.5+) >> if ("System.Net.Http.HttpClient" -as [type]) { >> try { >> $request = [System.Net.Http.HttpRequestMessage]::new() >> $request.Method = $Method >> $request.RequestUri = [Uri]::new($Uri) >> >> # 添加请求头 >> foreach ($key in $Headers.Keys) { >> if ($request.Headers.TryAddWithoutValidation($key, $Headers[$key])) { >> # 成功添加到请求头 >> } elseif ($request.Content -ne $null) { >> # 尝试添加到内容头 >> $request.Content.Headers.TryAddWithoutValidation($key, $Headers[$key]) >> } >> } >> >> # 处理请求体 >> if ($Body) { >> if ($Body -is [hashtable] -or $Body -is [pscustomobject]) { >> $jsonBody = $Body | ConvertTo-Json >> $request.Content = [System.Net.Http.StringContent]::new( >> $jsonBody, >> [System.Text.Encoding]::UTF8, >> 'application/json' >> ) >> } else { >> $request.Content = [System.Net.Http.StringContent]::new( >> $Body.ToString(), >> [System.Text.Encoding]::UTF8 >> ) >> } >> } >> >> # 发送请求 >> $handler = [System.Net.Http.HttpClientHandler]::new() >> $client = [System.Net.Http.HttpClient]::new($handler) >> $client.Timeout = [System.TimeSpan]::FromSeconds($Timeout) >> >> try { >> $response = $client.SendAsync($request).GetAwaiter().GetResult() >> >> # 获取详细的响应头信息 >> $responseHeaders = [ordered]@{} >> foreach ($header in $response.Headers) { >> $responseHeaders[$header.Key] = $header.Value -join ", " >> } >> foreach ($header in $response.Content.Headers) { >> $responseHeaders[$header.Key] = $header.Value -join ", " >> } >> >> # 返回结构化结果 >> return [PSCustomObject]@{ >> StatusCode = [int]$response.StatusCode >> StatusMessage = $response.StatusCode.ToString() >> Headers = $responseHeaders >> Content = $response.Content.ReadAsStringAsync().Result >> IsSuccess = $response.IsSuccessStatusCode >> Technology = "HttpClient" >> RawResponse = $response >> } >> } finally { >> $request.Dispose() >> $client.Dispose() >> } >> } catch { >> Write-Warning "HttpClient 请求失败: $($_.Exception.Message)" >> } >> } >> >> # 回退到 WebRequest(兼容旧版 .NET) >> try { >> $request = [System.Net.WebRequest]::Create($Uri) >> $request.Method = $Method >> $request.Timeout = $Timeout * 1000 >> >> # 添加请求头 >> foreach ($key in $Headers.Keys) { >> $request.Headers.Add($key, $Headers[$key]) >> } >> >> # 处理请求体 >> if ($Body) { >> $bodyJson = if ($Body -is [hashtable] -or $Body -is [pscustomobject]) { >> $Body | ConvertTo-Json >> } else { >> $Body.ToString() >> } >> $bytes = [System.Text.Encoding]::UTF8.GetBytes($bodyJson) >> $request.ContentLength = $bytes.Length >> $stream = $request.GetRequestStream() >> $stream.Write($bytes, 0, $bytes.Length) >> $stream.Close() >> } >> >> # 获取响应 >> $response = $request.GetResponse() >> $reader = New-Object System.IO.StreamReader($response.GetResponseStream()) >> $content = $reader.ReadToEnd() >> $reader.Close() >> >> # 获取响应头信息 >> $responseHeaders = [ordered]@{} >> foreach ($key in $response.Headers.AllKeys) { >> $responseHeaders[$key] = $response.Headers[$key] >> } >> >> return [PSCustomObject]@{ >> StatusCode = [int]$response.StatusCode >> StatusMessage = $response.StatusDescription >> Headers = $responseHeaders >> Content = $content >> IsSuccess = ([int]$response.StatusCode -ge 200 -and [int]$response.StatusCode -lt 300) >> Technology = "WebRequest" >> } >> } catch [System.Net.WebException] { >> $response = $_.Exception.Response >> if ($response -ne $null) { >> $reader = New-Object System.IO.StreamReader($response.GetResponseStream()) >> $content = $reader.ReadToEnd() >> $reader.Close() >> >> $responseHeaders = [ordered]@{} >> foreach ($key in $response.Headers.AllKeys) { >> $responseHeaders[$key] = $response.Headers[$key] >> } >> >> return [PSCustomObject]@{ >> StatusCode = [int]$response.StatusCode >> StatusMessage = $response.StatusDescription >> Headers = $responseHeaders >> Content = $content >> IsSuccess = $false >> Technology = "WebRequest" >> } >> } else { >> Write-Error "WebRequest 请求失败: $($_.Exception.Message)" >> throw >> } >> } catch { >> Write-Error "WebRequest 请求失败: $($_.Exception.Message)" >> throw >> } >> } >> PS C:\Users\Administrator> # 更新模块文件 PS C:\Users\Administrator> $moduleDir = "$env:ProgramFiles\WindowsPowerShell\Modules\PSHttpClient" PS C:\Users\Administrator> $moduleContent = @' >> # 添加程序集加载代码 >> try { >> Add-Type -AssemblyName System.Net.Http -ErrorAction Stop >> } >> catch { >> Write-Warning "无法加载System.Net.Http程序集: $($_.Exception.Message)" >> } >> >> # 插入增强型函数 >> function Invoke-EnhancedCurlRequest { >> # 上面的完整函数代码 >> } >> '@ >> PS C:\Users\Administrator> # 将函数定义插入模块文件 PS C:\Users\Administrator> $functionDefinition = (Get-Command Invoke-EnhancedCurlRequest).Definition PS C:\Users\Administrator> $moduleContent = $moduleContent -replace '# 插入增强型函数', $functionDefinition PS C:\Users\Administrator> PS C:\Users\Administrator> $moduleContent | Out-File "$moduleDir\PSHttpClient.psm1" -Encoding UTF8 -Force PS C:\Users\Administrator> PS C:\Users\Administrator> # 更新模块清单 PS C:\Users\Administrator> $manifestContent = @' >> @{ >> ModuleVersion = '1.2' >> RootModule = 'PSHttpClient.psm1' >> FunctionsToExport = @('Invoke-EnhancedCurlRequest') >> CompatiblePSEditions = @('Desktop', 'Core') >> PowerShellVersion = '5.1' >> Description = '增强版HTTP请求模块' >> } >> '@ >> $manifestContent | Out-File "$moduleDir\PSHttpClient.psd1" -Encoding UTF8 -Force >> PS C:\Users\Administrator> # 重新加载模块 PS C:\Users\Administrator> Remove-Module PSHttpClient -ErrorAction SilentlyContinue PS C:\Users\Administrator> Import-Module PSHttpClient -Force 所在位置 C:\Program Files\WindowsPowerShell\Modules\PSHttpClient\PSHttpClient.psm1:17 字符: 27 + [string]$Method = 'GET', + ~~~~~ 赋值表达式无效。赋值运算符输入必须是能够接受赋值的对象,例如变量或属性。 所在位置 C:\Program Files\WindowsPowerShell\Modules\PSHttpClient\PSHttpClient.psm1:18 字符: 31 + [hashtable]$Headers = @{}, + ~~~ 赋值表达式无效。赋值运算符输入必须是能够接受赋值的对象,例如变量或属性。 + CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException + FullyQualifiedErrorId : InvalidLeftHandSide Import-Module : 未能处理要处理“PSHttpClient.psm1”的模块(即模块清单“C:\Program Files\WindowsPowerShell\Modules\PSHttp Client\PSHttpClient.psd1”的“ModuleToProcess/RootModule”字段中列出的模块),因为在任何模块目录中都没有找到有效模块。 所在位置 行:1 字符: 1 + Import-Module PSHttpClient -Force + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : ResourceUnavailable: (PSHttpClient:String) [Import-Module], PSInvalidOperationException + FullyQualifiedErrorId : Modules_ModuleFileNotFound,Microsoft.PowerShell.Commands.ImportModuleCommand PS C:\Users\Administrator> PS C:\Users\Administrator> # 测试请求并显示完整响应 PS C:\Users\Administrator> $testResult = Invoke-EnhancedCurlRequest -Uri "https://httpbin.org/get" -Method GET PS C:\Users\Administrator> PS C:\Users\Administrator> # 显示结果 PS C:\Users\Administrator> if ($testResult.IsSuccess) { >> Write-Host "✅ 请求成功 (使用 $($testResult.Technology))" -ForegroundColor Green >> Write-Host "状态码: $($testResult.StatusCode) ($($testResult.StatusMessage))" >> >> # 显示响应头 >> Write-Host "`n响应头:" -ForegroundColor Cyan >> $testResult.Headers.GetEnumerator() | Format-Table @{ >> Name = 'Header' >> Expression = {$_.Key} >> }, @{ >> Name = 'Value' >> Expression = {$_.Value} >> } -AutoSize >> >> # 解析并显示JSON内容 >> try { >> $jsonContent = $testResult.Content | ConvertFrom-Json >> Write-Host "`n响应内容:" -ForegroundColor Cyan >> $jsonContent | Format-List >> } catch { >> Write-Host "`n原始响应内容:" -ForegroundColor Cyan >> $testResult.Content >> } >> } else { >> Write-Error "❌ 请求失败: $($testResult.StatusCode) ($($testResult.StatusMessage))" >> $testResult.Content >> } >> ✅ 请求成功 (使用 WebRequest) 状态码: 200 (OK) 响应头: Header Value ------ ----- Connection keep-alive Access-Control-Allow-Origin * Access-Control-Allow-Credentials true Content-Length 199 Content-Type application/json Date Sat, 16 Aug 2025 08:11:35 GMT Server gunicorn/19.9.0 响应内容: args : headers : @{Host=httpbin.org; X-Amzn-Trace-Id=Root=1-68a03d37-71c27db437d3d59d20740cf4} origin : 112.40.123.16 url : https://httpbin.org/get PS C:\Users\Administrator> $postData = @{ >> name = "John Doe" >> email = "john@example.com" >> } >> PS C:\Users\Administrator> $response = Invoke-EnhancedCurlRequest -Uri "https://httpbin.org/post" -Method POST -Body $postData PS C:\Users\Administrator> $response.Content | ConvertFrom-Json args : data : { "email": "john@example.com", "name": "John Doe" } files : form : headers : @{Content-Length=64; Host=httpbin.org; X-Amzn-Trace-Id=Root=1-68a03d5b-2eeeb7f52426357745dc3d4e} json : @{email=john@example.com; name=John Doe} origin : 112.40.123.16 url : https://httpbin.org/post PS C:\Users\Administrator> PS C:\Users\Administrator> $response = Invoke-EnhancedCurlRequest -Uri "https://self-signed-cert.example" -SkipCertificateCheck Invoke-EnhancedCurlRequest : WebRequest 请求失败: 未能解析此远程名称: 'self-signed-cert.example' 所在位置 行:1 字符: 13 + $response = Invoke-EnhancedCurlRequest -Uri "https://self-signed-cert ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-EnhancedCurlRequest 使用“0”个参数调用“GetResponse”时发生异常:“未能解析此远程名称: 'self-signed-cert.example'” 所在位置 行:106 字符: 9 + $response = $request.GetResponse() + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [], ParentContainsErrorRecordException + FullyQualifiedErrorId : WebException PS C:\Users\Administrator> $response = Invoke-EnhancedCurlRequest -Uri "https://httpbin.org/status/404" PS C:\Users\Administrator> if (-not $response.IsSuccess) { >> Write-Warning "请求失败: $($response.StatusCode) $($response.StatusMessage)" >> } >>” 我这边现在是2025-8-16 16:12 跟你验证出来的信息 好像不太一样
08-17
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值