Golang实战【IP池-自动切换IP爬虫】

继上次爬boss直聘数据IP被封，虽然后面发现boss直聘 只要稍微没个请求停一秒钟就可以一直爬下去，但是任然心有不甘。欺负我ip少？？？
于是这两天抽空写了个IP池加上可以自动切换IP 千千万万个IP 让你封···········哈哈

我这边用的是西刺的代理 还是比较稳的········拿着西刺的免费IP 爬着西刺的免费IP [:P]

实现效果如下

········································································································································

感兴趣的同学可以玩玩：不过需要redis可以先安装一个

package main

import (
    "github.com/PuerkitoBio/goquery"
    "log"
    "fmt"
    "github.com/garyburd/redigo/redis"
    "net/url"
    "net/http"
    "encoding/json"
    "os"
    "strconv"
    "math/rand"
    "time"
    "strings"

)

const(
        PAGE int = 40
)
var (
    xici string = "http://www.xicidaili.com/wn/"
)

func main() {
    //getIp("local")
    getIp("http://171.104.132.87:9999")
}

func getIp(ip string){
    var count int

    for i := 1; i <= PAGE; i++ {
   

        response := getRep(xici + strconv.Itoa(i),ip)

        if (response.StatusCode == 200) {
            dom, err := goquery