scrapy 爬取某图源码

最新推荐文章于 2024-04-11 16:47:57 发布
最新推荐文章于 2024-04-11 16:47:57 发布
阅读量807 收藏
点赞数
分类专栏： 文章 文章标签： python
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/gllvcpp/article/details/90116183
版权

spiders写法：中meizi.py

# -*- coding: utf-8 -*-
import scrapy
import os,random
import requests
from pypinyin import lazy_pinyin
from Meizi.items import MeiziItem
UA=[
    {'User-Agent':'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50'},

    {'User-Agent':'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50'},
    {'User-Agent':'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0'},
    {'User-Agent':'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)'},
    {'User-Agent':'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)'},
    {'User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)'},

    {'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1'},
    {'User-Agent':'Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1'},
    {'User-Agent':'Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11'},

    {'User-Agent':'Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11'},

    {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11'},
    {'User-Agent':'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon 2.0)'},
    {'User-Agent': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; TencentTraveler 4.0)'},
    {'User-Agent': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'},
    {'User-Agent': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)'},
    {'User-Agent': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SE 2.X MetaSr 1.0; SE 2.X MetaSr 1.0; .NET CLR 2.0.50727; SE 2.X MetaSr 1.0)'},
    {'User-Agent': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 360SE)'},
    {'User-Agent': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Avant Browser)'},
    {'User-Agent': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'},
    {'User-Agent':'Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5'},
    {'User-Agent':'Mozilla/5.0 (iPod; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5'},
    {'User-Agent':'Mozilla/5.0 (iPad; U; CPU OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5'},
    {'User-Agent': 'Mozilla/5.0 (Linux; U; Android 2.3.7; en-us; Nexus One Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1'},
    {'User-Agent': 'MQQBrowser/26 Mozilla/5.0 (Linux; U; Android 2.3.7; zh-cn; MB200 Build/GRJ22; CyanogenMod-7) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1'},
    {'User-Agent': 'Opera/9.80 (Android 2.3.4; Linux; Opera Mobi/build-1107180945; U; en-GB) Presto/2.8.149 Version/11.10'},
    {'User-Agent': 'Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13'},
    {'User-Agent': 'Mozilla/5.0 (BlackBerry; U; BlackBerry 9800; en) AppleWebKit/534.1+ (KHTML, like Gecko) Version/6.0.0.337 Mobile Safari/534.1'},
    {'User-Agent': 'Mozilla/5.0 (hp-tablet; Linux; hpwOS/3.0.0; U; en-US) AppleWebKit/534.6 (KHTML, like Gecko) wOSBrowser/233.70 Safari/534.6 TouchPad/1.0'},
    {'User-Agent': 'Mozilla/5.0 (SymbianOS/9.4; Series60/5.0 NokiaN97-1/20.0.019; Profile/MIDP-2.1 Configuration/CLDC-1.1) AppleWebKit/525 (KHTML, like Gecko) BrowserNG/7.1.18124'},
    {'User-Agent': 'Mozilla/5.0 (compatible; MSIE 9.0; Windows Phone OS 7.5; Trident/5.0; IEMobile/9.0; HTC; Titan)'},
    {'User-Agent': 'UCWEB7.0.2.37/28/999'},
    {'User-Agent': 'NOKIA5700/ UCWEB7.0.2.37/28/999'},
    {'User-Agent': 'Openwave/ UCWEB7.0.2.37/28/999'},
    {'User-Agent': 'Mozilla/4.0 (compatible; MSIE 6.0; ) Opera/UCWEB7.0.2.37/28/999'}]
class MeiziSpider(scrapy.Spider):
    name = 'meizi'
    allowed_domains = ['****.com']
    # start_urls = ['http://***.com/']
    
    def start_requests(self):
    	url="https://www.***.com/xinggan/page/"
    	for i in range(3,4):
    		yield scrapy.Request(url+str(i))

    def parse(self, response):

        name=response.xpath("//ul[@id='pins']/li/span/a/text()").extract()
       	url=response.xpath("//ul[@id='pins']/li/a/@href").extract()
       	header={
       	'referer': response.url,
		'user-agent': random.choice(UA)["User-Agent"]
		       	}
       	for i in url:
       		yield scrapy.Request(i,callback=self.parse_item,headers=header)
    def parse_item(self,response):
    	name=response.xpath("//h2[@class='main-title']/text()").extract()
    	header={
       	'referer': response.url,
		'user-agent': random.choice(UA)["User-Agent"]
		       	}
    	# os.mkdir('/gg/Images/%s'%name[0])
    	endpage=response.xpath("//div[@class='pagenavi']/a[5]/span/text()").extract()
    	# print(response.url)
    	for i in range(1,int(endpage[0])+1):
    		yield scrapy.Request(response.url+'/'+str(i),headers=header,callback=self.parse_img)
    def parse_img(self,response):
        item=MeiziItem()
        imgurl=response.xpath("//div[@class='main-image']/p/a/img/@src").extract()
        # print(imgurl)
        name=response.xpath("//div[@class='main-image']/p/a/img/@alt").extract()
        imgname=response.xpath("//h2[@class='main-title']/text()").extract()[0][15:]
        dirs='/home/gg/Images/%s'%name[0][15:]
        # print(imgurl[0])
        # url=response.url
        item['imgurl']=imgurl[0]
        item['imgname']=imgname
        item['name']=name[0][15:]
        yield item    #也可以定制自己个性化命名使用requests模块下载图片
       #  header={
       #  'Referer': '{}'.format(imgurl[0]),
       #  'User-Agent': random.choice(UA)['User-Agent'],
       # 'Accept-Encoding': 'gzip, deflate',
       #  'Accept-Language': 'zh-CN,zh;q=0.8,en;q=0.6',
       #  'Accept': 'image/webp,image/apng,image/*,*/*;q=0.8',
       #  'upgrade-insecure-requests': '1',
       #  'Connection': 'keep-alive'}
        
        # if not os.path.exists(dirs):
        #     print(111111111111)
        #     os.mkdir(dirs)
        # with open(dirs+'/'+imgname+'.jpg', "wb+") as jpg:
        #     jpg.write(requests.get(imgurl[0], headers=header).content)

items.py:

import scrapy


class MeiziItem(scrapy.Item):
    # define the fields for your item here like:
    name = scrapy.Field()
    imgurl=scrapy.Field()
    imgname=scrapy.Field()
pipelines.py:
import scrapy,random
from scrapy.exceptions import DropItem
from scrapy.pipelines.images import ImagesPipeline
from Meizi.spiders.meizi import UA 
class MeiziPipeline(object):
    def process_item(self, item, spider):
        return item
class ImagePipeline(ImagesPipeline):
    def get_media_requests(self, item, info):
    	# print(info)
        header={
        'Referer': '{}'.format(item['imgurl']),
        'User-Agent': random.choice(UA)['User-Agent'],
      	'Accept-Encoding': 'gzip, deflate',
        'Accept-Language': 'zh-CN,zh;q=0.8,en;q=0.6',
        'Accept': 'image/webp,image/apng,image/*,*/*;q=0.8',
        'upgrade-insecure-requests': '1',
        'Connection': 'keep-alive'}
        yield scrapy.Request(item['imgurl'],headers=header)
    def item_completed(self, results, item, info):
        image_paths = [x['path'] for ok, x in results if ok]
        if not image_paths:
            raise DropItem("Item contains no images")
        item['imgurl'] = image_paths
        return item 
 def file_path(self, request, response=None, info=None):
        imgname = request.meta['imgname'].strip(' ')
        # print(item)
        # image_guid = request.url.split('/')[-1]
        imgname = re.sub(r'[？\\*|“<>:/]', '', imgname)
        name=request.meta['name'].strip(' ')
        name=re.sub(r'[？\\*|“<>:/]', '', name)
        filename = u'full/{}/{}'.format(name,imgname)
        return filename

settings.py配置:

BOT_NAME = 'Meizi'

SPIDER_MODULES = ['Meizi.spiders']
NEWSPIDER_MODULE = 'Meizi.spiders'
ROBOTSTXT_OBEY = False
DOWNLOAD_DELAY = 3
ITEM_PIPELINES = {
   'Meizi.pipelines.MeiziPipeline': 300,
   'Meizi.pipelines.ImagePipeline': 300, #开启图片管道
}
IMAGES_STORE ='/home/gg/Meizi/Meizi/Images'  #设置存储路径

java抓取:

```java
package com.atgui.springboot.controller;

import org.apache.commons.collections.map.LinkedMap;
import org.apache.commons.io.FileUtils;
import org.apache.http.HttpEntity;
import org.apache.http.HttpStatus;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.utils.HttpClientUtils;
import org.apache.http.config.Registry;
import org.apache.http.config.RegistryBuilder;
import org.apache.http.conn.socket.ConnectionSocketFactory;
import org.apache.http.conn.socket.PlainConnectionSocketFactory;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.conn.ssl.TrustStrategy;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.apache.http.ssl.SSLContextBuilder;
import org.apache.http.util.EntityUtils;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import us.codecraft.xsoup.Xsoup;

import javax.net.ssl.SSLContext;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.channels.UnresolvedAddressException;
import java.security.KeyManagementException;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.*;

public class meizi {
    public static String UA(){
        List<String> list=new ArrayList<>();
        list.add("Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML; like Gecko) Version/5.1 Safari/534.50");
        list.add("Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML; like Gecko) Version/5.1 Safari/534.50");
        list.add("Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0");
        list.add("Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0");
        list.add("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0");
        list.add("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1");

        list.add("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1");
        list.add("Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1");
        list.add("Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11");

        list.add("Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11");

        list.add( "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML; like Gecko) Chrome/17.0.963.56 Safari/535.11");
        list.add("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon 2.0");
        list.add( "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; TencentTraveler 4.0");
        list.add( "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1");
        list.add( "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World");
        list.add( "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SE 2.X MetaSr 1.0; SE 2.X MetaSr 1.0; .NET CLR 2.0.50727; SE 2.X MetaSr 1.0");
        list.add( "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 360SE");
        list.add( "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Avant Browser");
        list.add( "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1");
        list.add("Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML; like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5");
        list.add("Mozilla/5.0 (iPod; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML; like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5");
        list.add("Mozilla/5.0 (iPad; U; CPU OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML; like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5");
        list.add( "Mozilla/5.0 (Linux; U; Android 2.3.7; en-us; Nexus One Build/FRF91) AppleWebKit/533.1 (KHTML; like Gecko) Version/4.0 Mobile Safari/533.1");
        list.add( "MQQBrowser/26 Mozilla/5.0 (Linux; U; Android 2.3.7; zh-cn; MB200 Build/GRJ22; CyanogenMod-7) AppleWebKit/533.1 (KHTML; like Gecko) Version/4.0 Mobile Safari/533.1");
        list.add( "Opera/9.80 (Android 2.3.4; Linux; Opera Mobi/build-1107180945; U; en-GB) Presto/2.8.149 Version/11.10");
        list.add( "Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/534.13 (KHTML; like Gecko) Version/4.0 Safari/534.13");
        list.add( "Mozilla/5.0 (BlackBerry; U; BlackBerry 9800; en) AppleWebKit/534.1+ (KHTML; like Gecko) Version/6.0.0.337 Mobile Safari/534.1");
        list.add( "Mozilla/5.0 (hp-tablet; Linux; hpwOS/3.0.0; U; en-US) AppleWebKit/534.6 (KHTML; like Gecko) wOSBrowser/233.70 Safari/534.6 TouchPad/1.0");
        list.add( "Mozilla/5.0 (SymbianOS/9.4; Series60/5.0 NokiaN97-1/20.0.019; Profile/MIDP-2.1 Configuration/CLDC-1.1) AppleWebKit/525 (KHTML; like Gecko) BrowserNG/7.1.18124");
        list.add( "Mozilla/5.0 (compatible; MSIE 9.0; Windows Phone OS 7.5; Trident/5.0; IEMobile/9.0; HTC; Titan");
        list.add( "UCWEB7.0.2.37/28/999");
        list.add( "NOKIA5700/ UCWEB7.0.2.37/28/999");
        list.add( "Openwave/ UCWEB7.0.2.37/28/999");
        list.add( "Mozilla/4.0 (compatible; MSIE 6.0; ) Opera/UCWEB7.0.2.37/28/999");
        list.add("Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0");
        list.add("Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0");
        list.add("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0");
        list.add("User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1");

        list.add("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1");

        list.add("Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1");
        list.add("Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11");

        list.add("Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11");

        list.add( "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML; like Gecko) Chrome/17.0.963.56 Safari/535.11");
        list.add("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon 2.0");
        list.add( "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; TencentTraveler 4.0");
        list.add( "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1");
        list.add( "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World");
        list.add( "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SE 2.X MetaSr 1.0; SE 2.X MetaSr 1.0; .NET CLR 2.0.50727; SE 2.X MetaSr 1.0");
        list.add( "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 360SE");
        list.add( "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Avant Browser");
        list.add( "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1");
        list.add("Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML; like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5");
        list.add("Mozilla/5.0 (iPod; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML; like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5");
        list.add("Mozilla/5.0 (iPad; U; CPU OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML; like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5");
        list.add( "Mozilla/5.0 (Linux; U; Android 2.3.7; en-us; Nexus One Build/FRF91) AppleWebKit/533.1 (KHTML; like Gecko) Version/4.0 Mobile Safari/533.1");
        list.add( "MQQBrowser/26 Mozilla/5.0 (Linux; U; Android 2.3.7; zh-cn; MB200 Build/GRJ22; CyanogenMod-7) AppleWebKit/533.1 (KHTML; like Gecko) Version/4.0 Mobile Safari/533.1");
        list.add( "Opera/9.80 (Android 2.3.4; Linux; Opera Mobi/build-1107180945; U; en-GB) Presto/2.8.149 Version/11.10");
        list.add( "Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/534.13 (KHTML; like Gecko) Version/4.0 Safari/534.13");
        list.add( "Mozilla/5.0 (BlackBerry; U; BlackBerry 9800; en) AppleWebKit/534.1+ (KHTML; like Gecko) Version/6.0.0.337 Mobile Safari/534.1");
        list.add( "Mozilla/5.0 (hp-tablet; Linux; hpwOS/3.0.0; U; en-US) AppleWebKit/534.6 (KHTML; like Gecko) wOSBrowser/233.70 Safari/534.6 TouchPad/1.0");
        list.add( "Mozilla/5.0 (SymbianOS/9.4; Series60/5.0 NokiaN97-1/20.0.019; Profile/MIDP-2.1 Configuration/CLDC-1.1) AppleWebKit/525 (KHTML; like Gecko) BrowserNG/7.1.18124");
        list.add( "Mozilla/5.0 (compatible; MSIE 9.0; Windows Phone OS 7.5; Trident/5.0; IEMobile/9.0; HTC; Titan");
        list.add( "UCWEB7.0.2.37/28/999");
        list.add( "NOKIA5700/ UCWEB7.0.2.37/28/999");
        list.add( "Openwave/ UCWEB7.0.2.37/28/999");
        list.add( "Mozilla/4.0 (compatible; MSIE 6.0; ) Opera/UCWEB7.0.2.37/28/999");
//        System.out.println(list.size());
        Random random=new Random();
        int n =random.nextInt(list.size());
        return list.get(n);

        }
    /**
     * 设置可访问https
     * @return
     */
    public static CloseableHttpClient createSSLClientDefault() {
        try {
            SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(null, new TrustStrategy() {
                //信任所有
                public boolean isTrusted(X509Certificate[] chain, String authType) throws CertificateException {
                    return true;
                }
            }).build();
            SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslContext,SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);

            return HttpClients.custom().setSSLSocketFactory(sslsf).build();
        } catch (KeyManagementException e) {
            e.printStackTrace();
        } catch (NoSuchAlgorithmException e) {
            e.printStackTrace();
        } catch (KeyStoreException e) {
            e.printStackTrace();
        }
        return HttpClients.createDefault();
    }
    public static String getHtml(String url) {

        CloseableHttpClient httpClient=createSSLClientDefault();
        String html=null;
        try{

            HttpGet httpGet=new HttpGet(url);

            httpGet.setHeader("accept", "text/html");
            httpGet.setHeader("accept-Encoding", "gzip, deflate,br");
            httpGet.setHeader("accept-Language", "zh-CN,zh;q=0.8");
//            httpGet.addHeader("Accept-Charset", "utf-8");
            httpGet.setHeader("connection", "keep-alive");
//
            httpGet.setHeader("referer","http://www.mzitu.com/xinggan/");
            httpGet.setHeader("host"," www.mzitu.com");
//            httpGet.setHeader("Cache-Control", "no-cache");
            httpGet.setHeader("user-agent", UA());
            CloseableHttpResponse response=httpClient.execute(httpGet);
            if (response.getStatusLine().getStatusCode()== HttpStatus.SC_OK){
                HttpEntity httpEntity=response.getEntity();
                html= EntityUtils.toString(httpEntity,"utf-8");
                try {
                    Thread.sleep(1000);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
            else {
                System.out.println("not 200");
                System.out.println(EntityUtils.toString(response.getEntity()));
            }

        }catch (Exception e){
            e.printStackTrace();
        } finally {
            HttpClientUtils.closeQuietly(httpClient);
        }
//        if (html==null){
//            getHtml(url);
//        }
        return html;

    }
    public static List<List<String>> getResponse(String url,int page){
        ArrayList<String> list=new ArrayList<>();
        for (int i = 1; i <=page; i++) {
            list.add(url+"/"+i);
        }
        List<List<String>> pageUrl=new ArrayList<>();
        for (String s : list) {
            String html = getHtml(s);
            if (html==null){
                return null;
            }
            Document document = Jsoup.parse(html);
            String name= Xsoup.compile("//ul[@id='pins']/li/span/a/text()").evaluate(document).get();
            List<String> url1 = Xsoup.compile("//ul[@id='pins']/li/a/@href").evaluate(document).list();
            System.out.println(url1);
            pageUrl.add(url1);

        }
        return pageUrl;
    }
    public static List<List<String>>  parse(String url,int page)
    {
        List<List<String>> pageUrl = getResponse(url, page);
        if (pageUrl==null){
            return null;
        }

        List<List<String>> img=new ArrayList<>();
        for (List<String> pageUrl_ : pageUrl){
        for (String s : pageUrl_) {

            String html = getHtml(s);
            if (html == null) {
               continue;
            }
            Document document = Jsoup.parse(html);
            String name = Xsoup.compile("//h2[@class='main-title']/text()").evaluate(document).get();
            String endpage = Xsoup.compile("//div[@class='pagenavi']/a[5]/span/text()").evaluate(document).get();
            if (name == null) {
               continue;
            }
            System.out.println("=====>>>" + name);
            System.out.println("=====>>>" + endpage);
            List<String> imgurl = new ArrayList<>();
            try {
                int i1 = Integer.parseInt(endpage);
                System.out.println(i1);
                if (i1 < 0) {

                    continue;
                }
                for (int i = 1; i <= i1; i++) {
                    imgurl.add(s + "/" + i);
                }
                img.add(imgurl);
            } catch (Exception e) {
                e.printStackTrace();
            }

        }

        }

        return img;
    }
    /*
        imgurl=response.xpath("//div[@class='main-image']/p/a/img/@src").extract()
        # print(imgurl)
        name=response.xpath("//div[@class='main-image']/p/a/img/@alt").extract()
        imgname=response.xpath("//h2[@class='main-title']/text()").extract()[0][15:]
        dirs='/home/gong/gg/Images/%s'%name[0][15:]
    */
    public static  void parse_img(String url,int page){
        List<List<String>> img = parse(url, page);
        if (img==null){
            System.out.println("网络");
            return;
        }
        for (List<String> list : img) {
            for (String s : list) {

                String html = getHtml(s);
                if (html==null){
                    continue;
                }
                Document document = Jsoup.parse(html);
                String name= Xsoup.compile("//div[@class='main-image']/p/a/img/@alt").evaluate(document).get();
                String imgname=Xsoup.compile("//h2[@class='main-title']/text()").evaluate(document).get();
                String imgurl=Xsoup.compile("//div[@class='main-image']/p/a/img/@src").evaluate(document).get();
                System.out.println("=====>>>"+name);
                System.out.println("=====>>>"+imgurl);
                System.out.println(imgname);
                String path="F:\\meizi\\"+name+"\\";
                CloseableHttpClient httpClient= createSSLClientDefault();
                try{
                    HttpGet httpGet=new HttpGet(imgurl);
                    System.out.println("url="+s);
                    System.out.println("img="+imgurl);
                    httpGet.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3");

                    httpGet.setHeader("Referer",s);
                    httpGet.setHeader("User-Agent", UA());
                    CloseableHttpResponse response=httpClient.execute(httpGet);
                    if (response.getStatusLine().getStatusCode()== HttpStatus.SC_OK){
                        HttpEntity httpEntity=response.getEntity();
                        InputStream imghtml = httpEntity.getContent();
//                        byte[] bs=new byte[4096];
//                        int len=0;
                        if (imghtml==null){
                           continue;
                        }
                        File file=new File(path);
                        if (!file.exists()){
                            file.mkdir();
                        }
                        FileUtils.copyToFile(imghtml,new File(path+"\\"+imgname+".jpg"));
//                        FileOutputStream os = new FileOutputStream("F:\\meizi" + "\\" + imgname+".jpg");
//                        while ((len=imghtml.read(bs))!=-1){
//                            os.write(bs,0,len);
//                        }
                        System.out.println(file.getPath()+"\\"+imgname+"下载完成");

                        Thread.sleep(1000);

                        response.close();

                    }
                    else {
                        System.out.println("not 200");
                        System.out.println(EntityUtils.toString(response.getEntity()));
                    }

                }catch (Exception e){
                    e.printStackTrace();
                } finally {
                    HttpClientUtils.closeQuietly(httpClient);
                }
                
            }
        }
    }
    public static void main(String[] args) {
//        String b = UA();
//        System.out.println(b);
        String url="https://www.mzitu.com/xinggan/page";

            parse_img(url,1);


//        parse(url,1);


    }
}
gllvcpp
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
【技术应用】java通过url爬虫获取公众号文章内容
dylan的博客
12-01 3101
java通过url爬虫获取公众号文章内容，同时涉及CSDN文章内容爬取，基于开源工具mlunit实现
哈哈哈，真香！妹子图终于可以多图预览了
不是太高的手的专栏
03-18 1万+
‍‍前言群里不少小伙伴在问妹子图为啥点开任何一张图都是第一张妹子图，真的不是难为各位小伙伴，这其实就是个 BUG，藏了很久很久很久的 BUG，不是撸主不改，是撸主没心情改，主要是阿里云的 ...
妹子图 Spider
努力奋斗
11-28 3549
#!bin/python #-*- coding: UTF-8 -*- '''妹子图 spider URL: www.meizitu.com 抓取网站所有图片 ''' import re import string import urllib, urllib2 import os, sys import multiprocessing def getImageUrls(pag
干货集中营（分享）-每日分享妹子图和技术干货
nyyin@qq.com
04-25 5万+
说明网址：http://gank.io/API接口：http://gank.io/api根据API接口做的一个客户端代码，客户端不是我做的，是@gaolonglong做的。 项目github地址：https://github.com/gaolonglong/GankGirl我学习了这个代码，然后在这里分享一下，对代码中的一些东西稍微改动了一下，然后分享出来，大家可以学习一下。本来添加了一些界面，然
Python学习之爬虫（小甲鱼）
热门推荐
jianglz的博客
03-06 13万+
依葫芦画瓢 用字符串查找图片地址下载 图片放在当前目录 GIF下载下来不会动..... import urllib.request import time def open_url(url): #return htmlpage print(url) req = urllib.request.Request(url) req.add_he...
基于scrapy爬取51job爬虫系统源码.zip
03-05
基于scrapy爬取51job爬虫系统源码.zip 基于scrapy爬取51job爬虫系统源码.zip 基于scrapy爬取51job爬虫系统源码.zip 基于scrapy爬取51job爬虫系统源码.zip 基于scrapy爬取51job爬虫系统源码.zip 基于scrapy爬取51job...
scrapy 爬取东方财富网源码
最新发布
05-02
总结来说，通过Scrapy爬取东方财富网源码，我们需要理解Scrapy框架的各个组件，定义Spider规则，处理请求和响应，定义Item和Pipeline，以及配置项目设置。在实践中，还需要注意遵循网站的robots.txt规则，尊重网站...
Scrapy demo 爬取某房产网站源码
08-07
3. 插入数据：通过会话实例将Scrapy爬取到的Item转化为模型对象并保存到数据库。 4. 查询数据：使用ORM的方式执行SQL查询，获取数据库中的数据。 在"t1"这个文件中，可能包含了Scrapy爬虫项目的配置文件、Spider类...
scrapy爬取文章站源码(可直接执行)
09-18
本文将深入探讨如何使用Scrapy框架来爬取文章网站的源码，并结合提供的`seventeen_news`项目，帮助初学者快速上手。 首先，我们需要安装Scrapy。在命令行中，使用以下命令进行安装： ```bash pip install scrapy ``...
Python使用Scrapy爬取妹子图
09-22
本示例中，我们将探讨如何使用Scrapy爬取妹子图网站（meizitu.com）上的图片。首先，我们需要了解Scrapy的基本结构，包括Spider、Item、Item Loader和Middleware等组件。 1. **Scrapy Spider**： Spider是Scrapy的...
妹子图APP（四）—— SQLite保存数据实现离线图片查看
xhuww的博客
03-10 6646
妹子图APP（三）—— SQLite保存数据实现离线图片查看,设置判断是否联网以及监听网络变化
妹子图APP（一）—— Retrofit+Glide+Gson加载网络图片
xhuww的博客
02-28 4万+
使用Retrofit+Glide+Gson实现简单的网络图片加载
妹子图简单爬取
weixin_42948137的博客
06-13 305
使用urllib库 和正则表达式 进行的简单图片爬取，由于还不是特别精通，有些过程处理的不尽人意… 妹子图网址 ''' 注：别忘了将file的路径重新设置 如果在自己电脑不行，可以自行加代理，或者延时一下 ''' import urllib.request import re #import time import urllib.error # ip = ['122.193.24...
福利！！！Python爬取好看的妹子图！！！
2301_81749759的博客
04-11 759
学习的零基础资料，也有适合3年以上经验的小伙伴深入学习提升的进阶课程，基本涵盖了95%以上前端开发知识点，真正体系化！**
利用scrapy爬取图片注意点以及源码
Iridescent_1的博客
01-16 777
版本scrapy==2.5.1 目标url是图片、图片下载、高清图片、图片大全_站长素材，选择爬取第一页的高清图片。利用xpath去进行数据解析的工作。 利用terminal使用命令scrapy startproject imgsPro创建一个新的项目 之后需要将文件位置转移到imgsPro，利用命令cd imgsPro（返回上一层使用命令cd ..），然后使用命令scrapy genspider img www.xxx.com创建出一个py文件。 首先，项目大体结构有： test1 ├─im
教你用Python爬取妹子图APP
weixin_34253539的博客
08-30 2790
教你用Python爬美之图APP全站图片 爬取结果 程序只运行了2h,最后认为程序没有问题了就关了(我可不是去杀生去了...... 运行环境 Python 3.5+ Windows 10 VSCode 如何使用 下载项目源码 https://github.com/cexll/Mzitu.net_Spider.git 安装依赖 $ pip install requests 运行 p...
妹子图APP（二）—— SwipeRefreshLayout实现下拉刷新上拉加载
xhuww的博客
02-28 2493
SwipeRefreshLayout+RecyclerView实现下拉刷新上拉加载
100万+的妹子图微服务版本终于开源了
不是太高的手的专栏
07-20 2862
点击▲关注 “爪哇笔记” 给公众号标星置顶更多精彩 第一时间直达前言今天，妹子图微服务版本终于出炉了，各位小伙伴久等了。2019年11月22日鉴黄图床上线了，网友们也都很积极，甚是踊...
scrapy爬取源码是一行
05-24
使用Scrapy爬取网页源码通常需要编写一个爬虫，具体的步骤如下： 1. 创建项目：在命令行中通过 `scrapy startproject project_name` 命令创建一个新的Scrapy项目。 2. 定义Item：定义一个Item类来存储我们要爬取的内容。 3. 编写Spider：编写一个Spider类来定义如何爬取数据。 4. 定义Pipeline：定义一个Pipeline类来处理爬取到的数据。 5. 启动爬虫：通过命令行运行 `scrapy crawl spider_name` 命令启动爬虫。 在Spider中，使用`response.body`可以获取网页源码。例如，在`parse`方法中可以使用如下代码获取网页源码： ``` def parse(self, response): html = response.body # 处理html yield item ```

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值