Java 爬虫

条件

目标网页：天眼查
条件：注册资金3000万左右
结果数据：公司名称，法人姓名，公司logo

代码

依赖

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>easyexcel</artifactId>
            <version>2.2.5</version>
        </dependency>
        <dependency>
            <groupId>org.jsoup</groupId>
            <artifactId>jsoup</artifactId>
            <version>1.10.2</version>
        </dependency>
        <dependency>
            <groupId>cn.hutool</groupId>
            <artifactId>hutool-all</artifactId>
            <version>5.4.1</version>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>

代码

首先找到天眼查的搜索链接
https://www.tianyancha.com/search

然后通过页面操作看url变化得知，加上/or.xxx.xxx是注册资金范围，再加/px代表页数，再加?key=xxx代表公司关键词

    public static List<CompanyInfo> getData(Integer start, Integer end, String keyWord) throws Exception {
   
        String rowUrl = "https://www.tianyancha.com/search/or." + start + "." + end;
        List<CompanyInfo> infos = new ArrayList<>();
        Map<String, String> map = new HashMap<>();
        //这是验证token，登录后可从浏览器中获得
        map.put("auth_token","eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiIxNzMyODg2MjQ0MyIsImlhdCI6MTYxODM5MzY3OSwiZXhwIjoxNjQ5OTI5Njc5fQ.QJNomA-k4MROR1DdbF7ulCJitlR74tR93PCnJT2mmBgjj7WBz7mwdyV1Fi0069PuZx8S4t6QBh8Y1JiHxvrK5g");
        //由于非vip用户只能免费看前五页，所以这里我循环了5次
        for (int i = 1; i <= 5; i++) {
   
            String url = rowUrl + "/p" + i;
            if (StrUtil.isNotBlank(keyWord)) {
   
                url = url + "?key=" + keyWord;
            }
            Document document = Jsoup.