对爬取查询数据demo的功能优化与扩充

对爬取查询数据demo的功能优化与扩充

在之前写的那篇一个简单的爬取查询数据的springboot项目基础之上，考虑到前端展示任务进度的效果，添加了Bootstrap自带的组件–进度条的功能(更多组件参考官方文档)，同时也为了能在页面看到后台控制台的日志，查了下怎样能够实时输出日志，websocket配置简单点用来监听。

先看改良后的效果图

项目上传到了github，有兴趣可以看看：

https://github.com/eli719/crawler

记录下修改的地方：

1.实体类Customer添加属性映射:

	//文件状态
	private List<Boolean> exist;
	//生成的文件名
    private List<String> filesName;
	//任务执行状态：0 未执行;1 已执行
    private List<Integer> doStatus;

2.去掉了Customers，dao层合并了CustomerDao和VerifyDao,用PageInfo来存储json数据内容

	public class PageInfo {
    public static Map<String, Customer> map;
    public static Map<String, Customer> verifyMap;

    public static void instance () {
        map = init ("customer.json");
        verifyMap = init ("verifyCustomer.json");
    }

3.不用链表的方式来存储表头，改用json存储，配置TitleConfig读取解析

public class TitleConfig {
    public static List<List<String> > getTitle(){
        JSONArray customers = parseJsonFile ("title.json", "titles");
        List<List<String>> lists=new ArrayList<> ();
        Iterator<Object> iterator = customers.iterator ();
        while (iterator.hasNext ()){
            Object next = iterator.next ();
            lists.add ((List<String>) next);
        }
        return lists;

    }

4.添加拦截器拦截登录及静态资源映射

 @Override
    public void addInterceptors (InterceptorRegistry registry) {
        registry.addInterceptor (new HandlerInterceptor () {
            @Override
            public boolean preHandle (HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
                Object user = request.getSession ().getAttribute ("user");
                if (user == null) {
                    request.setAttribute("msg","无权限请先登录");
                    request.getRequestDispatcher("/").forward(request, response);
                    return false;
                }
                return true;
            }
        }).addPathPatterns ("/**").excludePathPatterns ("/login","/index.html","/","/asserts/**");
    }

    @Override
    public void addResourceHandlers (ResourceHandlerRegistry registry) {
        String path = System.getProperty("user.dir")+"\\src\\main\\resources\\static\\verifyCode\\";
        String os = System.getProperty("os.name");
        if (os.toLowerCase().startsWith("win")) {
            registry.addResourceHandler("/verifyCode/**").
                    addResourceLocations("file:"+path);
        }else{
            registry.addResourceHandler("/verifyCode/**").
                    addResourceLocations("file:"+path);
        }
    }

5.添加了对象封装状态码和信息及数据

public class Result<T> {

    private Integer code;
    private String message;
    private T data;

6.引入websocket,监听控制台日志及文件执行状态

		<dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-websocket</artifactId>
        </dependency>

后台要配置一下websocket服务端，@ServerEndpoint，value = "/websocket/{id}"来映射处理的客户端请求，configurator = MyEndpointConfigure.class是注入服务的，encoders = MessageEncoder.class是将string类型信息封装到对象PageMessage中发送到客户端。

@Slf4j
@Component
@ServerEndpoint (value = "/websocket/{id}", configurator = MyEndpointConfigure.class,encoders = MessageEncoder.class)
public class LoggingWSServer {
    private static Pattern r = Pattern.compile("[\\d+][\\d+][\\d+][\\d+]-[\\d+][\\d+]-[\\d+][\\d+] [\\d+][\\d+]:[\\d+][\\d+]:[\\d+][\\d+]");

    @Value ("${spring.application.name}")
    private String applicationName;

    /**
     * 连接集合
     */
    private static Map<String, Session> sessionMap = new ConcurrentHashMap<> ();
    private static Map<String, Integer> lengthMap = new ConcurrentHashMap<> ();

    
     private void send(Session session, String message) {
        try {
            if("ws://localhost:18080/websocket/2".equals (session.getRequestURI ().toString ())){
                CustomerController.pageMessage.setLog (message);
                session.getBasicRemote ().sendObject (CustomerController.pageMessage);
            }
            if("ws://localhost:18080/websocket/1".equals (session.getRequestURI ().toString ())){
                VerifyController.pageMessage.setLog (message);
                session.getBasicRemote ().sendObject (VerifyController.pageMessage);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

客户端配置在对应请求页面：

 function socket(add) {
        //websocket对象
        let websocket = null;

        //判断当前浏览器是否支持WebSocket
        if ('WebSocket' in window) {
            websocket = new WebSocket("ws://localhost:18080/websocket/2");

        } else {
            console.error("不支持WebSocket");
        }
        //连接发生错误的回调方法
        websocket.onerror = function (e) {
            console.error("WebSocket连接发生错误");
        };

        //连接成功建立的回调方法
        websocket.onopen = function () {
            console.log("WebSocket连接成功")
        };

        //接收到消息的回调方法
        websocket.onmessage = function (event) {
            var count = 0;
            var list = JSON.parse(event.data);
            var load = list.doStatus;
            for (var key in load) {
                var element = $('#' + key);
                var sum = load[key][0] + load[key][1] + load[key][2];
                switch (sum) {
                    case 1:
                        element.css("width", "30%");
                        element.text("30%");
                        break;
                    case 2:
                        element.css("width", "60%");
                        element.text("60%");
                        break;
                    case 3:
                        element.css("width", "100%");
                        element.text("100%");
                        count++;
                        break;
                }
            }
            var message = list.log;
            //追加
            if (message!=null) {
                //日志内容
                let $loggingText = $("#loggingText");
                $loggingText.append(message);

                //是否开启自动底部
                if (window.loggingAutoBottom) {
                    //滚动条自动到最底部
                    $loggingText.scrollTop($loggingText[0].scrollHeight);
                }
            }
            if (count === Object.keys(load).length) {
                websocket.close();
            }

        };

        //连接关闭的回调方法
        websocket.onclose = function () {
            console.log("WebSocket连接关闭")
        };
    }

7.页面样式修改，加了进度条组件

<td style="padding-top: 10px">
                        <div class="progress">
                            <div th:id="${customer.orgcode}" class="progress-bar" role="progressbar"
                                 aria-valuenow="0" aria-valuemin="0" aria-valuemax="100" style="width: 0;">
                            </div>
                        </div>
                    </td>

更多的组件可以看看Bootstrap组件文档