fastapi日志重复打印_正则分析 Ng 日志

今天丁**想要知道****环境接口的历史并发数，然而运维并没有做相关的统计，没办法，只能拿到服务器近一个月的 Nginx access 日志，根据正则匹配所有我的接口服务的日志，然后统计每一秒内有多少个请求，最后做下排序即可

下载日志

首先从服务器打包下载 Nginx 近一个月的 access log，运维配置了两个 Nginx，对应的日志也是两份，将两份日志下载到本地解压后如下图：

首先为了方便编程，执行命令将所有日志合并为一个文件：

cd ~/Downloads/nginx-api1/
cat access_api_2020-0* > test.log
cat ../nginx-api2/access_api_2020-0* >> test.log

编写正则

然后我们随意打开一份日志文件开始分析如何写正则：

从这份日志文件可以看出，红圈中的都是我们接口的 Url，我们要做的就是用正则匹配这些 Url，不难写出如下正则：

[[0-3][0-9]/(May|Apr)/2020:[0-2][0-9](:[0-5][0-9]){2} +0800] "POST /((register)|(v2/register)|([a-f0-9]{32}(/[a-z_]+)+)|(v2/[a-f0-9]{32}(/[a-z_]+)+))

编写代码

但是我们可以发现这份日志中同一秒内同一辆车的同一请求被打印了多次，这是因为运维 Nginx 配置有误导致的，所以我们需要编写程序，剔除这种重复的日志打印 代码如下：

public static void main(String[] args) throws FileNotFoundException {

    Pattern pattern = Pattern.compile("[[0-3][0-9]/(May|Apr)/2020:[0-2][0-9](:[0-5][0-9]){2} +0800] "POST /(" +
        "(register)|(v2/register)|([a-f0-9]{32}(/[a-z_]+)+)|(v2/[a-f0-9]{32}(/[a-z_]+)+))");
    Scanner scanner = new Scanner(new BufferedInputStream(new FileInputStream("/Users/jing/Downloads/nginx-api1" +
        "/test.log")));

    Map<String, Set<String>> map = new HashMap<>();
    while (scanner.hasNextLine()) {
        String line = scanner.nextLine();
        Matcher matcher = pattern.matcher(line);
        if (matcher.find()) {
            String group = matcher.group();
            String key = group.substring(1, 21);
            String value = group.substring(35);
            if (map.containsKey(key)) {
                Set<String> set = map.get(key);
                set.add(value);
            } else {
                Set<String> set = new HashSet<>();
                set.add(value);
                map.put(key, set);
            }
        }
    }

    ArrayList<Map.Entry<String, Set<String>>> list = new ArrayList<>(map.entrySet());
    list.sort(Comparator.comparingInt(o -> o.getValue().size()));
    
    for (int i = list.size() - 1; i > list.size() - 10; i--) {
        System.out.println(list.get(i).getKey() + " " + list.get(i).getValue().size());
    }
}

程序将每一秒的时间字符串作为 Map 的 Key，每一秒内的所有请求 Url 放入 Set 中，这样重复的 Url 会被自动去除，并将该 Set 作为 Map 的 value，所有数据都放入 Map 后，将 Map 转为 List，并对 List 按照 Set 的 size 从小到大排序，因为这里丁**想要知道每一秒的最大并发请求数，直接输出 list.get(list.size() - 1).getValue().size() 的值就可以了。

程序运行输出如下：

26/Apr/2020:17:18:22 10
05/May/2020:17:10:32 9
26/Apr/2020:18:27:46 9
20/Apr/2020:07:48:15 9
13/May/2020:18:33:20 9
08/May/2020:07:41:20 9
11/May/2020:08:12:20 8
08/May/2020:17:34:17 8
14/May/2020:17:19:23 8

验证

可以看到这近一个月来，****环境每一秒内最多的并发数是 10，发生在 26/Apr/2020:17:18:22。接下来我们找到这个时间点的日志验证，是否真的是 10 个请求：

请求 1 - 100.67.193.14 - - [26/Apr/2020:17:18:22 +0800] "POST /************md5value************/report_sub_info HTTP/1.0" 200 30 "-" "Dalvik/1.6.0 (Linux; U; Android 4.4.2; SABRESD-MX6DQ Build/F5.37)" "-/-" "36.98.184.146" "0.014" "http" "80"
请求 2 - 100.67.95.8 - - [26/Apr/2020:17:18:22 +0800] "POST /************md5value************/check HTTP/1.0" 200 2252 "-" "Dalvik/1.6.0 (Linux; U; Android 4.4.2; SABRESD-MX6DQ Build/F5.37)" "-/-" "122.97.178.42" "0.095" "http" "80"
请求 1 - 100.64.38.11 - - [26/Apr/2020:17:18:22 +0800] "POST /************md5value************/report_sub_info HTTP/1.0" 200 30 "-" "Dalvik/1.6.0 (Linux; U; Android 4.4.2; SABRESD-MX6DQ Build/F5.37)" "-/-" "36.98.184.146" "0.063" "http" "80"
请求 3 - 100.64.83.14 - - [26/Apr/2020:17:18:22 +0800] "POST /************md5value************/logFile/report HTTP/1.0" 200 30 "-" "Dalvik/1.6.0 (Linux; U; Android 4.4.2; SABRESD-MX6DQ Build/F5.37)" "-/-" "122.97.174.37" "0.091" "http" "80"
请求 4 - 100.64.38.6 - - [26/Apr/2020:17:18:22 +0800] "POST /************md5value************/report_sub_info HTTP/1.0" 200 30 "-" "Dalvik/1.6.0 (Linux; U; Android 4.4.2; SABRESD-MX6DQ Build/F5.37)" "-/-" "122.97.178.251" "0.219" "http" "80"
请求 5 - 100.64.23.16 - - [26/Apr/2020:17:18:22 +0800] "POST /************md5value************/report_sub_info HTTP/1.0" 200 30 "-" "Dalvik/1.6.0 (Linux; U; Android 4.4.2; SABRESD-MX6DQ Build/F5.37)" "-/-" "122.97.174.64" "0.008" "http" "80"
请求 6 - 100.67.95.15 - - [26/Apr/2020:17:18:22 +0800] "POST /************md5value************/check HTTP/1.0" 200 39 "-" "Dalvik/1.6.0 (Linux; U; Android 4.4.2; SABRESD-MX6DQ Build/F5.37)" "-/-" "122.97.179.60" "0.122" "http" "80"

请求 7 - 100.64.35.7 - - [26/Apr/2020:17:18:22 +0800] "POST /************md5value************/check HTTP/1.0" 200 39 "-" "Dalvik/1.6.0 (Linux; U; Android 4.4.2; SABRESD-MX6DQ Build/F5.37)" "-//-" "122.97.175.91" "0.104"80
请求 8 - 100.64.23.4 - - [26/Apr/2020:17:18:22 +0800] "POST /************md5value************/check HTTP/1.0" 200 2252 "-" "Dalvik/1.6.0 (Linux; U; Android 4.4.2; SABRESD-MX6DQ Build/F5.37)" "-//-" "122.97.174.64" "0.114"80
请求 9 - 100.67.95.11 - - [26/Apr/2020:17:18:22 +0800] "POST /************md5value************/report_sub_info HTTP/1.0" 200 30 "-" "Dalvik/1.6.0 (Linux; U; Android 4.4.2; SABRESD-MX6DQ Build/F5.37)" "-//-" "122.97.178.42" "0.008"80
请求 9 - 100.67.93.17 - - [26/Apr/2020:17:18:22 +0800] "POST /************md5value************/report_sub_info HTTP/1.0" 200 30 "-" "Dalvik/1.6.0 (Linux; U; Android 4.4.2; SABRESD-MX6DQ Build/F5.37)" "-//-" "122.97.178.42" "0.007"80
请求 10 - 100.64.24.13 - - [26/Apr/2020:17:18:22 +0800] "POST /************md5value************/upgrade/report HTTP/1.0" 200 30 "-" "Dalvik/1.6.0 (Linux; U; Android 4.4.2; SABRESD-MX6DQ Build/F5.37)" "-//-" "122.97.178.128" "0.007"80
请求 5 - 100.64.69.5 - - [26/Apr/2020:17:18:22 +0800] "POST /************md5value************/report_sub_info HTTP/1.0" 200 30 "-" "Dalvik/1.6.0 (Linux; U; Android 4.4.2; SABRESD-MX6DQ Build/F5.37)" "-//-" "122.97.174.64" "0.103"80
请求 5 - 100.64.32.17 - - [26/Apr/2020:17:18:22 +0800] "POST /************md5value************/report_sub_info HTTP/1.0" 200 30 "-" "Dalvik/1.6.0 (Linux; U; Android 4.4.2; SABRESD-MX6DQ Build/F5.37)" "-//-" "122.97.174.64" "0.108"80

去除重复请求后，果然是 10 个请求，证明计算结果无问题

前两天刚学的正则表达式真的派上了用场，哈哈哈

欢迎大家关注我的微信公众号，会不定期分享实际业务架构、算法和学习笔记：