模拟公网ip访问接口,elk配合geoip采集日志展示各城市对网站的访问情况

参考博客:

ELK-图示nginx中ip的地理位置 - dance_man - 博客园

如何映射用户位置与GeoIP的和ELK（Elasticsearch，Logstash和Kibana）

实践步骤: ( 实操代码在https://download.csdn.net/download/weixin_36013896/33930564)

1 测试环境首先模拟出公网访问的日志(生产环境不需要模拟)

中国电信IP段表：http://www.023wg.com/ISP-CN-IP/ISP-CN-DX.html

把上面的ip段,保存到iplist.txt,通过test.py找出n个公网ip,写入到ip.txt

然后通过api.py发起请求,从ip.txt里的ip中随机取出作为X-Forwarded-For的值,达到伪装ip的目的,

api.py发起请求的接口是通过nginx进行代理的,

nginx的日志格式如下:

log_format main '$remote_addr - $remote_user [$time_local] "$request" '

'$status $body_bytes_sent "$http_referer" '

'"$http_user_agent" "$http_x_forwarded_for"';

首字段remote_addr会作为后续的真实ip,这里需要在nginx配置中把remote_addr,改为X-Forwarded-For,必应出来的做法:

nginx安装需要添加--with-http_realip_module 参数,然后在nginx配置加上以下红色的两行:

                set_real_ip_from   0.0.0.0/0;
                real_ip_header     X-Forwarded-For;
 

        server {
            listen           80;
            server_name      dev-gin-k8s.kkkk.com;
            access_log  /home/logs/ms/dev/backend/access.log main;
            error_log  /home/logs/ms/dev/backend/error.log ;
            location / {
                set_real_ip_from   0.0.0.0/0;
                real_ip_header     X-Forwarded-For;
                proxy_set_header Host      $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_http_version  1.1;
                proxy_connect_timeout  5s;
                proxy_pass http://gin-k8s:8090;
            }
        }

查看nginx日志: 查看首字段变成了公网ip

[root@flask-backend ~]# tail -f /home/logs/ms/dev/backend/access.log

222.90.0.18 - - [23/Oct/2021:00:37:42 +0800] "POST /k8s/get_deployment_list HTTP/1.1" 200 1262 "-" "python-requests/2.26.0" "222.90.0.18"

61.144.0.1 - - [23/Oct/2021:00:37:42 +0800] "POST /k8s/get_deployment_list HTTP/1.1" 200 1262 "-" "python-requests/2.26.0" "61.144.0.1"

60.176.0.23 - - [23/Oct/2021:00:37:42 +0800] "POST /k8s/get_deployment_list HTTP/1.1" 200 1262 "-" "python-requests/2.26.0" "60.176.0.23"

114.138.0.117 - - [23/Oct/2021:00:37:42 +0800] "POST /k8s/get_deployment_list HTTP/1.1" 200 1262 "-" "python-requests/2.26.0" "114.138.0.117"

接下来就简单了:

filebeat采集nginx日志给logstash扔到es里:

filebeat配置

[root@flask-backend ~]# cat /app/filebeat/filebeat.yml
filebeat.prospectors:
- type: log
  paths:
    - /home/logs/ms/dev/backend/access.log
  fields:
    project: ms
    env: dev
    role: proxy
    log_type: nginx
  scan_frequency: 10s
  tail_files: true
  fields_under_root: true
output.logstash:
  hosts: ["192.168.11.200:5045"]

logstash的配置:

[root@flask-backend ~]# cat /app/logstash/etc/config/geoip.conf

[root@flask-backend ~]# cat /app/logstash/etc/config/geoip.conf
input {
    beats {
        port => 5045
    }

}
filter {
    grok {
        match => ["message","%{COMBINEDAPACHELOG}%{SPACE}%{QS:x_forwarded_for}"]
    }
    geoip {
        source => "clientip"
        target => "geoip"
        database => "/data/filebeat/geoip/GeoLite2-City.mmdb"
        add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}" ]
        add_field => ["[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
        remove_field => ['beat']
    }
    date {
        match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
}

output {
     if "_grokparsefailure" not in [tags] {
          elasticsearch {
              hosts => ["http://192.168.11.200:9200"]
              index => "logstash-kibana_nginx-%{+YYYY.MM}"
          }
     }
}

上述的GeoLite2-City.mmdb 文件是进入https://www.maxmind.com/en/accounts/625659/geoip/downloads,注册账号,下载的免费版城市文件,解压就有GeoLite2-City.mmdb,

位置随意,和logstash对应上就可以.

最后通过kibana查看日志:会发现多了geoip.xx等地理位置字段:

kibana Visulize添加的Coordinate Map图表 老版本叫tile map,选择好参数就可以出图了

总结:                 运维无他,就是干