如何把 nginx log 批量导入 mongodb

最新推荐文章于 2022-03-27 20:12:05 发布
最新推荐文章于 2022-03-27 20:12:05 发布
阅读量504 收藏 2
点赞数
分类专栏： linux 文章标签： linux shell mongodb nginxlog
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/HC1986/article/details/101529972
版权

@[TOC] 如何把 nginx log 批量导入 mongodb

如何把 nginx log 批量导入 mongodb

记一次需要统计nginx log，之前使用过 用 filebase+ logstash + Elasticsearch 的方式 导入 es库中，用于数据统计。但之前的环境都被我删除了，现在要重新搭 一整套 elk 环境还是比较花时间。所以就想 把 nginx log导入到mongodb来做一些简单的 数据统计。

  1. 把一个大的日志文本 取前100行先做文本处理测试，目标是把nginx log文件格式化为 csv格试，来导入mongodb。
# head -100 access.2019.09.26.log > access.test.log

下面的我服务器的 nginx log 日志格式。

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" "$http_host"';

115.223.196.221 - - [26/Sep/2019:00:00:22 +0800] "GET /static/js/jquery-1.6.2.min.js?ver=1.0 HTTP/1.1" 200 91556 "https://yhh5.sipaphoto.com/h5/fdc8bd32-9f7c-a94a-9984-cecbae942f9b.html?__noHeadPic__=0&__adcf__=&__docSource__=91110105MA01GJ8G99&t_oaid=ls_m3%2BCoXIKqR0Un2Rd7424SQ%3D%3D" "Mozilla/5.0 (Linux; Android 5.1; OPPO A37m Build/LMY47I; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/43.0.2357.121 Mobile Safari/537.36" "115.223.196.221" "yhh5.sipaphoto.com"
123.139.85.100 - - [26/Sep/2019:00:00:22 +0800] "GET /static/js/jquery-1.6.2.min.js?ver=1.0 HTTP/1.1" 200 91556 "-" "Mozilla/5.0 (Linux; Android 7.1.1; OPPO R11 Build/NMF26X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/55.0.2883.91 Mobile Safari/537.36" "123.139.85.100" "yhh5.sipaphoto.com"
140.143.221.200 - - [26/Sep/2019:00:00:22 +0800] "GET / HTTP/1.1" 200 27 "-" "-" "-" "yhh5.sipaphoto.com"
49.119.241.255 - - [26/Sep/2019:00:00:22 +0800] "GET /static/js/jquery-1.6.2.min.js?ver=1.0 HTTP/1.1" 200 91556 "-" "Mozilla/5.0 (Linux; Android 7.1.1; OPPO R11 Plus Build/NMF26X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/55.0.2883.91 Mobile Safari/537.36" "49.119.241.255" "yhh5.sipaphoto.com"

2.目示格式
cvs文本格式是第一行对应的是字段，第二行开始是数据，每个字段的数据是双引号加豆号隔开。
如：

ip,date,request,status,body_bytes_sent,http_user_agent,http_x_forwarded_for,http_host
“49.119.241.255”,"26/Sep/2019:00:00:22 +0800", "GET /static/js/jquery-1.6.2.min.js?ver=1.0 HTTP/1.1"," 200" ,"91556", "Mozilla/5.0 (Linux; Android 7.1.1; OPPO R11 Plus Build/NMF26X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/55.0.2883.91 Mobile Safari/537.36","49.119.241.255" ,"yhh5.sipaphoto.com"

3.处理步骤

3.1.处理下特殊字符串（ 中括号，豆号，冒号，分好，括号）替换成 空格
# perl -pi -e 's|\"||g' access.test.log //把 " 替换成空格
#perl -pi -e 's/\[//g' test1.log //中括号替换成空格
#perl -pi -e 's/\]//g' test1.log //中括号替换成空格
#perl -pi -e 's/\(//g' test1.log //括号替换成空格
#perl -pi -e 's/\)//g' test1.log //括号替换成空格
#perl -pi -e 's/\;//g' test1.log //分号替换成空格
#perl -pi -e 's/\,//g' test1.log //豆号替换成空格
# perl -pi -e 's/uuid\///g' access.test.log  //把uuid/替换成空格
#sed -ie '/static/d' access.test.log //删除 带有static字符串的行
#sed -ie '/140.143.221.200/d' access.test.log  //删除 ip为 114.143.221.200 的行，这是负载载均的空数据没有统计的意义。
#perl -pi -e 's| |","|g' access.test.log //把空格替换成","
#sed -i 's/^/\"&/g' access.test.log  //在每行的开头添加一个冒号 ”;
#sed -i 's/$/&\"/g' access.test.log  //在每行的行尾添加一个冒号 ”;
#perl -pi -e 's/\"-\"\,//g' access.test.log  把 "-", 这个特殊字符中 替换成空格

处理后如下

"113.16.250.180","26/Sep/2019:00:00:06","+0800","GET","/h5/fdc8bd32-9f7c-a94a-9984-cecbae942f9b.html?__noHeadPic__=0&__adcf__=&__docSource__=91110105MA01GJ8G99&t_oaid=ls_o48XbnwI7W8RzbZZDDmfpw%3D%3D","HTTP/1.1","200","13553","Mozilla/5.0","Linux","Android","6.0.1","OPPO","R9s","Build/MMB29M","wv","AppleWebKit/537.36","KHTML","like","Gecko","Version/4.0","Chrome/55.0.2883.91","Mobile","Safari/537.36","113.16.250.180","yhh5.sipaphoto.com"
"106.124.36.115","26/Sep/2019:00:00:11","+0800","GET","/h5/fdc8bd32-9f7c-a94a-9984-cecbae942f9b.html?__noHeadPic__=0&__adcf__=&__docSource__=91110105MA01GJ8G99&t_oaid=ls_PBvTCu1u1ijnBgWfn1liNw%3D%3D","HTTP/1.1","200","13553","Mozilla/5.0","Linux","Android","5.1","OPPO","A59s","Build/LMY47I","wv","AppleWebKit/537.36","KHTML","like","Gecko","Version/4.0","Chrome/43.0.2357.121","Mobile","Safari/537.36","106.124.36.115","yhh5.sipaphoto.com"
"106.127.214.37","26/Sep/2019:00:00:14","+0800","GET","/h5/45af4e52-6043-6725-69b5-a43d45306488.html?__noHeadPic__=0&__adcf__=0_21_1&__docSource__=91110105MA01GJ8G99","HTTP/1.1","200","11890","Mozilla/5.0","Linux","Android","8.1.0","PBBM00","Build/O11019","wv","AppleWebKit/537.36","KHTML","like","Gecko","Version/4.0","Chrome/62.0.3202.84","Mobile","Safari/537.36","pictorial_version/5.6.1","4ae2c231da7b717c4b5d18ff11f497cb","channel/OPPO","language/zh-CN","106.127.214.37","yhh5.sipaphoto.com"
"117.136.39.203","26/Sep/2019:00:00:16","+0800","GET","/h5/fdc8bd32-9f7c-a94a-9984-cecbae942f9b.html?__noHeadPic__=0&__adcf__=&__docSource__=91110105MA01GJ8G99&t_oaid=ls_Ist8GUiaQVIhZQX48WxyvQ%3D%3D","HTTP/1.1","200","13553","Mozilla/5.0","Linux","Android","6.0.1","OPPO","A57t","Build/MMB29M","wv","AppleWebKit/537.36","KHTML","like","Gecko","Version/4.0","Chrome/55.0.2883.91","Mobile","Safari/537.36","117.136.39.203","yhh5.sipaphoto.com"
"223.104.64.8","26/Sep/2019:00:00:18","+0800","GET","/h5/817bd461-731b-4bcf-9da2-f511796914aa.html?__noHeadPic__=0&__adcf__=0_21_1&__docSource__=91110105MA01GJ8G99","HTTP/1.1","200","15699","Mozilla/5.0","Linux","Android","7.1.1","OPPO","A83","Build/N6F26Q","wv","AppleWebKit/537.36","KHTML","like","Gecko","Version/4.0","Chrome/62.0.3202.84","Mobile","Safari/537.36","pictorial_version/5.7.1","337f6ac48692146db8f5123eb14fa82e","channel/OPPO","language/zh-CN","223.104.64.8","yhh5.sipaphoto.com"
"223.104.1.199","26/Sep/2019:00:00:21","+0800","GET","/h5/e748af6e-ad70-2b49-729f-bb7e105733ee.html?__noHeadPic__=0&__adcf__=0_21_1&__docSource__=91110105MA01GJ8G99","HTTP/1.1","200","11375","Mozilla/5.0","Linux","Android","8.1.0","PBBT00","Build/O11019","wv","AppleWebKit/537.36","KHTML","like","Gecko","Version/4.0","Chrome/62.0.3202.84","Mobile","Safari/537.36","pictorial_version/5.7.1","4126a51d25e779b9c31fbcef7e8902b0","channel/OPPO","language/zh-CN","223.104.1.199","yhh5.sipaphoto.com"
第一行添加对应用的字段名
#sed -i "1i ip,date,timezone,request_mode,url,server_protocol,status,body_bytes_sent,agent_0,agent_1,agent_2,agent_3,agent_4,agent_5,agent_6,agent_7,agent_8,agent_9,agent_10,agent_11,agent_12,agent_13,agent_14,agent_15,uuid,mobile_mode,language,host_ip,host_name" access.test.log

对应表

"223.104.1.199"=>ip
"26/Sep/2019:00:00:21"=>date
"+0800"=>timezone
"GET"=>request_mode
"/h5/e748af6e-ad70-2b49-729f-bb7e105733ee.html?__noHeadPic__=0&__adcf__=0_21_1&__docSource__=91110105MA01GJ8G99"=>url
"HTTP/1.1"=>server_protocol
"200"=>status
"11375"=>body_bytes_sent
"Mozilla/5.0"=>agent_0
"Linux"=>agent_1
"Android"=>agent_2
"8.1.0"=>agent_3
"PBBT00"=>agent_4
"Build/O11019"=>agent_5
"wv"=>agent_6
"AppleWebKit/537.36"=>agent_7
"KHTML"=>agent_8
"like"=>agent_9
"Gecko"=>agent_10
"Version/4.0"=>agent_11
"Chrome/62.0.3202.84"=>agent_12
"Mobile"=>agent_13
"Safari/537.36"=>agent_14
"pictorial_version/5.7.1"=>agent_15
"4126a51d25e779b9c31fbcef7e8902b0"=>uuid
"channel/OPPO"=>mobile_mode
"language/zh-CN"=>language
"223.104.1.199"=>host_ip
"yhh5.sipaphoto.com"=>host_name
把cvs格式的nginx log 导入mongodb

1.把格式化好的log日志扩展名改为.csv

蓝_星
关注
  • 0
    点赞
  • 踩
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
使用MongoDB分析Nginx日志的方法详解
09-09
在项目开发过程中,总是离不开日志解析的工作,虽然有些时候觉得确实挺繁琐的,但是静下心来会发现有时候也是挺有趣的1件工作。下面这篇文章主要介绍了使用MongoDB分析Nginx日志的方法,需要的朋友可以参考借鉴，一起来看看吧。
nginx日志写入mongodb
weixin_34267123的博客
02-11 484
2019独角兽企业重金招聘Python工程师标准>>> ...
Go语言实战-nginx日志处理之MongoDB篇
qq_36034503的博客
03-27 2763
golang处理nginx日志，mongodb实战，nginx日志分析处理
nginx mongodb相关配置
zhaoyangjian724的专栏
01-27 1233
mongdb外部访问: redis01:/root# /usr/local/mongodb/bin/mongod --bind_ip 192.168.32.34 -f /usr/local/mongodb/bin/mongodb.conf about to fork child process, waiting until server is ready for connections. for
nginx 中用 sed 批量增加配置文件内容
weixin_33738555的博客
01-09 535
#!/bin/bash confs=`ls /etc/nginx/conf.d/cms_vhosts/` cd /etc/nginx/conf.d/cms_vhosts for log in $confs;do name=`echo ${log} |awk -F '.' '{print $1}'` sed -i "/root/a\error_log /zz_data/n...
nginx_log_analysis:nginx日志分析
05-26
nginx_log_analysis是一个Nginx日志实时分析系统，目前已经在折800旗下的全部Nginx代理上运行，每天负责数十亿日志的实时分析。整个传输操作在log执行阶段，对请求的响应时间没有影响。如果对Ngx_Lua开发有兴趣的...
MongoDB+Nginx配置
01-11
MongoDB+Nginx配置,本文档主要以Centos为例,一步步演示如何进行操作,帮助小伙伴们快速完成功能搭建操作.
基于Storm的Nginxlog实时监控系统
02-21
本文介绍的是基于Storm的Nginxlog实时监控系统。Hadoop的缺点也和它的优点同样鲜明——延迟大，响应缓慢，运维复杂。被人广受诟病，但是 有需求就有创造，在Hadoop基本奠定了大数据霸主地位的时候，很多的开源项目...
docfony：使用PHP，NGINX，MySQL，MongoDB和Redis的Docker Symfony开发堆栈
02-06
NGINX 1.17 MySQL 8.0 蒙哥4.0 Redis 5.0 要求 Docker支持的操作系统（请签出） Docker版本17.x （请访问） 应该启用Docker Compose Symfony框架4+ 下载 在开始运行容器之前，请确保在主机的相同端口上没有...
实战6 Nginx的web日志导入到MongoDB数据库
weixin_34252090的博客
06-17 264
背景公司是做视频服务器的，为了统计每个视频的点击率来做热播的排行榜，于是采用了从nginx的访问日志来着手，并且有Cookid来做唯一值，防止重复的访问。一、获取Cookid二、nginx日志截取三、nginx的日志导入一、获取Cookid二、nginx日志截取感谢群友的帮助：雪落纷飞-1009556020#!/bin/bash file=/usr/local/nginx/...
针对小度在家的设备分析（以1C为例，非破解）
热门推荐
LeeKwen的专栏
08-01 8万+
概述 语音类的机器人已经接触和分析了很多了，但百度出的类似产品没有写过！ 主要是没有机会接触到实体的机器，大写的尴尬。 因对接的需要，特申请了一台，那就借此机会把玩一下吧。 拆机的部分请移步他处，给你个火箭，点击链接直达！ 小度在家的拆解过程，见如下的地址： http://www.52audio.com/archives/3290.html 但，实际我拿到的1C设备，不是上面型号的机...
Nginx反向代理报504超时错误
OperationsMAN
08-06 1万+
nginx+tomcat 后端为tomcat,nginx代理报504超时错误。 问题描述： #错误 1.198.17.123 - - [06/Jul/2018:01:48:57 +0000] "POST /mapbj3/getticket HTTP/1.1" 504 537 "https://XXXXXXXXXX.com/walkcode3/index.html?openId=oB6U...
springboot批量上传文件，在nginx和网关解决文件总量过大的问题
qq_43173523的博客
12-26 752
最近在做上传文件，当批量上传文件的时候，报错：413 Request Entity Too Large的解决办法 于是，查询得知是nginx对于文件大小有控制，所以在nginx中的http节点下增加配置如下： client_max_body_size 100m; nginx中解决后，批量上传文件后又报网关超时，所以在网关和上传文件的微服务中增加如下配置： spring.http.multipa...
[bigdata-032] 把json文件导入到mongodb
大数据和机器学习研究
01-19 581
用mongoimport命令 mongoimport -d db1 -c colle1 --type json --file cc.json --host 114.55.66.77 --upsert -u user1 -p passwd1 这个命令，如果同id的记录在，就更新，如果记录id不重复，就插入。因为upsert，所以会比较慢。 mongoimport -d db1 -c colle
python写入文件到mongoDB
Ivan_XZ的博客
07-20 2876
将图片、文档等存入mogodb中，需要用到两个模块，pymongo和gridfs，在存入过程中必须将文件转换为二进制的格式才能存入，存入mongoDB中会生成两个文件xxx.chunk 和xxx.files。直接上代码：import pymongo import gridfs from bson import ObjectIdclient = pymongo.MongoClient('localho
将数据写入MongoDB数据库
runs_after_the_wind的博客
12-12 1440
import requests from lxml import etree import pymongo class article(object): def __init__(self): # 创建连接，ip地址和端口号 self.client = pymongo.MongoClient("iP地址", 端口号) # 连接的数据库 ...
Nginx服务器可能影响文件上传的因…
只为成功找方法-不为失败找理由
09-13 1750
修改了php.ini文件如下： 参数 设置 说明 file_uploads on 是否允许通过HTTP上传文件的开关。默认为ON即是开 upload_tmp_dir – 文件上传至服务器上存储临时文件的地方，如果没指定就会用系统默认的临时文件夹 upload_max_filesize 8m 望文生意，即允许上传文件大小的最大值。默认为2M post_max_size 8m
Nginx访问MongoDB
liyjw的专栏
08-29 8670
MongoDB在网站中使用较多，主要使用它来存储静态文件，例如图像文件，因此在这一讲我们将来认识一下nginx-gridfs模块，使用该模块来实现对MongoDB的访问，实际上就是MongoDB的客户端。在具体的安装中要注意它和数据库驱动的版本问题，有时候不是很好安装。  原文链接http://space.itpub.net/?uid-27043155-action-viewspace-item
nginx反向代理 mongodb
最新发布
11-08
nginx反向代理mongodb是指使用nginx作为反向代理服务器，将客户端的请求转发到后端的mongodb服务器上。通过这种方式，可以实现负载均衡和高可用性。具体实现方法是在nginx服务器上进行配置，将mongodb服务器的地址和...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值