mongodb
linux安装
创建一个/etc/yum.repos.d/mongodb-org-3.6.repo文件
然后在里面写入:
[mongodb-org-3.6]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/testing/x86_64/
gpgcheck=0
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-3.6.asc
yum -y install mongodb-org
记得关闭selinux
service mongod start 启动
如果用本地连 ,进入 /etc/mongod.conf
把 127.0.0.1 改为 0.0.0.0 (如果注释不管用的话)
基本操作
#coding:utf8
import pymongo
# 建立连接
mongoclient = pymongo.MongoClient(host='192.168.6.6',port=27017)
# 指定操作数据库
db = mongoclient['alice']
# 指定操作表(collection)
sheet = db['host']
try:
# 增加一条
# sheet.insert({'ip' : '192.168.1.1'})
# db.host.insert({'ip' : '192.168.1.5', 'port' : 27017})
# 修改
# sheet.update({'ip' : '192.168.1.2'},{'$set':{'port':'3306'}},multi=True)
# 删除
# sheet.remove({'ip' : '192.168.1.1'})
#查询
# res = sheet.find({'port' : 27017})
# for item in res:
# print item,type(item)
pass
except Exception,e:
print str(e)
在scrapy里写入 pipeline
写入
import pymongo
class MongoPipeline(object):
def __init__(self):
try:
self.mongoclient = pymongo.MongoClient(host='39.106.37.83',port=27017)
self.db = self.mongoclient['demo']
self.sheet = self.db['host']
except Exception,e:
print ('连接失败')
print str(e)
def process_item(self, item, spider):
try:
self.sheet.insert({'ip' :item['ip'], 'port' : item['dk']})
except Exception,e:
print str(e)
redis
python-redis(分布式爬虫)
pip install python-redis
linux安装
1 下载源码包
2 tar zxvf 解压
3 mv 移动到 /etc/local
4 cd redis里 make 编译
5 make install 安装
6 cd src
7 ./redis-server 启动服务
8 编辑redis-4.0.2下面的redis.conf
69
88 protected-mode no
136 daemonize no
9 复制到etc下
cp redis.conf /etc/redis.conf
7)添加全局变量
export PATH=$PATH:/usr/local/redis-4.0.2/src
8)设置好全局变量之后就可以启动redis了
redis-server /etc/redis.conf
9)关闭redis
kill -9 redis端口
ps -aux | grep redis
mysql
本地连接mysql
1 上阿里云 设置安全组 3306
2 进来linuxmysql
mysql -u root -p
use mysql
grant all privileges on *.* to root@"xxx.xxx.xxx.xxx" identified by "密码";
flush privileges;
select user,password,host from user; #查看是否添加成功
如果还是不行,在linux界面运行
iptables -L -n
iptables -D INPUT 5