Python connects to MongDB

Today, I finished the connection work from Python to MongDB. It uses the module pymongo for purpose.


I'll give the codes directly, because it's very cold here, and I gotto go now...


import re
import json
import codecs
import urllib.request
from bs4 import BeautifulSoup
from pymongo import MongoClient

def getHtml(url):
    page = urllib.request.urlopen(url)
    html = page.read()
    return html

def New(link, dep, son, far):
    global col
    url = {
                'link': link,
                'dep': dep,
                'son': son,
                'far': far
            }
    urllist.append(url)
    """print(json.dumps(url, sort_keys=True, indent=4, separators=(',', ': ')))"""
    col.insert(url)

def PrintSoup(soup):
    file = codecs.open('soup.txt', 'w+', 'utf-8')
    file.write(soup.prettify())
    file.close()

def Develop(point):
    url = urllist[point]
    link = url.get('link')
    html = getHtml(link)
    soup = BeautifulSoup(html, 'html.parser')
    for ai in soup.find_all('a'):
        href = str(ai.get('href'))
        if re.match(r'^https?://.+$', href):
            New(href, url.get('dep') + 1, 0, url.get('link'))
            url['son'] += 1

client = MongoClient('localhost', 27017)
db = client.Links
col = db.Links
point = 0
urllist = []
New("https://image.baidu.com", 1, 0, "")
while len(urllist) < 3000:
    if point >= len(urllist):
        break
    Develop(point)
    point = point + 1

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值