nutch2.3.1爬取marker流程

crawlstatus:
    STATUS_UNFETCHED = 0x01;     //Page was not fetched yet
    STATUS_FETCHED = 0x02;       //Page was successfully fetched
    STATUS_GONE = 0x03;          //Page no longer exists
    STATUS_REDIR_TEMP = 0x04;    //Page temporarily redirects to other page
    STATUS_REDIR_PERM = 0x05;    //Page permanently redirects to other page
    STATUS_RETRY = 0x22;         //Fetching unsuccessful, needs to be retried (transient errors)
    STATUS_NOTMODIFIED = 0x26;   //Fetching successful - page is not modified
injectorjob
    **_injmrk_ :'y'**
    distance:0
generatorjob
    生成batchId
    判断distance> maxDistance return
    _gnmrk_ 有值return
    **fetchTime 太近return**
    count >= limit return
    计算url的score
    **_gnmrk_ = batchId**
    page.batchId = batchId
fetcherjob
    _gnmrk_ 无值return
    _ftcmrk_ 有值return
    **batchId.equals(_gnmrk_) 则fetch**
    **_ftcmrk_ = _gnmrk_ **
parsejob
    **batchId.equals(_ftcmrk_) 则parse**
    batchId.equals("-reparse") 或 force 强制parse
    _ftcmrk_ 无值return
    __prsmrk__ 有值return
    skipTruncated return
    status 不等STATUS_FETCHED return
    parse:setSignature setOutlinks
    __prsmrk__ = _ftcmrk_
dbupdaterjob
    **batchId.equals(_gnmrk_) 则update**
    *outlinks插入数据库
    *更新score
    *判断Signature的变动
    *更新fetchTime,modifiedTime
    *计算newDistance
    *删除_ftcmrk_ / _gnmrk_
    *如果 __prsmrk__ 存在:_updmrk_ = __prsmrk__ ; __prsmrk__=NULL
indexingjob
    **batchId.equals(_updmrk_) 则 indexing**
    if (pstatus == null || !ParseStatusUtils.isSuccess(pstatus)) return
    indexing
    _idxmrk_ = _updmrk_
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值