由于下载速度太慢,打算重载QueueAssignmentPolicy这个类,并重写它的getClassKey()这个方法,网上的代码有:
- public String getClassKey(CrawlController controller, CandidateURI cauri) {
- String uri = cauri.getUURI().toString();
- long hash = ELFHash(uri);
- String a = Long.toString(hash % 100);
- return a;
- }
- public long ELFHash(String str)
- {
- long hash = 0;
- long x = 0;
- for(int i = 0; i < str.length(); i++)
- {
- hash = (hash << 4) + str.charAt(i);
- if((x = hash & 0xF0000000L) != 0)
- {
- hash ^= (x >> 24);
- hash &= ~x;
- }
- }
- return (hash & 0x7FFFFFFF);
- }
但是重写后怎么才能用呢?这个问题我研究了一上午。
这个类的作用是在Frontier为线程提供链接的时候提供链接队列的分配策略(默认是org.archive.crawler.frontier.HostnameQueueAssignmentPolicy这个类来提供)由于一般是选用BdbFrontier,(但是BdbFrontier中没有)-->查看父类WorkQueueFrontier(也没有)-->查看父类AbstractFrontier(发现了)
- public final static String ATTR_QUEUE_ASSIGNMENT_POLICY =
- "queue-assignment-policy";
并在其构造方法中发现了对此变量引用的足迹
- // Read the list of permissible choices from heritrix.properties.
- // Its a list of space- or comma-separated values.
- //通过“ ATTR_QUEUE_ASSIGNMENT_POLICY”(queue-assignment-policy)获得此键的属性
- String queueStr = System.getProperty(AbstractFrontier.class.getName() +
- "." + ATTR_QUEUE_ASSIGNMENT_POLICY,
- HostnameQueueAssignmentPolicy.class.getName() + " " +
- IPQueueAssignmentPolicy.class.getName() + " " +
- BucketQueueAssignmentPolicy.class.getName() + " " +
- SurtAuthorityQueueAssignmentPolicy.class.getName() + " " +
- TopmostAssignedSurtQueueAssignmentPolicy.class.getName());
- //
- Pattern p = Pattern.compile("//s*,//s*|//s+");
- //通过正在表达式切分改String,返回一个String数组
- String [] queues = p.split(queueStr);
- if (queues.length <= 0) {
- throw new RuntimeException("Failed parse of " +
- " assignment queue policy string: " + queueStr);
- }
- //创建一个SimpleType加入到definitionMap中
- /*具体方法体为:
- public Type addElementToDefinition(Type type) {
- if (isInitialized()) {
- throw new IllegalStateException(
- "Elements should only be added to definition in the " +
- "constructor.");
- }
- if (definitionMap.containsKey(type.getName())) {
- definition.remove(definitionMap.remove(type.getName()));
- }
- definition.add(type);
- definitionMap.put(type.getName(), type);
- return type;
- }
- */
- t = addElementToDefinition(new SimpleType(ATTR_QUEUE_ASSIGNMENT_POLICY,
- "Defines how to assign URIs to queues. Can assign by host, " +
- "by ip, and into one of a fixed set of buckets (1k).",
- queues[0], queues));
其注释也清楚的说明了://Read the list of permissible choices from heritrix.properties
查看heritrix.properties:
- #####################################################
- # FRONTIER
- #####################################################
- # List here all queue assignment policies you'd have show as a
- # queue-assignment-policy choice in AbstractFrontier derived Frontiers
- # (e.g. BdbFrontier).
- org.archive.crawler.frontier.AbstractFrontier.queue-assignment-policy =/
- org.archive.crawler.frontier.HostnameQueueAssignmentPolicy/
- org.archive.crawler.frontier.IPQueueAssignmentPolicy/
- org.archive.crawler.frontier.BucketQueueAssignmentPolicy/
- org.archive.crawler.frontier.SurtAuthorityQueueAssignmentPolicy/
- org.archive.crawler.frontier.TopmostAssignedSurtQueueAssignmentPolicy
- org.archive.crawler.frontier.BdbFrontier.level = INFO
进一步看:
- new SimpleType(ATTR_QUEUE_ASSIGNMENT_POLICY,
- "Defines how to assign URIs to queues. Can assign by host, " +
- "by ip, and into one of a fixed set of buckets (1k).",
- queues[0], queues)
该方法最终继承至attribute并把ATTR_QUEUE_ASSIGNMENT_POLICY,queues[0]传入Attribute构造方法中。
queues[0]即为org.archive.crawler.frontier.HostnameQueueAssignmentPolicy。
由以上论述分析可知:
如果想重写QueueAsignmentPolicy,yournameQueueAsignmentPolicy放到AbstraceFrontier.java中的:
- String queueStr = System.getProperty(AbstractFrontier.class.getName() +
- "." + ATTR_QUEUE_ASSIGNMENT_POLICY,
- MyQueueAsignmentPolicy.class.getName()+" "+
- HostnameQueueAssignmentPolicy.class.getName() + " " +
- IPQueueAssignmentPolicy.class.getName() + " " +
- BucketQueueAssignmentPolicy.class.getName() + " " +
- SurtAuthorityQueueAssignmentPolicy.class.getName() + " " +
- TopmostAssignedSurtQueueAssignmentPolicy.class.getName());
在heritrix.propertits中修改如下:
- #############################################################################
- # FRONTIER
- #############################################################################
- # List here all queue assignment policies you'd have show as a
- # queue-assignment-policy choice in AbstractFrontier derived Frontiers
- # (e.g. BdbFrontier).
- org.archive.crawler.frontier.AbstractFrontier.queue-assignment policy = /userP.myQueueAssignmentPolicy/
- org.archive.crawler.frontier.HostnameQueueAssignmentPolicy/
- org.archive.crawler.frontier.IPQueueAssignmentPolicy/
- org.archive.crawler.frontier.BucketQueueAssignmentPolicy/
- org.archive.crawler.frontier.SurtAuthorityQueueAssignmentPolicy/
- org.archive.crawler.frontier.TopmostAssignedSurtQueueAssignmentPolicy
- org.archive.crawler.frontier.BdbFrontier.level = INFO
OK!