These days I got interested in web scrawler, it seems that getting data from web is very easy, so I wondered if I can write a simple application for this work, but later I got several urgent issues and did not have time to think about this topic,...... the road to happiness full of hardships, finally,I can take a break and continue to investigate.
This work is amazing, at first, I think it might be funny to collect the data and see the connection between members, but before I can do that, I started to realize the the most difficult problem turns out to be how to store the data and how to index them,rather than how to pick them out from the web.
Then I found a book "Managing gigabytes: compressing and indexing documents and images" , after reading the introduction part of it, I thought this should be the book that can provide useful solutions to my problems
indexing and compressing problem in web scrawler
最新推荐文章于 2023-09-23 11:24:45 发布