Written by Ian Lurie |
Wednesday, 28 September 2011 12:43 |
You can learn a lot about a search engine by scraping its results.It’s the only easy way you can get an hourly or daily record of exactly what Google, Bing or Yahoo! (you know, backwhen Yahoo! was a search engine company) show their users. It’salso the easiest way to track your keyword rankings. Like it or not, whether you use a third-party tool or your own, ifyou practice SEO then you’re scrapingsearch results. If you follow a few simple rules, it’s a lot easier than youthink. The problem with scrapingAutomated scraping — grabbing search results using your own‘bot’— violates every search engine’s terms ofservice. Search engines sniff out and block major scrapers. If you ever perform a series of searches that match the behavior ofa SERP crawler, Google and Bing willinterrupt your search with a captcha page. You have to enter thecaptcha or perform whatever test the page requires before performinganother query. That (supposedly) blocks bots and other scripts from automaticallyscraping lots of pages at once. The reason? Resources. A single automated SERP scraper can perform tens, hundreds or eventhousands of queries per second. The only limitations are bandwidthand processing power. Google doesn’t want to waste server cycleson a bunch of sweaty-palmed search geeks’ Python scripts. So,they block almost anything that looks like an automatic query. Your job, if you ever did anything like this,which you wouldn’t, is to buy or createsoftware that does not look like an automatic query. Here area few tricks my friend told me:
Stay on the right side of the equationNote that I said “almost” anything. The search enginesaren’t naive. Google knows every SEO scrapes their results. So does Bing. Both engines have to decide whento block a scraper. Testing shows that equation to make theblock/don’t block decision balances:
At a minimum, any SERP bot must have thepotential of tying up server resources. If it doesn’t, thesearch engine won’t waste the CPU cycles. It’s not worth the effort required to block the bot. So, if you’re scraping the SERPs, you need to stay on theright side of that equation: Be so unobtrusive that, even ifyou’re detected, you’re not worth squashing. DisclaimerUnderstand, now, that everything I talked about in this article istotally hypothetical. I certainly don’t scrapeGoogle. That would violate their terms of service. And I’m sure companies like AuthorityLabs and SERPBuddy haveworked out special agreements for hourly scraping of the major searchengines. But I have this… friend… who’sbeen experimenting a bit, testing what’s allowed andwhat’s not, see… Cough. How I did my testI tested all these theories with three Python scripts. All of them:
Script #1 had no shame. It hit Google as fast as possible anddidn’t attempt to behave like a ‘normal’ webbrowser. Script #2 was a little embarrassed. It pretended to be MozillaFirefox and only queried Google once every 30 seconds. Script #3 was downright bashful. It selected a random user agentfrom a list of 10, and paused between queries for anywhere between 15and 60 seconds. The resultsScript #3 did the best. That’s hardly a surprise. But thedifference is:
There’s no way any of these scripts fooled Google. The searchengine had to know that scripts 1, 2 and 3 were all scrapers.But it only blocked 1 and 2. My theory: Script 3 created so small a burden that it wasn’tworth it for Google to block it. Just as important, though, was thefact that script 3 didn’t make itself obvious.Detectable? Absolutely. I didn’t rotate IP addresses or do anyother serious concealment. But script 3 behaved like it was, well,embarrassed. And a little contrition goes a long way. If youacknowledge you’re scraping onthe square and behave yourself, Google may cut you some slack. The rulesBased on all of this, here are my guidelines for scraping results:
Follow all three of these and you’re a well-behaved scraper.Even if Google and Bing figure out what you’re up to, they leaveyou alone: You’re not a burglar. You’re scouring thegutter for loose change. And they’re OK with that. |
How to: Scrape search engines without pissing them off
最新推荐文章于 2024-04-20 18:32:02 发布