Results 1 to 3 of 3
November 21st, 2007, 12:34 PM #1
What's with Page-Shop.com?
- Join Date
- January 17th, 2005
Over the last two days I have received several thousand searches from a robot heritrix from page-shop.com. I went to their site and found a single page with a sales blurb that sounded like a sales pitch from the Internet Bubble days. Am I the only lucky person? I can ban them but usually in cases like this I don't - even Google started scanning their first site once...
November 21st, 2007, 12:59 PM #2
- Join Date
- January 18th, 2005
page-shop.com is a Japanese language site. It's also registered to a Japanese company.
December 21st, 2007, 10:27 PM #3
Heritrix started hammering one of my sites tonight. Must be about 20 different IPs got blocked by my bad spider zapper.
The only way that happens if they disobey my robots file.
It looks like it had innocent enough beginnings as open source project at SourceForge.
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
By bigsite in forum PopShopsReplies: 2Last Post: June 5th, 2009, 01:29 PM
By writerguy in forum PopShopsReplies: 7Last Post: July 30th, 2007, 07:25 PM