Results 1 to 13 of 13
  1. #1
    ABW Ambassador mousejockey's Avatar
    Join Date
    January 18th, 2005
    Posts
    2,494
    How could one of my domains have a robot txt file without me putting it there?
    Does anyone know?

  2. #2
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,402
    Never heard of that happening....are you sure you didn't put it there a while back or something...never heard of a file appearing (except in hacks, but if you were hacked, I don't think they would to the robots.txt file for you)...

    Tom Pyles

  3. #3
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    ÄúsTrálíĺ
    Posts
    1,372
    Maybe it is a feature that the host puts there by default to protect it's non-tech-savvy customers.
    ie, to auto disallow cgi-bin etc.

  4. #4
    ABW Ambassador mousejockey's Avatar
    Join Date
    January 18th, 2005
    Posts
    2,494
    It's a domain I picked up last year and I noticed there are a heap of links to it, all strangly have nothing to do with the domain name, and are pretty strange links too.

    A while ago I messed around and put up some pages on the domain, I had a look at the source code and noticed the robot meta's were different, no index?
    Thought this was strange so went back and checked my copy and it wasn't there, then checked the site again and it was gone, so I thought I was just seeing things!

    Earlier today I checked it with the archive tool which said it had a robots txt file?
    No, it's not the host.
    I put up a roughy to get it listed in the engines and I'm having a devil of a time getting it ranked.
    It's a heavy populated keyword domain, but that's never stopped me from getting a site up on the first page.
    Still scratching my head [img]/infopop/emoticons/icon_smile.gif[/img]

  5. #5
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,402
    Mouse,
    That is a strange one. The only reason I brought up that perhaps you put one up previously is I've seen some of our customers create something (such as a txt file) and forget they put it up....or some editors like to add things to their sites (FP seems to be a problem with this)...Now that you have explained it, that is pretty weird and I wish I could provide an answer for you. I will send it off to our techs and see if anybody has seen something like you describe.

    Tom Pyles

  6. #6
    ABW Founder Haiko de Poel, Jr.'s Avatar
    Join Date
    January 18th, 2005
    Location
    New York
    Posts
    21,609
    MJ,

    I see this as a sheer and utter impossibility!

    The link remnants would be to the old [previously] cached page links, and nothing else .

    Haiko
    PS. My hypothesis :cool:

  7. #7
    ABW Ambassador mousejockey's Avatar
    Join Date
    January 18th, 2005
    Posts
    2,494
    Thanks everyone [img]/infopop/emoticons/icon_smile.gif[/img]
    It's a strange one, I did check with my host and no robot txt there, I didn't think I'd shoved one in.

    You know when you have one of those moments and you can't believe your eyes, but you know you've seen it, well it was one of those.
    I did try to convince myself I had been hallucinating, but there was always that nagging doubt.
    Then when archive.org came back with robots.txt got me thinking again.

    Well guess I'll go back to work and do something constructive besides playing slueth [img]/infopop/emoticons/icon_smile.gif[/img]

  8. #8
    Resident Genius and Staunch Capitalist Leader's Avatar
    Join Date
    January 18th, 2005
    Location
    Florida
    Posts
    12,817
    MJ,

    I'm sure you were NOT hallucinating! Everything's possible!!!

    Also, your problems getting the site indexed would indicate that something is still fishy.

    What did the robots.txt tell the robots to do? If it's telling them to stay away from the entire site, I would indeed suspect hacking. Perhaps the old site had collected some enemies!

    If your password isn't weird enough, it could also have been plain old Guessed (as opposed to any sophisticated hacking).

    With the "weird links" you say were going into the site from before you owned it, I'd say it's entirely possible that some yo-yo was trying to "fix" the old site, not realizing that it was under new ownership!

    The first thing I'd say is, change your password to something really obscure! And check your log files if you can, if you see anything else fishy showing up!

  9. #9
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,402
    Mouse,
    After talking to one of my techs, he too had suggested a possible hack as well. I didn't even think to ask you what was in the file earlier, but as leader points out, it sounds fishy...if you still have access to the file, take a look and see what's in there.

    Tom Pyles

  10. #10
    ABW Ambassador mousejockey's Avatar
    Join Date
    January 18th, 2005
    Posts
    2,494
    Thanks Tom, I'll see if I can and let you know [img]/infopop/emoticons/icon_smile.gif[/img]

    Leader, I'll change my password too, I'm sooo bad with passwords, scribble them down then forget so I tend to keep the same ones.
    Just thought no-one would want to hack into one of my sites, but this domain has a heap of links to some serbian movement and lots of weightlifting stuff, the domain has nothing to do with weights, health of political movements.
    Quiet possible someone had something going with them.
    Thank you everyone for your help and suggestions [img]/infopop/emoticons/icon_smile.gif[/img]

  11. #11
    Resident Genius and Staunch Capitalist Leader's Avatar
    Join Date
    January 18th, 2005
    Location
    Florida
    Posts
    12,817
    Say, maybe if you used the Wayback Machine you could find out more about what the domain used to have on it!

  12. #12
    ABW Ambassador mousejockey's Avatar
    Join Date
    January 18th, 2005
    Posts
    2,494
    Nothing on the wayback machine, said robot.txt file..lol! going round in circles [img]/infopop/emoticons/icon_smile.gif[/img]

    Alexa gave me all the strange links to serb political movement and weightlifters.

    [ 01-14-2002: Message edited by: mousejockey ]

  13. #13
    Newbie
    Join Date
    January 18th, 2005
    Posts
    37
    I was using a host about a year ago and had the same problem. After a little investigating it turned out that my host was building porn link farms using our domain name and submitting it to tons of search engines. They were using the robots txt and htaccess to redirect to directories I was unable to view and had no idea even existed.

  14. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Google Robot Searching for robot.txt
    By ahmar in forum Search Engine Optimization
    Replies: 4
    Last Post: December 26th, 2004, 01:26 PM
  2. Robot.txt example...
    By eggerda in forum Search Engine Optimization
    Replies: 7
    Last Post: September 18th, 2003, 11:56 AM
  3. Robot.txt versus Amazon.PL
    By beggers in forum Cusimano.com Scripts
    Replies: 12
    Last Post: March 18th, 2003, 04:16 PM
  4. A Lesson About Robot.txt
    By seaslug44 in forum Search Engine Optimization
    Replies: 1
    Last Post: February 22nd, 2002, 07:49 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •