Results 1 to 25 of 25
  1. #1
    Member
    Join Date
    January 18th, 2005
    Posts
    155
    Twiceler bot using 4 GIGS of my Bandwidth?
    I did contact my host and they see nothing wrong in my logs, but this site gets about 10 hits a week if that, from GoldenCan pages - I know that from using Extreme Tracking non-public tracker.

    Would a hacker be in my site or anyone know what could be happening here?

    Any insite or idea would really be appreciated!

    Karin

    (someone here recommended Hostgator .. is that Apache? If a dumb question it's becuz I'm dum)

  2. #2
    ABW Ambassador meadowmufn's Avatar
    Join Date
    January 18th, 2005
    Location
    Seattle
    Posts
    2,587
    Quote Originally Posted by KariBon
    I did contact my host and they see nothing wrong in my logs, but this site gets about 10 hits a week if that, from GoldenCan pages - I know that from using Extreme Tracking non-public tracker.

    Would a hacker be in my site or anyone know what could be happening here?

    Any insite or idea would really be appreciated!

    Karin

    (someone here recommended Hostgator .. is that Apache? If a dumb question it's becuz I'm dum)
    Are you using CPanel? I know they had an issue with inaccurate bandwidth calculation. Contact your host and ask for their calculation of bandwidth used on your account.
    -Don't criticize anyone til you've walked a mile in their shoes. Then when you do criticize them, you'll be a mile away and have their shoes.
    - Silence is golden. Duct Tape is silver.

  3. #3
    Member
    Join Date
    January 18th, 2005
    Posts
    155
    I use cPanel but just to check logs, etc. Otherwise I use CuteFTP. I did contact host and this was the reply I got:
    >Our system use our logs, so you can be sure about your Bandwich.I have checked it and as I see our log is not hacked, so your Bandwich is correct.<

    This make sense to You?

  4. #4
    ABW Ambassador meadowmufn's Avatar
    Join Date
    January 18th, 2005
    Location
    Seattle
    Posts
    2,587
    Quote Originally Posted by KariBon
    I use cPanel but just to check logs, etc. Otherwise I use CuteFTP. I did contact host and this was the reply I got:
    >Our system use our logs, so you can be sure about your Bandwich.I have checked it and as I see our log is not hacked, so your Bandwich is correct.<

    This make sense to You?
    That's interesting. My host calculates the bandwidth separately and said there was a big difference between what they were seeing and what cpanel was reporting. That's when they mentioned the cpanel bandwidth bug. Google "cpanel bandwidth bug" and you'll see what I'm talking about. You should probably point your host's tech support team to some of those search results.
    -Don't criticize anyone til you've walked a mile in their shoes. Then when you do criticize them, you'll be a mile away and have their shoes.
    - Silence is golden. Duct Tape is silver.

  5. #5
    Member
    Join Date
    January 18th, 2005
    Posts
    155
    Thanks MeadowMuffin .. will do that right away.

    Can you (or anyone) recommend a host that handles ALL scripts like GoldenCan, PopShops and the eBay "EasyNicheStore" without changing their PHP rules all the time, caching your directories, and on and on! When I wasn't looking for a new host I'd see recommendations all the time and now can't find any.
    I don't care if it's an affiliate recommendation - just a host that's kind to dumbells!

  6. #6
    2005 Linkshare Golden Link Award Winner  ecomcity's Avatar
    Join Date
    January 18th, 2005
    Location
    St Clair Shores MI.
    Posts
    17,328
    FastNext.com might have their ABW special still going. Nothing wrong with clicking your own GoDaddy or Dotster creative ( on CJ) and using their low cost hosting services. I about flipped when I saw my monthly EPC at CJ over 1450.00 so I know those 2 report.... LOL
    Webmaster's... Mike and Charlie

    "What have you done today to put real value into a referral click...from a shoppers viewpoint!"

  7. #7
    general fuq mrbshouse's Avatar
    Join Date
    January 18th, 2005
    Location
    Argieville
    Posts
    1,381
    i've seen bandwidth go out the window when i had a cron running wild or a loop in a script somewhere. Check to see if google or the other bots are hitting a bunch of pages.

    do you also have access to WHM...there are some good bandwidth reports there too.

  8. #8
    Full Member
    Join Date
    January 18th, 2005
    Posts
    396
    My bandwidth used goes through the roof when I get hit with the scraping bots from Europe, they can 40-100 hits/minute @ 10K/ - when they decide to go after me it goes for several hours - I had to incorporate a 'trap' to block them at the start of my index.php - blocking them at .htaccess ... won't work because they change IP every couple of minutes

  9. #9
    OPM and Moderator Chuck Hamrick's Avatar
    Join Date
    April 5th, 2005
    Location
    Park City Utah
    Posts
    16,646
    Back in the day we had unprotected cgi forms that were used by spammers as an email gateway. micheck is right that you need to check to see if bots are causing it.

  10. #10
    Member
    Join Date
    January 18th, 2005
    Posts
    155
    In cPanel in Latest Hits, this morning I'm using over 3GB and a directory using GoldenCan with thousands and thousands of little links of Overstock items have this ..
    http://www.cuill.com/twiceler/robot.html
    When I followed the above link it says on the page

    >Webmaster Information
    Twiceler is an experimental robot. The user-agent is “twiceler.” It could take 24-48 hours for us to re-read your robots.txt file. If you need something blocked immediately, please let us know. <

    So this is what's causing it, right? Would you tell them to stop (before they get to my Walmart directory! oh my! ) or keep an eye on your bandwidth and let it go?

    Thanks for all the replies in this thread BTW!

    Karin

  11. #11
    Moderator BurgerBoy's Avatar
    Join Date
    January 18th, 2005
    Location
    jacked by sylon www.sylonddos.weebly.com
    Posts
    9,618
    Wink
    Quote Originally Posted by KariBon
    In cPanel in Latest Hits, this morning I'm using over 3GB and a directory using GoldenCan with thousands and thousands of little links of Overstock items have this ..
    http://www.cuill.com/twiceler/robot.html
    When I followed the above link it says on the page

    >Webmaster Information
    Twiceler is an experimental robot. The user-agent is “twiceler.” It could take 24-48 hours for us to re-read your robots.txt file. If you need something blocked immediately, please let us know. <

    So this is what's causing it, right? Would you tell them to stop (before they get to my Walmart directory! oh my! ) or keep an eye on your bandwidth and let it go?

    Thanks for all the replies in this thread BTW!

    Karin
    Twiceler is a bad bot. It WILL NOT obey your robots.txt file.


    Put the following in your .htacess file on your server.

    Code:
    SetEnvIfNoCase User-Agent .*Twiceler.* bad_bot
    
    order allow,deny
    deny from env=bad_bot
    allow from all
    It will block the bot and send it a 403.

    You can block any bot you want to by just adding it to your .htacess file by adding a new line for it.

    Example:

    Code:
    SetEnvIfNoCase User-Agent .*Exabot.* bad_bot
    You just add the
    Code:
    order allow,deny
    deny from env=bad_bot
    allow from all
    once.

    After that add a new line for each bot that you want blocked.

    Vietnam Veteran 1966-1970 USASA
    ABW Forum Rules - Advertise At ABW

  12. #12
    ABW Founder Haiko de Poel, Jr.'s Avatar
    Join Date
    January 18th, 2005
    Location
    New York
    Posts
    21,609
    Admin Note: Moved and renamed thread
    Continued Success,

    Haiko
    The secret of success is constancy of purpose ~ Disraeli

  13. #13
    Advocate mellie's Avatar
    Join Date
    January 18th, 2005
    Location
    Here
    Posts
    1,925
    Thanks Burgerboy, I was just going to ask how to block them.
    Melanie
    President - Affiliate Advocacy 2008 ShareaSale Performance Industry Advocate Award, 2009 Affiliate Summit Pinnacle Award - Affiliate Advocate
    Affiliate Advocacy
    NYAffiliateVoice Seery Writing

  14. #14
    Member
    Join Date
    January 18th, 2005
    Posts
    155
    Thank you, Burger Boy! Did it. I have some other sites on the same server so will do that on the others . Shucks, I thought it might be a new search engine. Many many years ago when Google started up I'd see Google in my hits and wonder "What's a Google".

    Karin

  15. #15
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Quote Originally Posted by BurgerBoy
    Twiceler is a bad bot. It WILL NOT obey your robots.txt file.
    Are you sure it doesn't obey robots.txt? I've seen this engine getting press lately and they seem to be on the up and up. I don't believe they'd intentionally ignore robots.txt.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  16. #16
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Nunya, Business
    Posts
    23,684
    There are other threads saying the same - http://www.google.com/search?sourcei...30&q=Twiceler+

    I just checked the first three, theadminzone, digital point, phpbb forums, all the same. Sucking up a lot of bandwidth, not obeying the robots.txt file. Looking over the others, not good.

  17. #17
    Moderator BurgerBoy's Avatar
    Join Date
    January 18th, 2005
    Location
    jacked by sylon www.sylonddos.weebly.com
    Posts
    9,618
    Quote Originally Posted by Snib
    Are you sure it doesn't obey robots.txt? I've seen this engine getting press lately and they seem to be on the up and up. I don't believe they'd intentionally ignore robots.txt.

    - Scott
    Yes - I'm sure.

    I banned it in my robots.txt and it was still hitting my sites 2 months later and it never requested my robots.txt that I could find in those 2 months.

    Vietnam Veteran 1966-1970 USASA
    ABW Forum Rules - Advertise At ABW

  18. #18
    ABW Ambassador writerguy's Avatar
    Join Date
    January 17th, 2005
    Location
    Springfield, Missouri, USA
    Posts
    3,248
    Quote Originally Posted by BurgerBoy
    Twiceler is a bad bot. It WILL NOT obey your robots.txt file.


    Put the following in your .htacess file on your server.

    Code:
    SetEnvIfNoCase User-Agent .*Twiceler.* bad_bot
    
    order allow,deny
    deny from env=bad_bot
    allow from all
    It will block the bot and send it a 403.

    You can block any bot you want to by just adding it to your .htacess file by adding a new line for it.

    Example:

    Code:
    SetEnvIfNoCase User-Agent .*Exabot.* bad_bot
    You just add the
    Code:
    order allow,deny
    deny from env=bad_bot
    allow from all
    once.

    After that add a new line for each bot that you want blocked.
    Thanks so much for that, BurgerBoy. One question, though, for those of us who are pretty ".htaccess impaired" about these things:

    Does it matter WHERE in the order of lines in .htaccess I put this code?

    I'm never sure when adding stuff to .htaccess just where I should put it. Most of my sites are WordPress installs, and I know WordPress puts several lines of code in the .htaccess file -- do you know where these commands would go in relation to the WP lines?

    Also -- my webhost had me put this in the .htaccess file:

    php_value register_globals 1

    Any ideas about how THAT line fits into the mix?

    Did I say "one question"? Hmmm ... need to work on my basic counting skills, I guess.

    Thanks.
    Generate more fake news.

  19. #19
    Moderator BurgerBoy's Avatar
    Join Date
    January 18th, 2005
    Location
    jacked by sylon www.sylonddos.weebly.com
    Posts
    9,618
    Anytime I add to the .htaccess file I just add it at the bottom of what is already there and then save it.

    I never change anything that is already there.

    Never had any problems doing that way so far.

    Here's what I banning right now

    Code:
    SetEnvIfNoCase User-Agent .*Twiceler.* bad_bot
    SetEnvIfNoCase User-Agent .*NaverBot/1.0.* bad_bot
    SetEnvIfNoCase User-Agent .*heritrix/1.12.1.* bad_bot
    SetEnvIfNoCase User-Agent .*panscient.com.* bad_bot
    SetEnvIfNoCase User-Agent .*GurujiBot/1.0.* bad_bot
    SetEnvIfNoCase User-Agent .*Exabot-Thumbnails.* bad_bot
    SetEnvIfNoCase User-Agent .*libwww-perl.* bad_bot
    SetEnvIfNoCase User-Agent .*Student study spider#007.* bad_bot
    SetEnvIfNoCase User-Agent .*Exabot.* bad_bot
    SetEnvIfNoCase User-Agent .*Java/1.6.0.* bad_bot
    SetEnvIfNoCase User-Agent .*MJ12bot.* bad_bot
    SetEnvIfNoCase User-Agent .*Zeus.* bad_bot
    SetEnvIfNoCase User-Agent .*YodaoBot/1.0.* bad_bot
    SetEnvIfNoCase User-Agent .*ia_archiver.* bad_bot
    
    order allow,deny
    deny from env=bad_bot
    allow from all
    As I find new bad bots on my sites I just add a new line banning them also.

    Vietnam Veteran 1966-1970 USASA
    ABW Forum Rules - Advertise At ABW

  20. #20
    Moderator BurgerBoy's Avatar
    Join Date
    January 18th, 2005
    Location
    jacked by sylon www.sylonddos.weebly.com
    Posts
    9,618
    Wow!

    Five more post and I'll have 3,000 post.

    Vietnam Veteran 1966-1970 USASA
    ABW Forum Rules - Advertise At ABW

  21. #21
    ABW Ambassador writerguy's Avatar
    Join Date
    January 17th, 2005
    Location
    Springfield, Missouri, USA
    Posts
    3,248
    Quote Originally Posted by BurgerBoy
    Anytime I add to the .htaccess file I just add it at the bottom of what is already there and then save it.

    I never change anything that is already there.

    Never had any problems doing that way so far.

    Here's what I banning right now

    Code:
    SetEnvIfNoCase User-Agent .*Twiceler.* bad_bot
    SetEnvIfNoCase User-Agent .*NaverBot/1.0.* bad_bot
    SetEnvIfNoCase User-Agent .*heritrix/1.12.1.* bad_bot
    SetEnvIfNoCase User-Agent .*panscient.com.* bad_bot
    SetEnvIfNoCase User-Agent .*GurujiBot/1.0.* bad_bot
    SetEnvIfNoCase User-Agent .*Exabot-Thumbnails.* bad_bot
    SetEnvIfNoCase User-Agent .*libwww-perl.* bad_bot
    SetEnvIfNoCase User-Agent .*Student study spider#007.* bad_bot
    SetEnvIfNoCase User-Agent .*Exabot.* bad_bot
    SetEnvIfNoCase User-Agent .*Java/1.6.0.* bad_bot
    SetEnvIfNoCase User-Agent .*MJ12bot.* bad_bot
    SetEnvIfNoCase User-Agent .*Zeus.* bad_bot
    SetEnvIfNoCase User-Agent .*YodaoBot/1.0.* bad_bot
    SetEnvIfNoCase User-Agent .*ia_archiver.* bad_bot
    
    order allow,deny
    deny from env=bad_bot
    allow from all
    As I find new bad bots on my sites I just add a new line banning them also.
    Thank you, thank you, thank you, BurgerBoy. Very helpful. I'll give it a shot. I have a couple of sites that are getting a lot of bot activity, now I'll give 'em a closer look and see what I want to do. Great!
    Generate more fake news.

  22. #22
    Member
    Join Date
    January 18th, 2005
    Posts
    155
    Unhappy
    I did put your instructions in the .htaccess file, Burgerboy, but Twiceler is still grinding away through all those GoldenCan datafeed links this morning.

    I wrote to Jim per the message on their website - so far my bandwidth this morning has used 76% - and asked him to make it stop and keep the bot from moving on to the other sites on this server - one of them has PopShops along With GoldenCan.

    Yesterday was probably too late to stop it .. locking the barn afer the horse. Probly today will keep it from starting fresh on other site.

    Karin

  23. #23
    ABW Ambassador writerguy's Avatar
    Join Date
    January 17th, 2005
    Location
    Springfield, Missouri, USA
    Posts
    3,248
    Just found this evening that Twiceler has been poking around one of my sites. I put the code BurgerBoy suggested in the end of my .htaccess -- and Twiceler has been there several times anyway??
    Generate more fake news.

  24. #24
    Member
    Join Date
    January 18th, 2005
    Posts
    155
    WriterGuy - It took a day to get it off my site - and I did also write to Jim of Twiceler per that websites instructions.
    http://www.cuill.com/twiceler/robot.html
    >Webmaster Information
    Twiceler is an experimental robot. The user-agent is “twiceler.” It could take 24-48 hours for us to re-read your robots.txt file. If you need something blocked immediately, please let us know. <

    In the meantime, If you didn't see the info about another thing that was attacking my site it's at
    http://forum.abestweb.com/showpost.p...4&postcount=16

    It would be nice to know since you've had Twiceler on if a host IP number 84.32.87.250 is in your logs too.

    Thanks!
    Karin

  25. #25
    Moderator BurgerBoy's Avatar
    Join Date
    January 18th, 2005
    Location
    jacked by sylon www.sylonddos.weebly.com
    Posts
    9,618
    Add this at the top of your .htaccess file
    Code:
    <Limit GET POST>
    #The next line modified by DenyIP
    order allow,deny
    #The next line modified by DenyIP
    #deny from all
    allow from all
    </Limit>
    <Limit PUT DELETE>
    order deny,allow
    deny from all
    </Limit>
    Change what you added earlier to

    Code:
    <Files 403.shtml>
    order allow,deny
    allow from all
    </Files>
    
    SetEnvIfNoCase User-Agent .*Twiceler.* bad_bot
    SetEnvIfNoCase User-Agent .*NaverBot/1.0.* bad_bot
    SetEnvIfNoCase User-Agent .*heritrix/1.12.1.* bad_bot
    SetEnvIfNoCase User-Agent .*panscient.com.* bad_bot
    SetEnvIfNoCase User-Agent .*GurujiBot/1.0.* bad_bot
    SetEnvIfNoCase User-Agent .*Exabot-Thumbnails.* bad_bot
    SetEnvIfNoCase User-Agent .*libwww-perl.* bad_bot
    SetEnvIfNoCase User-Agent .*Student study spider#007.* bad_bot
    SetEnvIfNoCase User-Agent .*Exabot.* bad_bot
    SetEnvIfNoCase User-Agent .*Java/1.6.0.* bad_bot
    SetEnvIfNoCase User-Agent .*MJ12bot.* bad_bot
    SetEnvIfNoCase User-Agent .*Zeus.* bad_bot
    SetEnvIfNoCase User-Agent .*YodaoBot/1.0.* bad_bot
    SetEnvIfNoCase User-Agent .*ia_archiver.* bad_bot
    SetEnvIfNoCase User-Agent .*BPImageWalker/2.0.* bad_bot
    SetEnvIfNoCase User-Agent .*shelob v1.0.* bad_bot
    SetEnvIfNoCase User-Agent .*LinkWalker/2.0.* bad_bot
    
    order allow,deny
    deny from env=bad_bot
    allow from all
    You noticed that I added
    Code:
    <Files 403.shtml>
    order allow,deny
    allow from all
    </Files>
    to it.

    Vietnam Veteran 1966-1970 USASA
    ABW Forum Rules - Advertise At ABW

  26. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Twiceler bot using 4 GIGS of my Bandwidth?
    By KariBon in forum Programming / Datafeeds / Tools
    Replies: 10
    Last Post: May 12th, 2008, 09:28 AM
  2. Twiceler - cuill.com --- ring any bells
    By micheck in forum Midnight Cafe'
    Replies: 9
    Last Post: August 15th, 2007, 02:56 PM
  3. Where is my bandwidth going???
    By knoxb3 in forum Newbie Affiliate FAQs & Helpful Articles
    Replies: 3
    Last Post: July 19th, 2005, 11:56 AM
  4. Bandwidth
    By frank3iii in forum Domains & Hosting
    Replies: 7
    Last Post: June 13th, 2003, 05:35 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •