Server Help :: View topic - Need some help

Server Help

Trash Talk - Need some help

PoLiX - Tue May 26, 2009 12:37 am
Post subject: Need some help

So googlebot is what is crashing SSF right now.

It is for some reason following links to post, but it doesn't know how to post. It seems to load up the server everytime I find it in latest visitors visiting link where it tries to post.

I have added "*" wildcard to robots.txt to block all of them now since just googlebot alone didn't work. I banned the ip on the forums. I then banned the IP through IP Deny in cpanel.

Somehow googlebot is getting around ALL of these measures and still locking everything up.

I'm looking everywhere for how to stop it until IPB can fix this (or I find a fix of my own). Really need some help here guys.

Edit: nvm, seems restarting apache finally knocked it out. But now yahoobot is at it... sob. Isn't there a setting for links to keep bots from going to them?

Mine GO BOOM - Tue May 26, 2009 10:28 am
Post subject:

From http://forums.minegoboom.com/robots.txt:

Code: Show/Hide

User-agent: Googlebot
Disallow: /makecrap.php
Disallow: /makepost.php
Disallow: /ignore.php
Disallow: /search.php
Disallow: /templates
Disallow: /images
Disallow: /download.php
Disallow: /privmsg.php
Disallow: /login.php
Disallow: /memberlist.php
Disallow: /profile.php
Disallow: /admin
Allow: /
Crawl-delay: 20

User-Agent: OmniExplorer_Bot
Disallow: /

User-Agent: Fuck you Omni Explorer Bot, seriously, FUCK YOU
Disallow: /

User-agent: ia_archiver
Disallow: /makepost.php
Disallow: /ignore.php
Disallow: /search.php
Disallow: /privmsg.php
Disallow: /login.php
Disallow: /memberlist.php
Disallow: /profile.php
Disallow: /admin
Allow: /
Crawl-delay: 20

User-agent: *
Disallow: /makecrap.php
Disallow: /makepost.php
Disallow: /ignore.php
Disallow: /search.php
Disallow: /templates
Disallow: /images
Disallow: /download.php
Disallow: /privmsg.php
Disallow: /login.php
Disallow: /memberlist.php
Disallow: /profile.php
Disallow: /admin
Allow: /
Crawl-delay: 20

Maverick - Tue May 26, 2009 11:15 am
Post subject:

makecrap.php ?

Sonic VI - Thu May 28, 2009 5:09 pm
Post subject:

From http://forums.minegoboom.com/robots.txt:

Code: Show/Hide

Disallow: /makepost.php
Disallow: /ignore.php
Disallow: /search.php
Disallow: /templates
Disallow: /images
Disallow: /download.php
Disallow: /privmsg.php
Disallow: /login.php
Disallow: /memberlist.php
Disallow: /profile.php
Disallow: /admin
Allow: /
Crawl-delay: 20

User-Agent: OmniExplorer_Bot
Disallow: /

User-Agent: Fuck you Omni Explorer Bot, seriously, FUCK YOU
Disallow: /

User-agent: ia_archiver
Disallow: /makepost.php
Disallow: /ignore.php
Disallow: /search.php
Disallow: /privmsg.php
Disallow: /login.php
Disallow: /memberlist.php
Disallow: /profile.php
Disallow: /admin
Allow: /
Crawl-delay: 20

User-agent: *
Disallow: /makecrap.php
Disallow: /makepost.php
Disallow: /ignore.php
Disallow: /search.php
Disallow: /templates
Disallow: /images
Disallow: /download.php
Disallow: /privmsg.php
Disallow: /login.php
Disallow: /memberlist.php
Disallow: /profile.php
Disallow: /admin
Allow: /
Crawl-delay: 20

[/code]

Fixed

Doc Flabby - Thu May 28, 2009 5:47 pm
Post subject:

Polix, did you see my post about google webmaster tools?

Mine GO BOOM - Thu May 28, 2009 6:14 pm
Post subject:

Maverick wrote:

makecrap.php ? :)

Take a look at the cache of viewtopic.php on google's page. Notice the link to make new topics.

Sonic VI wrote:

Fixed :-D

... was there ever a time when "fixed!" posts were ever any good?

PoLiX - Thu May 28, 2009 10:21 pm
Post subject:

doc yes, already had done that long ago. Have SSCentral and SSF on there.

They we're just ignoring my robots.txt, but the IP bans seem to have stopped them after an apache restart.

All seems good for now while me and IPB work on figuring out why the bots do that.