Bad bot block list

Welcome To

This topic is empty.

Viewing 15 posts - 1 through 15 (of 38 total)

1 2 3 →

Author

Posts
May 25, 2011 at 9:33 pm

#698077
[deleted]

@[deleted]

Guest
I’ve decided to start limiting bot access to the site so I’ve disallowed them all baring facebook, yahoo, bing, msn and google in robots. txt, I’ve started blocking the IPs of bots who don’t respect robots.txt and expanded Who’s Online with better spider reporting. So if you’re ever in Who’s Online and see a bot you think should be banned please paste their details below such as:

Spinn3r Spider 22:29 Viewing Page Home 174-36-241-156.robot.spinn3r.com
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spin

Thanks!
May 25, 2011 at 10:10 pm

#948890
[deleted]

@[deleted]

Guest
Magpie Spider 23:09 /forums/calendar.php?do=getinfo&e=2899 Viewing Event
illumiNaughty ” The Return” DINO PSARES.SABERTOOTH.ZEUS.DRAVEN.NOSF.MAZIEG
94.228.34.233
magpie-crawler/1.1 (U; Linux amd64; en-GB; +Social Media Monitoring Tools | Brandwatch)
May 26, 2011 at 9:04 am

#948912
1

sinner69

@sinner69

Member
I’m not sure of what you mean by not respecting robots…Is it when the operate outside their normal ip range?
May 26, 2011 at 10:04 am

#948895
6

General Lighting

@general-lighting

Moderator
Staff
there’s a file robots.txt in the server (I’ve shown it below).

it tells robots where they are allowed on the site – and where they are not – if they disobey it they are booted.

i.e these are the droids we are looking for – and others can JTFO

Code:

User-agent: Googlebot-Image
Allow: /

User-agent: Googlebot-Mobile
Allow: /

User-agent: Googlebot-Mobile
Allow: /

User-agent: Yahoo! Slurp
Allow: /

user-Agent: Yahoo-MMCrawler
Allow: /

user-Agent: YahooSeeker
Allow: /

user-Agent: msnbot-media
Allow: /

user-Agent: facebookexternalhit
Allow: /

User-agent: Googlebot
Disallow: /forums/index.php
Disallow: /forums/members/list/
Disallow: /forums/admincp/
Disallow: /forums/clientscript/
Disallow: /forums/cpstyles/
Disallow: /forums/images/
Disallow: /forums/modcp/
Disallow: /forums/ajax.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/global.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/showpost.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php
Disallow: /forums/faq.php

User-agent: Bingbot
Disallow: /forums/index.php
Disallow: /forums/members/list/
Disallow: /forums/admincp/
Disallow: /forums/clientscript/
Disallow: /forums/cpstyles/
Disallow: /forums/images/
Disallow: /forums/modcp/
Disallow: /forums/ajax.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/global.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/showpost.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php
Disallow: /forums/faq.php

User-agent: *
Disallow: /
May 27, 2011 at 5:43 am

#948905
6

Angel

@angel

Moderator
Staff
Isn’t it possible in the Admin Panel to just allow the ones you want, or do the unwanted ones get in anyway?
May 27, 2011 at 5:51 am

#948906
6

Angel

@angel

Moderator
Staff
Like Oddvar I’m not totally sure what you mean :crazy_diz

This one ???

Baidu Spider

07:41 showthread.php?t=35973&pagenumber= Viewing Thread

Nimai Le Santo (RIP): An Overdose and his Father’s Reaction

123.125.71.106

Mozilla/5.0 (compatible; Baiduspider/2.0; +???????????Baiduspider)
May 27, 2011 at 6:11 am

#948913
1

sinner69

@sinner69

Member
some bot is is also here as guests, last night I saw one looking in thrash for penis pills….it was stopped though. Infact there are many bots in here, I think they follow when people are asking bing, yahoo, google etc….
May 27, 2011 at 10:29 am

#948907
6

Angel

@angel

Moderator
Staff
Whitevector Crawler Spider
12:08

forumdisplay.php?f=29&pagenumber= Viewing Forum The Law

217.149.51.37

Whitevector Crawler (+Whitevector)
May 27, 2011 at 11:00 pm

#948891
[deleted]

@[deleted]

Guest
In Who’s Online use display only Bots, sort by Last Activity Date and select Show User Agent… 🙂
May 29, 2011 at 8:31 pm

#948896
6

General Lighting

@general-lighting

Moderator
Staff
not quite a bad bot but it made me laugh…

MSNBot Spider 21:30 showthread.php?t=28033&pagenumber= Viewing Thread
Microsoft wants to record your life… Check this out!!!
207.46.192.64
May 30, 2011 at 8:14 am

#948914
1

sinner69

@sinner69

Member
What is this?

Guest 10:10
Viewing Who’s Online
173.193.228.66
libwww-perl/5.837
May 30, 2011 at 9:27 am

#948915
1

sinner69

@sinner69

Member
And how to deal with this? ;

GeoTime – Geo-Temporal Information Visualization

Metropolitan Police trials GeoTime tracking software | Security Management | ZDNet UK

Police Buys GeoTime Software, Will Track Suspects Online | ITProPortal.com

Is there a way to sort this out?
May 30, 2011 at 11:16 am

#948897
6

General Lighting

@general-lighting

Moderator
Staff
@!sinner69! 437330 wrote:

What is this?

Guest 10:10
Viewing Who’s Online
173.193.228.66
libwww-perl/5.837

thats one of our external IP’s for the main site. I think its our own bot, and part of the software to make the site work..
May 30, 2011 at 11:19 am

#948898
6

General Lighting

@general-lighting

Moderator
Staff
@!sinner69! 437333 wrote:

And how to deal with this? ;

GeoTime – Geo-Temporal Information Visualization

Metropolitan Police trials GeoTime tracking software | Security Management | ZDNet UK

Police Buys GeoTime Software, Will Track Suspects Online | ITProPortal.com

Is there a way to sort this out?

this is up to the users to be careful with their devices. That said on forums there is little correct geographical info in a post (IP addresses can be hundreds of miles off) unless people are silly enough to repeatedly brag about crime.

in fact what gets people nicked from forum posts is not feds and smart technology but folk what lurk on forums and grass up stuff and we have been good at warning the users not to incriminate themselves.
May 31, 2011 at 7:24 pm

#948916
1

sinner69

@sinner69

Member
Whats this?;
Google Wireless Transcoder Spider
21:15
Viewing Archives
egg tek was amazing
209.85.228.81
Mozilla/5.0 (en-us) AppleWebKit/534.14 (KHTML, like Gecko; Google Wireless Transcoder) Chrome/9.0.59
Author

Posts

Viewing 15 posts - 1 through 15 (of 38 total)

1 2 3 →

You must be logged in to reply to this topic.