Jump to content

What's with MSN.com?


Guest Gir489

Recommended Posts

  • Replies 7
  • Views 5.7k
  • Created
  • Last Reply
  • Administrator

MSN Web Crawler.

Basically "archiving"/ listing nsanes forum (and site) into its search list.

Link to comment
Share on other sites


Guest Gir489
MSN Web Crawler.

Basically "archiving"/ listing nsanes forum (and site) into its search list.

<{POST_SNAPBACK}>

:mellow:

Link to comment
Share on other sites


Guest Blank Space

Google, Archive.com, Yahoo, and a few others may crawl onto the page.

Although, I hate crawlers. I saw an IP that was DoD on a forum before.

Link to comment
Share on other sites


Guest Gir489
Google, Archive.com, Yahoo, and a few others may crawl onto the page.

Although, I hate crawlers.  I saw an IP that was DoD on a forum before.

<{POST_SNAPBACK}>

Maybe we got fans that work at DoD. :unsure:

Link to comment
Share on other sites


it's cool, bots & guests can't see anything but the news etc. (hash n' stuff are visible for members only) ;)

btw, here's the botlist i'm using (so they might show up at the bottom too)...

4anything.com=4Anything.com
AgentName=LinkoMatic.com Link Checker
ALink=ALink Link Checker
AMeta=AMeta Link Checker
ASPSearch=ASPSearch Link Checker
BeebwareDirectory=Beepware.co.uk Link Checker
Big Brother=Big Brother Link Checker
BigBrother=Big Brother Network Monitor
BlogBot=BlogBot Link Checker
BMChecker=BMLink Checker
Bookmark Buddy=Bookmark Buddy Link Checker
cg-eye=HtmlHelp.com
Check&Get=Check&Get Link Checker
Checkbot=Link Checker
CheckLinks=Link Checker
CheckUrl=NTL.com Link Checker
CheckWeb=CheckWeb Link Checker
CJ Spider=Commision Junction Link Checker
CNET_Snoop=CNET_Snoop Link Checker
ColdFusion=Networld.com Link Checker
CreativeCommons=CreativeCommons.org
CSE HTML=CSE HTML Link Checker
CurryGuide=CurryGuide Link Checker
Custo=Netwu.com Link Checker
DeadLinkCheck=Dead Link Checker
Denmex=Denmax.com Link Checker
DISCo=DISCo Watchman Link Checker
DoctorHTML=DoctorHTML Link Checker
DRKSpider=DRKSpider Link Checker
elsop.com=elsop.com Link Checker
TimelyWeb=TimelyWeb Link Checker
FavOrg=FavOrg Link Checker
Favorites Sweeper=Favorites Sweeper Link Checker
Firstsbot=Firstsbot Link Checker
FreshLinks.exe=FreshLinks Link Checker
Funnel Web=Funnel Web Profiler Link Checker
GeonaBot=Geona Link Checker
Haste=Haste Site Monitoring
Html Link=Html Link Checker
IconSurf=FavIcon Finder
IEFav172Free=Favorites Link Checker
InfoLink=InfoLink Link Checker
InternetLinkAgent=Internet Link Checker
InternetPeriscope=InternetPeriscope Link Checker
InternetSeer=InternetSeer.com Web Monitoring
javElink=javElink Link Checker
JCheckLinks=JCheckLinks Link Checker
jdwhatsnew=jdwhatsnew Link Checker
JRT=Bookmark Checker
Lambda=Lambda Link Checker
Link Valet=Link Valet Online Link Checker
LinkAlarm=LinkAlarm Link Checker
Linkbot=Linkbot Link Checker
LinkLint-checkonly=LinkLint Link Checker
Linkman=Linkman Link Checker
LinkProver=LinkProver Link Checker
Links=Link Checker
LinkScan=LinkScan Link Checker
LinkSweeper=LinkSweeper Link Checker
LinkVerify=LinkVerify Link Checker
LinkWalker=LinkWalker Link Checker
MetaGer=Metager.de Link Checker
Minder=NetMind Link Checker
Morning Paper=Morning Paper Link Checker
MoveAnnouncer=MoveAnnouncer Link Checker
NetLookout=NetLookout Link Checker
NetMechanic=NetMechanic Link Checker
NetMonitor=NetMonitor Link Checker
Netprospector=Netprospector Link Checker
dead-links.com=Dead-Links.com Link Validation
PingALink=Pingalink Site Monitoring
ProWebGuide=Prowebguide Link Checker
Rational=Rational Link Checker
Robozilla=Robozilla Link Checker
RPT=Link Checker
semanticdiscovery=Domain Checker
internetseer.com=Internetseer.com Site Monitoring
SurfMaster=SurfMaster Link Checker
SyncIT=SyncIT Link Checker
targetblaster.com=Targetblaster.com Link Checker
Informant=The Informant Link Checker
Intraformant=The Intraformant Link Checker
VeryGoodSearch.com=VeryGoodSearch.com Link Checker
WebXM=Watchfire WebXM Link Checker
WatzNew=WatzNew Link Checker
Weblink=Weblink Link Checker
WebSite-Watcher=WebSite-Watcher Link Checker
WebTrends=WebTrends Link Checker
WiseWire=Wisewire.com Domain Checker
Xenu=Xenu's Link Sleuth Link Checker
Franklin=Franklin Spam Bot
Industry=Industry Spam Bot
IUPUI=Spam Bot
Lincoln=Spam Bot
MacFinder=MacFinder Spam Bot
Missauga=Missauga Spam Bot
Missouri=Missouri Spam Bot
Program=Spam Bot
rutgers.edu=Rutgers.edu
AbachoBOT=Crawler.de
abcdatos=Abcdatos.com
About=About.com
AcoiRobot=Acoi Picture Robot
Acoon=Accon.de
ActiveWorlds=ActiveWorlds.com
maxbot.com=maxbot.com
ADSARobot=ADSARobot
AESOP_com_SpiderMan=Aesop.com
agadine=Agada.de
Agent-admin=Daum.net
ah-ha.com=Ah-ha.com
Aladin=Aladin.de
Aleksika=Aleksika.de
AlkalineBOT=AlkalineBOT
Allesklar=Allesklar.de
AltaVista=Altavista.co.uk
AmfibiBOT=Amfibi.com
AWeb=AWeb.com
AnswerChase=AnswerChase.com
Antibot=AntiSeach.net
antibot=AntiSeach.net
AnzwersCrawl=Anzwers
A-Online=Jet2Web.com
Aport=Aport.ru
appie=Walhello.com
euroseek.net=Euroseek.net
Arachnoidea=Euroseek.com
Aranha=Girafa.com
ArchitectSpider=Excite.com
archive_org=Archive.org
archive.org_bot=Archive.org
jeeves=Ask Jeeves
ask.24x.info=Ask.24x.info
ASPSeek=ASPSeek.org
asterias=SingingFish.com
Asterias=SingingFish.com
Atomz=Atomz.com
Augurfind=Augurnet.ch
augurnfind=Augurnet.ch
autohttp=Elsop.com
Fetch=Altavista.com
AVSearch=Altavista.com
Baiduspider=Baidu.com
BaiDuSpider=Baidu.com
BanBots=Banbots.com
BarraHomeCrawler=BarraHome.com
bdcindexer=Business.com
BDFetch=Brandimensions.com
BDNcentral=Bdncentral.com
beautybot=Cosmoty.com
Bilbo=Biblo Bot
BlackWidow=BlackWidow Bot
Blaiz-Bee=Blaiz.net
tricus.net=Blitzsuche.rp-online.de
BlogBot=Blogdex.net
blogWatcher=Blogwatcher.pi.titech.ac.jp
boitho.com=Boitho.com
BSDSeek=Inktomi.com
BullsEye=Intelliseek.com
relevare.com=Relevare.com
Buscaplus Robi=Buscaplus.com
CanSeek=Cansee.ca
ChristCRAWLER=Christcrawler.com
CipinetBot=Cipinet.com
citenikbot=Cenik.co.uk
ClariaBot=Searchscout.com
Claymont.com=Claymont.com
Clushbot=Clush.com
CLIPS=Cips-index.imag.fr
Computer_and_Automation=Research Bot
kies.co.kr=Kies.co.kr
Convera=Convera.com
cosmos=Xyleme.com
Combine=Combine Bot
ip3000=IP3000.com
CrawlConvera=Converna.net
alexa.com=Alexa.com
brainbot.com=Brainbot.com
fast.no=Alltheweb.com
Crawler=Crawler.de
CrawlerBoy=Pinpoint.com
CrocCrawler=CrocCrawler.com
bisnisseek.com=Bisnisseek.com
DaAdLe.com=Daadle.com
DataparkSearch=DataparkSearch.com
DaviesBot=Wholeweb.net
dbDig=Prairielandconsulting.com
DBrowse=DBrowse Bot
dCSbot=Openmarket.com
DeepIndex=DeepIndex.com
deepweb=Deepweb.com
Demo Bot=Btopenworld.com
DiaGem=Skyrocket.gr.jp
Diamond=Searchscout.com
Digger=Diggit.com
Digimarc=Digimarc.com
DigOut4U=Arisem.com
DIIbot=Digital-Integrity.com
DittoSpyder=Ditto.com
NaverRobot=Naver.com
dtSearch=Dtsearch.com
Dumbot=Dumbfind.com
dumrobo=Naver.com
EARTHCOM.info=Earthcom.info
EasyDL=Keywen.com
EchO=Voila.com
egothor=Xdefine.com
EgotoBot=Egoto.com
Enterprise=Innerprise.net
Innerprise=Innerprise.net
malfunction.org=Malfunction.org
EroCrawler=Erocrawler.com
ES.NET=Innerprise.net
eseek=Exactseek.com
ESISmartSpider=Travel-Finder.com
ExactSeek=Exactseek.com
exactseek=Exactseek.com
Exalead=Exabot.com
Excalibur=Excalib.com
Mirago=Mirago.co.uk
EyeCatcher=Download-tipp.de
EZResult=Ezresults.com
Fast=Alltheweb.com
FAST=Alltheweb.com
Faxo=FaXo.com
Felix=Mixcat.com
FFC=Frequentfinders.com
fido=Planet Search
FinaleRobot=expressus.com
FindAnISP.com=FindAnISP.com
FineBot=Finesearch.com
Firefly=Fireball.de
FlickBot=DivX.com
Fluffy=SearchHippo.com
Flunky=Metacarta.com
Francis=Neomo.de
FreeFind.com=Freefind.com
Fyber=Fybersearch.com
Gagglebot=Gagglebot
GAIS=Seed.net.tw
Gais=Seed.net.tw
Galaxy=Galaxy.com
Gamma=Gammasite.com
gazz=Infobee.ne.jp
Geckobot=Geckobot.com
geckobot=Geckobot.com
GenCrawler=Gendoor.com
generic=Cmu.edu
GentleSpider=Research.att.com
GeonaBot=Geona.com
GigaBaz=Brainbot.com
GigaBlast=Gigablast.com
Gigabot=Gigablast.com
Giskard=oralco.com
GNODSPIDER=Gnod.net
Goblin=Goguides.org
GoForIt.com=Goforit.com
Goofer=Wanadoo.fr
Google=Google.com
GrigorBot 0.8=Grigor.biz
grub=Grub.com
gsa-crawler=Google.com
Gulliver=Northernlight.com
Gulper=Gulper Bot
Hatena=a.hatena.ne.jp
HeinrichderMiragoRobot=Mirago.de
HenrytheMiragoRobot=Mirago.co.uk
HenryTheMiragoRobot=Mirago.co.uk
Hippias=Hippias Bot
holmes=Morfeo.centrum.cz
HomePageSearch=Hpsearch.uni-trier.de
Homerbot=Homerweb.com
htdig=Computerorgs.com
abcdatos.com=Abcdatos.com
Hubater=Hubat.com
archiver=Archive.org
Archiver=Archive.org
icsbot=Icseoul.org
ideare=Ideare.com
Ideare=Ideare.com
IlTrovatore-Setaccio=Iltrovatore.it
iltrovatore-setaccio=Iltrovatore.it
IncyWincy=Incywincy.com
IndexTheWeb.com=IndexTheWeb.com
Inet library=Inetlibrary.com
searchhippo.com=Searchhippo.com
InfoFly=Versions-project.org
InfoSeek SideWinder=infoseek.com
Inktomi=Yahoo.com
InsumaScout=InsumaScout
Ninja=Dti.ne.jp
InternetSeer=InternetSeer.com
ip3000.com=ip3000.com
ipiumBot=Laurion.com
Iron33=Verno.ueda.info.waseda.ac.jp
Jabot=Ingrid.org
Jack=Domanova.co.uk
gelin=Altavista.co.uk
jeeves=Ask Jeeves
Jetbot/1.0=Jeteye.com
JoocerBot=Joocer.com
Jyxobot/x=Jyxo.cz
Katriona=Katriona
KE_=Voila.fr
Kenjin Spider=Kenjin.ne.jp
Fireball=Fireball.de
Knowledge.com=Knowledge.com
Kototoi=s.u-tokyo.ac.jp
Kulokobot=Kuloko.com
kulturarw=National Library of Sweden
LEIA=Gseek.com
LexiBot=Lexibot.com
LNSpiderguy=Lexis-nexis.com
Look.com=Look.com
lwp-trivial=Search4free.com
Lycos=Lycos.com
lycos=Lycos.com
Mackster=Ukwizz.com
Mag-Net=Wind.it
bea.com=Beasys.com
mammoth=Sli-Systems.com
MantraAgent=Looksmart.com
teradex.com=Mapper.teradex.com
Mariner=Kolibri.de
Martini=Looksmart.com
Marvin=Webseek.de
marvin=Webseek.de
Mata Hari=Lexibot.com
MediaCrawler=Mediacrawler.de
Mediapartners-Google=Google.com
MediaSearch=Www.fi
MegaSheep=SearchUK.com
Mercator=Altavista.com
metabot=metabot
MFC_Tear_Sample=Microsoft.com
MicrosoftPrototypeCrawler=Microsoft.com
Minibot=Minibot
MiracleAlphaTest=Nec.co.jp
MnogoSearch=mNoGoSearch.org
moget=Goo.ne.jp
Mole=Intags.de
Morris=Mixcat.com
Mouse=Spidermonkey.ca
mozDex=Mozdex.com
somewhere.com=Somewhere.com
EZResult=DirectHit.com
euroferret.com=Euroferret.com
muscat.co.uk=Euroferret.com
Links2Go=Links2go.com
fast.no=AllTheWeb.com
TrueRobot=Echo.com
galaxy.com=Galaxy.com
Zealbot=LookSmart.com
grub-client=Grub.org
MP3Bot=mp3bot.de
MSNBOT=MSN.com
msnbot=MSN.com
MultiText=MultiText
MuscatFerret=Webtop.com
NationalDirectory=NationalDirectory.com
Nazilla=Websmostlinked.com
NEC=NEC Research Agent
NetCruiser=NetCruiser
NetResearchServer=Look.com
NetZippy=Netzippy.com
NextopiaBOT=Nextopia.com
NG/=Exabot.com
Noago=Noago.com
NPBot=NameProtect.com
nttdirectory_robot=Navi.ocn.ne.jp
Nutch=Nutch.org
NZBot=NavigationZone.com
ObjectsSearch=ObjectsSearch.com
oBot=Cobion.com
OliverPerry=Claymont.com
OntoSpider=Ontospider.i-n.info
Openbot=Openfind.com.tw
Opencola=Opencola.com
Openfind=Openfind.com
OpenText=Opentext.net
Oracle=Oracle.com
Oxxbot1=Oxxfordinfo.com
Pansophica=Pansophica
ParaSite=Ianett.com
post.sk=Post.sk
peerbot=Peerbot.com
Phoaks=Phoaks
hanmail.net=Daum.net
PICgrabber=PICgrabber
PicoSearch=PicoSearch.com
PictureOfInternet=PictureOfInternet
PintaSpider=PintaSpider
Pita=Pita Bot
Pizilla=He.net
PJspider=Portaljuice.com
polybot=PolyBot
PolyBot=PolyBot
dir.com=Dir.com
iliad.fr=Iliad.fr
Pompos=Free.fr
Popdexter=Popdex.com
PortalBSpider=Portalb.com
XP5=XP5 Bot
AnswerBot=Answerchase.com
psbot=Picsearch.com
Qango.com=Qango.com
QPCreep=Quepasa.com
QuepasaCreep=Quepasa.com
QueryN=Queryn.com
rabaz=Brainbot.com
Rabaz=Brainbot.com
RaBot=Daum.net
ramBot=Intersearch.de
Reaper=Sitesearch.ca
roach.smo.av.com=Altavista.com
RoboCrawl=CanadianContent.net
pj-search.com=PJ-Search.com
SuperSnooper.Com=SuperSnooper.com
xyleme.com=Xyleme.com
expressus.com=Expressus.com
openfind.com.tw=Openfind.com.tw
Robozilla=Dmoz.org
Rotondo=Qualigo.de
ru-robot=Rutgers.edu
Scooter=Altavista.com
scooter=Altavista.com
Scrubby=Scrubtheweb.com
search.at=Search.at
search.ch=Search.ch
search.ch=Search.ch
SearchExpress=SearchExpress.com
Searchit-Now=Searchit-now.co.uk
SearchSpider=Searchspider.com
Searchspider=Searchspider.com
inria.fr=Inria.fr
lookseek.com=Lookseek.com
SeznamBot=Seznam.cz
Shark=Openfind.com
sherlock=Sherlock.com.cn
SightQuestBot=Sightquest.com
SiteSpider=Sitespider.com
SiteXpert=Xtreeme.com
Skampy=Skaffe.com
Slarp=Only.com
Slider=Slider.com
Slurp=Yahoo.com
slurp=Yahoo.com
SlySearch=Turnitin.com
speedfind=Speedfind.de
Speedy_Spider=EntireWeb.com
Spida=Only.com
TraficDublu=Traficdublu.ro
maxbot.com=Maxbot.com
aeneid.com=Yahoo.com
Spider_Monkey=Spidermonkey.ca
spider_monkey=Spidermonkey.ca
SpiderMonkey=Spidermonkey.ca
Sqworm=Aol.com
Steeler=Steeler Bot
Suchbot=Suchbot.de
Suchknecht.at-Robot=Suchknecht.at
navi.ocn.ne.jp=Navi.ocn.ne.jp
canseek.ca=Canseek.ca
Surfnomore=Surfnomore.com
SwissSearch=SwissSearch
SynoBot=Synomia.fr
TaWWWantula=TaWWWantula
hawkholdings.com=Teoma.com
teoma=Teoma.com
TeraCrawl=TeraCrawl
Teradex_Mapper=Mapper.teradex.com
Informant=The Informant
Toutatis=Hoppa.com
TurnitinBot=Turnitin.com
Tutorial Crawler=Tutorgig.com
_Merc=Altavista.com
TygoBot=Tygo.com
UCmore=Ucmore.com
UdmSearch=Mnogosearch.org
UK Searcher Spider=Uksearcher.co.uk
updated=Updated.com
Ultraseek=Infoseek.com
unlostBot=Unlost.com
URLBlaze=Urlblaze.net
UrlScope=UrlScope
Vagabondo=Wise-guys.nl
vspider=Verity.com
webbandit=WebBandit
Webbandit=WebBandit
Webclipping.com=WebClipping.com
webcollage=WebCollage.com
WebCompass=WebCompass
WebCrawler=WebCrawler
WebFindBot=Web-find.com
WebGather=WebGather
WebGenie=WebGenie
intags.de=Intags.de
whatUseek=Whatuseek.com
whatuseek=Whatuseek.com
WhizBang=Whizbanglabs.com
WIRE WebRefiner=Wire.co.uk
wisenutbot=Looksmart.com
WSCbot=WorldSearchCenter.com
inktomisearch.com=Yahoo.com
WebWombat.com.au=Webwombat.com.au
Yandex=Yandex.com
YottaShopping=YottaShopping.com
Yellopet-Spider=YellowPet.com
Zao=Zao
Zippyfinder.com=Zippyfinder.com
ZyBorg=Wisenut.com
atSpider=atSpider Email Extractor
autoemailspider=Auto Email Extractor
Crescent Internet ToolPak=Crescent Email Extractor
Email Extractor=Email Extractor
EmailSiphon=EmailSiphon Extractor
EmailWolf=EmailWolf Extractor
ExtractorPro=Email Extractor
LARBIN-EXPERIMENTAL=Email Collector
MFHttpScan=Email Extractor
Scan4Mail=Scan4Mail Email Extractor

note: this is a VERY cleaned up version of CoolDude's Botlist

:unsure:

Link to comment
Share on other sites


Guest Gir489

I noticed that when I went on my friend's comp.

I was looking for the DooM 3 Resurrection of Evil link, and I couldn't find it.

Then I figured out that I needed to log in. :unsure:

Link to comment
Share on other sites


Archived

This topic is now archived and is closed to further replies.

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...