Robots.txt Validator

Note: Please use the FULL http://www.mydomain.com/robots.txt url to your robots.txt file. This allows you to use alternate filenames in order to test development copies of your robots.txt file before making it live on the web and risking a robot running into it when it is invalid.

Robots.txt Validator

Още за четене :

Search Engines Robots ListSearch Engines Robots List Home page/search engine Robot identifier IP address(es) www.abacho.com AbachoBOT srv-ze-robot1.tricus.com www.aesop.com AESOP_com_SpiderMan 209.189.115.49 www.ah-ha.com ah-ha.com crawler (crawler@ah-ha.com) c7pub-216-250-141-186.center7.com www.alexa.com ia_archiver green.alexa.com sarah.alexa.com www.altavista.com Scooter test-scooter.pa.alta-vista.net brillo.pa.alta-vista.net av-dev4.pa.alta-vista.net scooter.aveurope.co.uk bigip1-snat.sv.av.com   Mercator mercator.pa-x.dec.com scooter.pa.alta-vista.net election2000crawl-complaints-to-admin.webresearch.pa-x.dec.com   Scooter2_Mercator_3-1.0 scooter.sv.av.com   roach.smo.av.com-1.0 avfwclient.sv.av.com   Tv<nn>_Merc_resh_26_1_D-1.0 tv<nn>.sv.av.com www.altavista.co.uk AltaVista-Intranet jan.gelin@av.com host-119.altavista.se www.alltheweb.com FAST-WebCrawler crawler@fast.no 209.67.247.154 [...]...