## Robots.txt file created by http://www.beatlabs.com ## For domain: http:// ## All robots will spider the domain # ADD Disallow: / to block User-agent: * Disallow: ## Disallow FAST-WebCrawler/2.1-pre14 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) #User-agent: FAST-WebCrawler/2.1-pre14 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) #Disallow: / ## Disallow directory /images/ User-agent: * Disallow: /assets/