#disallow webcrawler/spider/robot access to site


User-agent: *
Disallow: /