User-agent: *
Disallow: admin.php
Disallow: config
Disallow: header
Disallow: footer
Disallow: pntables
Disallow: /docs
Disallow: /javascript



User-agent: MLBot
Disallow: /

User-agent: Slurp
Crawl-delay: 5