mycorrhiza/static/robots.txt

mirror of https://github.com/osmarks/mycorrhiza.git synced 2024-12-14 14:20:25 +00:00
User-agent: *
Allow: /help/
Allow: /page/
Allow: /hypha/
Allow: /recent-changes
Disallow: /
Crawl-delay: 5
Extract robots.txt into a file, simplify MIMEs 2021-06-23 15:44:27 +00:00			`User-agent: *`
Allow crawling /help/ 2021-07-31 14:33:15 +00:00			`Allow: /help/`
Extract robots.txt into a file, simplify MIMEs 2021-06-23 15:44:27 +00:00			`Allow: /page/`
			`Allow: /hypha/`
			`Allow: /recent-changes`
			`Disallow: /`
			`Crawl-delay: 5`