mycorrhiza/static/robots.txt

User-agent: *
Allow: /help/
Allow: /hypha/
Allow: /recent-changes
Disallow: /page/
Disallow: /admin/
Disallow: /lock
Disallow: /text/
Disallow: /binary/
Disallow: /rev/
Disallow: /rev-text/
Disallow: /primitive-diff/
Disallow: /media/
Disallow: /edit/
Disallow: /delete/
Disallow: /rename/
Disallow: /remove-media/
Disallow: /history/
Disallow: /list
Disallow: /random
Disallow: /title-search/
Disallow: /backlinks/
Disallow: /user-list
Crawl-delay: 5
Extract robots.txt into a file, simplify MIMEs 2021-06-23 15:44:27 +00:00			`User-agent: *`
Allow crawling /help/ 2021-07-31 14:33:15 +00:00			`Allow: /help/`
Extract robots.txt into a file, simplify MIMEs 2021-06-23 15:44:27 +00:00			`Allow: /hypha/`
			`Allow: /recent-changes`
Update robots.txt 2022-02-26 13:27:20 +00:00			`Disallow: /page/`
Disallow: /admin/ too 2021-10-01 06:51:06 +00:00			`Disallow: /admin/`
Replace Disallow: / with a more specific list 2021-09-29 15:20:30 +00:00			`Disallow: /lock`
			`Disallow: /text/`
			`Disallow: /binary/`
			`Disallow: /rev/`
			`Disallow: /rev-text/`
			`Disallow: /primitive-diff/`
Update robots.txt 2022-02-26 13:27:20 +00:00			`Disallow: /media/`
Replace Disallow: / with a more specific list 2021-09-29 15:20:30 +00:00			`Disallow: /edit/`
Update robots.txt 2022-02-26 13:27:20 +00:00			`Disallow: /delete/`
			`Disallow: /rename/`
			`Disallow: /remove-media/`
Replace Disallow: / with a more specific list 2021-09-29 15:20:30 +00:00			`Disallow: /history/`
			`Disallow: /list`
			`Disallow: /random`
			`Disallow: /title-search/`
			`Disallow: /backlinks/`
			`Disallow: /user-list`
Extract robots.txt into a file, simplify MIMEs 2021-06-23 15:44:27 +00:00			`Crawl-delay: 5`