mycorrhiza/static/robots.txt

User-agent: *
Allow: /help/
Allow: /hypha/
Allow: /recent-changes
Allow: /list
Disallow: /page/
Disallow: /admin/
Disallow: /lock
Disallow: /text/
Disallow: /binary/
Disallow: /rev/
Disallow: /rev-text/
Disallow: /primitive-diff/
Disallow: /media/
Disallow: /edit/
Disallow: /delete/
Disallow: /rename/
Disallow: /remove-media/
Disallow: /history/
Disallow: /orphans
Disallow: /random
Disallow: /title-search/
Disallow: /backlinks/
Disallow: /user-list
Crawl-delay: 5
Extract robots.txt into a file, simplify MIMEs 2021-06-23 22:44:27 +07:00			`User-agent: *`
Allow crawling /help/ 2021-07-31 19:33:15 +05:00			`Allow: /help/`
Extract robots.txt into a file, simplify MIMEs 2021-06-23 22:44:27 +07:00			`Allow: /hypha/`
			`Allow: /recent-changes`
robots.txt: Allow /list, disallow /orphans 2022-05-07 02:00:40 +05:00			`Allow: /list`
Update robots.txt 2022-02-26 16:27:20 +03:00			`Disallow: /page/`
Disallow: /admin/ too 2021-10-01 09:51:06 +03:00			`Disallow: /admin/`
Replace Disallow: / with a more specific list 2021-09-29 22:20:30 +07:00			`Disallow: /lock`
			`Disallow: /text/`
			`Disallow: /binary/`
			`Disallow: /rev/`
			`Disallow: /rev-text/`
			`Disallow: /primitive-diff/`
Update robots.txt 2022-02-26 16:27:20 +03:00			`Disallow: /media/`
Replace Disallow: / with a more specific list 2021-09-29 22:20:30 +07:00			`Disallow: /edit/`
Update robots.txt 2022-02-26 16:27:20 +03:00			`Disallow: /delete/`
			`Disallow: /rename/`
			`Disallow: /remove-media/`
Replace Disallow: / with a more specific list 2021-09-29 22:20:30 +07:00			`Disallow: /history/`
robots.txt: Allow /list, disallow /orphans 2022-05-07 02:00:40 +05:00			`Disallow: /orphans`
Replace Disallow: / with a more specific list 2021-09-29 22:20:30 +07:00			`Disallow: /random`
			`Disallow: /title-search/`
			`Disallow: /backlinks/`
			`Disallow: /user-list`
Extract robots.txt into a file, simplify MIMEs 2021-06-23 22:44:27 +07:00			`Crawl-delay: 5`