nightpool a5992e5883 Change robots.txt to exclude only media proxy URLs (#10038)
* Revert "Change robots.txt to exclude some URLs (#10037)"

This reverts commit 80161f43510ad9316c60c9b50dd5c09c2dae4d54.

* Let's block media_proxy

/media_proxy/ is a dynamic route used for requesting uncached media, so it's
probably bad to let crawlers use it

* misleading comment
2019-02-14 03:11:47 +01:00
..
2018-07-06 01:58:07 +02:00
2017-07-23 02:40:39 +02:00
2017-07-23 02:40:39 +02:00