mirrors/mastodon
1
0
Fork 0
mirror of https://github.com/mastodon/mastodon.git synced 2024-11-20 03:25:17 +01:00
Code Issues Projects Releases Wiki Activity

Update robots.txt to include extra bots and fix a UA typo

Browse Source
This commit is contained in:
UmbreonShapeshifter 2024-08-17 10:14:28 +10:00 committed by GitHub
parent 1b97290155
commit efb70372c0
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 7 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
public/robots.txt
View File

@ -4,6 +4,12 @@
User-agent: anthropic-ai User-agent: anthropic-ai
Disallow: / Disallow: /
User-agent: OmgiliBot
Disallow: /
User-agent: The Knowledge AI
Disallow: /
User-agent: Meta-ExternalAgent User-agent: Meta-ExternalAgent
Disallow: / Disallow: /
@ -52,7 +58,7 @@ Disallow: /
User-agent: Amazonbot User-agent: Amazonbot
Disallow: / Disallow: /
User-agent: 0AI-SearchBot User-agent: OAI-SearchBot
Disallow: / Disallow: /
User-agent: PerplexityBot User-agent: PerplexityBot