diff options
| author | Marc Coquand <marc@mccd.space> | 2024-02-14 07:12:57 -0600 |
|---|---|---|
| committer | Marc Coquand <marc@mccd.space> | 2024-02-14 07:12:57 -0600 |
| commit | a954dfb0a52c4ea58e3f263d9beafab385a5b1a7 (patch) | |
| tree | 9c06054f38d7b05b690b069836ec4fa79f50c554 | |
| parent | ebbdacce30ee511e1133aeed5842941a35959c44 (diff) | |
| download | mccd.space-a954dfb0a52c4ea58e3f263d9beafab385a5b1a7.tar.gz mccd.space-a954dfb0a52c4ea58e3f263d9beafab385a5b1a7.tar.bz2 mccd.space-a954dfb0a52c4ea58e3f263d9beafab385a5b1a7.zip | |
Update robots.txt
Diffstat (limited to '')
| -rw-r--r-- | robots.txt | 14 |
1 files changed, 14 insertions, 0 deletions
@@ -1,3 +1,17 @@ +# OpenAI’s web crawler: GPT3.5, GPT4, ChatGPT +User-agent: GPTBot + +# ChatGPT plugins User-agent: ChatGPT-User +# Google's web crawler: Bard, VertexAI, Gemini +User-agent: Google-Extended + +# Claude +User-agent: anthropic-ai + +# Common Crawl +User-agent: CCBot + +# Censorship area Disallow: / |